Mobile phone RPA automatic system based on natural language analysis

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a mobile phone RPA automatic system based on natural language analysis. The system comprises a voice input interface module which is used for providing an interactive interface for a user and converting voice input by the user into a text; the natural language processing module is used for performing preprocessing, intention classification, entity extraction and task disassembly on the text converted by the voice input interface module and generating structured task description; the RPA module is used for generating a corresponding RPA instruction list by utilizing a mapping rule according to the structured task description, calling an RPA execution engine to sequentially analyze RPA instructions, and simulating user operation on a mobile phone through a UI automatic tool to complete a task; and the feedback display interface module is used for providing feedback information according to the task completion condition and allowing the user to select to continue the operation or re-initiate the task.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a mobile RPA automation system based on natural language parsing. Background Technology

[0002] With the rapid development of artificial intelligence and automation technologies, speech recognition (ASR), natural language processing (NLP), and robotic process automation (RPA) are being widely applied in various fields. Currently, while voice assistants on smartphones (such as Siri and Google Assistant) can perform predefined functions such as making phone calls, sending text messages, and opening applications through voice commands, they still have significant limitations when dealing with complex, multi-step automated tasks, often requiring manual user intervention.

[0003] At the same time, although RPA technology has significantly improved the efficiency of enterprise process automation on PCs, most solutions on mobile devices still rely on users to manually configure or record operation processes. They lack intelligent semantic understanding and task decomposition, and cannot allow users to conveniently define and execute automated tasks through natural language.

[0004] While existing mobile RPA technology can automate multi-step tasks, it has the following shortcomings: 1. Manual configuration is complex and has a high coding threshold. Traditional RPA technology requires users to have a high level of technical expertise, needing to write code or record automation scripts to implement specific tasks. Users often need to set up each operation node step by step, such as clicking locations, entering text, swiping the screen, etc., making the entire process cumbersome and error-prone.

[0005] 2. Lack of natural interaction capabilities Existing mobile RPA tools cannot automatically convert voice commands into executable operations. Users cannot complete complex cross-application automation operations with a single natural language command, and the interaction method is rigid and lacks flexibility.

[0006] 3. Lack of intelligent semantic understanding and task decomposition Current voice assistants (such as Siri and Google Assistant) support certain voice commands, but their task scope is limited and they mostly rely on fixed command sets. Existing mobile RPA tools mostly use hard-coding, meaning the operation process is predefined and fixed, unable to flexibly adjust to real-time environmental changes, and unable to extract task intent from user natural language commands and automatically generate operation steps based on these intents.

[0007] 4. The execution process is not visualized, resulting in a poor user experience. Most mobile RPA tools run in the background while performing tasks, meaning users cannot visually observe the automated process. This lack of visual feedback can raise concerns about the system's transparency and accuracy, especially with complex, multi-step tasks. Furthermore, users cannot intervene or correct problems promptly if issues arise during automated task execution. Therefore, it is very important to design a mobile RPA automation system based on natural language parsing that can combine ASR, NLP and RPA technologies to automatically parse user voice commands and intelligently generate executable operation processes, thereby breaking through the limitations of existing technologies and providing users with a more efficient and convenient mobile automated operation experience. Summary of the Invention

[0008] This invention aims to overcome the limitations of existing mobile RPA technologies, which suffer from complex manual configuration, high coding barriers, lack of natural interaction capabilities, lack of intelligent semantic understanding and task decomposition, and poor user experience due to the lack of visualization of the execution process. It provides a mobile RPA automation system based on natural language parsing that combines ASR, NLP, and RPA technologies to automatically parse user voice commands and intelligently generate executable operation processes. This breaks through the limitations of existing technologies and provides users with a more efficient and convenient mobile automated operation experience.

[0009] To achieve the above-mentioned objectives, the present invention adopts the following technical solution: A mobile RPA automation system based on natural language parsing includes: The voice input interface module provides an interactive interface for users and converts the user's voice input into text. The natural language processing module is used to preprocess, classify intents, extract entities, decompose tasks, and generate structured task descriptions from the text converted by the voice input interface module. The RPA module is used to generate a corresponding RPA instruction list based on the structured task description and mapping rules. At the same time, it calls the RPA execution engine to parse the RPA instructions in sequence and completes the task by simulating user operations on the mobile phone through UI automation tools. The feedback display interface module is used to provide feedback information based on the task completion status and allow users to choose to continue the operation or re-initiate the task.

[0010] Preferably, the voice input interface module includes a voice acquisition module, an audio preprocessing module, an automatic speech recognition module, and a result display and feedback module; The voice acquisition module is used to record audio using audio software; The audio preprocessing module is used to perform noise reduction and echo cancellation, voice activity detection and format conversion on the recording data to obtain preprocessed audio data. The automatic speech recognition module adopts local offline recognition or cloud recognition; after receiving audio data, the cloud or local automatic speech recognition module uses a deep learning model to perform speech recognition, outputs the corresponding text, and returns the output text to the client application. The result display and feedback module is used to display the recognized text, user feedback, and real-time display effect. Displaying the recognized text means displaying the returned text on the user interface, allowing users to confirm, edit, or directly perform subsequent processing. User feedback means providing error prompts and retry functions. If the recognition result does not meet the user's requirements, users can re-enter the voice or perform manual correction. The real-time display effect means using a progress bar or waveform graph to provide real-time feedback on the recording status, allowing users to understand the recording and recognition process.

[0011] Preferably, in the audio preprocessing module, the noise reduction and echo cancellation refer to the use of a noise reduction method based on Fast Fourier Transform (FFT) to denoise the recording data. The speech activity detection refers to using a speech activity detection method to detect whether there is speech in the audio stream; if so, the speech segment is effectively extracted. The format conversion refers to converting the raw PCM data into the format required by the speech recognition engine.

[0012] Preferably, in the automatic speech recognition module, if cloud recognition is used, the preprocessed audio data is encapsulated into a request and sent to the cloud API; the request includes authentication information, audio format and sampling rate.

[0013] As a preferred approach, the specific process for preprocessing text in the natural language processing module is as follows: Clean text: Remove noisy characters, extra spaces, and punctuation marks; Word segmentation: The Chinese word segmentation tool Jieba was used for word segmentation to prepare for subsequent parsing; Standardization: unifying and standardizing different ways of expression.

[0014] Preferably, in the natural language processing module, the intent classification refers to identifying the user's operational intent, and the specific process is as follows: Rule matching: Quickly identify common intents using predefined keywords and regular expressions; Deep learning: uses pre-trained models for semantic understanding and classifies complex or ambiguous expressions.

[0015] Preferably, in the natural language processing module, entity extraction and task decomposition refer to extracting key information from the text and breaking down complex tasks into sub-tasks. The specific process is as follows: Regular expression extraction: using regular expressions to match numbers, units, and specific words; Named Entity Recognition (NER): Extracts entity information using existing Chinese NER models; Task decomposition rules: Based on intent, the extracted entity information is logically combined to form a sequence of task steps; In the natural language processing module, generating a structured task description refers to combining the intent and extracted entity information into structured data for subsequent mapping to specific RPA instructions.

[0016] Preferably, the mapping rules in the RPA module are designed as follows: Establish a mapping table to map each task intent and parameter to a specific RPA operation instruction sequence.

[0017] Preferably, the RPA module includes: The instruction generation module is used to convert structured task descriptions into a list of RPA instructions; The RPA execution module receives a list of RPA instructions and sequentially calls the underlying RPA execution tools to simulate user operations; the specific implementation process is as follows: The generated instructions are converted into corresponding function calls or commands, which are then executed via the Application Programming Interface (API).

[0018] Preferably, the feedback display interface module includes: The results monitoring and log collection module is used to report operation results through internal events or callback interfaces and store result data using a log collector during the execution of the RPA execution module, and then pass it to the feedback processing module. The feedback processing module is used to determine the status field of the received return data; if the status field is determined to be "success", a success message is output in a formatted manner; if the status field is determined to be "failure", the error information is extracted and a recommended operation prompt is added. The user interface module includes a feedback information display area, a progress bar, and operation buttons; The error retry and task resend mechanism module is used to define a unified interface so that the corresponding function is triggered when the operation button in the user interface module is clicked. The unified interface includes: The "Retry" interface is used to re-invoke the RPA execution module, passing in the current task description and parameters, waiting for the return result, and updating the user interface (UI). The "Redescribe the task with voice" interface is used to exit the current task execution state, switch to the voice input interface, reset the task state, and wait for the user to enter a new voice command.

[0019] Compared with the prior art, the beneficial effects of this invention are: (1) Users only need to download and install the App to enjoy a fully intelligent, voice-driven automated operation experience without any tedious manual configuration; (2) The system of this invention can intelligently parse complex voice commands, automatically decompose tasks, generate and execute operation codes, greatly reducing the technical threshold; (3) This invention achieves natural and efficient automated processes through voice interaction, significantly improving user experience and operational efficiency; (4) This invention is widely applicable to the field of smartphone operation automation, such as social application automation (automatic red envelope sending, automatic message sending), mobile payment, online shopping, smart home control and other scenarios; (5) This invention is suitable for enterprise-level process automation needs, and can also be used as a personal intelligent assistant to provide convenient and efficient automation tools for a wide range of non-technical users. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the architecture of the mobile RPA automation system based on natural language parsing according to the present invention; Figure 2 This is a flowchart illustrating the implementation of the mobile RPA automation system based on natural language parsing in a practical application. Detailed Implementation

[0021] To more clearly illustrate the embodiments of the present invention, specific implementation methods will be described below with reference to the accompanying drawings. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings and other implementation methods can be obtained based on these drawings without any creative effort.

[0022] like Figure 1 As shown, this invention provides a mobile RPA automation system based on natural language parsing, which mainly includes the following modules: This invention designs a voice input interface module to provide users with a user-friendly interactive interface, converting user-input voice into text. This interface includes the following modules: 1. Voice acquisition module: Hardware and permissions: (1) Use the phone's built-in microphone. (2) The application must request microphone access permission before it starts (Android uses RECORD_AUDIO permission, and iOS uses AVAudioSession recording permission).

[0023] Implementation method: (1) Record audio using the AudioRecord class. (2) Configure the sampling rate (e.g., 16kHz), sampling format (16-bit PCM), and mono input.

[0024] 2. Audio preprocessing module: Noise reduction and echo cancellation: (1) An FFT-based noise reduction algorithm is used. (2) The recording data is denoised to ensure clear audio signal.

[0025] Voice Activity Detection (VAD): The VAD algorithm is used to detect the presence of speech in the audio stream, effectively extracting speech segments and avoiding blank data from affecting recognition.

[0026] Format conversion: Converts raw PCM data to formats required by the ASR engine (such as WAV, FLAC, or compressed MP3 format), and performs sampling rate or encoding format conversion on audio data when necessary.

[0027] 3. Automatic Speech Recognition (ASR) Module: ASR Engine Selection: Local offline recognition: Uses the PocketSphinx offline recognition library, suitable for scenarios with low network requirements.

[0028] Cloud-based recognition: Integrates with Google Speech-to-Text service, sends audio data to the cloud for processing via network request, and returns the recognition results.

[0029] Implementation process: Data transmission: If cloud-based recognition is used, the preprocessed audio data is encapsulated into a request (such as an HTTP POST request) and sent to the cloud API. The request needs to include parameters such as authentication information, audio format, and sampling rate.

[0030] ASR processing: After receiving audio data, the cloud-based or local ASR module uses a deep learning model (the Transformer model) to perform speech recognition and output the corresponding text.

[0031] Result return: The recognition results are returned to the client application in text format.

[0032] 4. Results Display and Feedback Module: Display the recognized text: Display the text returned by the ASR module on the user interface, allowing users to confirm, edit, or directly perform subsequent processing (such as triggering the NLP module for further semantic analysis).

[0033] User feedback: Provides error messages and retry functions. If the recognition result is not ideal, it allows users to re-enter their voice or perform manual correction.

[0034] Real-time display: Progress bars or waveform graphs can be used to provide real-time feedback on the recording status, allowing users to understand the recording and recognition process.

[0035] The Natural Language Processing (NLP) module preprocesses, classifies, extracts entities, and decomposes the text returned by the ASR module to generate a structured task description. The specific process is as follows: 1. Text preprocessing step: Cleanse text: Remove noisy characters, extra spaces, and punctuation marks.

[0036] Word segmentation: The Chinese word segmentation tool Jieba was used for word segmentation to prepare for subsequent parsing.

[0037] Standardization: unifying different expressions, such as standardizing "10 yuan" and "10 yuan" into "10 yuan".

[0038] 2. Intent Classification Objective: To identify the user's primary operational intent, such as "open the application", "send a red envelope", "send a message", etc.

[0039] method: Rule matching: Quickly identify common intents using predefined keywords and regular expressions.

[0040] Deep learning: uses pre-trained models for semantic understanding and classifies complex or ambiguous expressions.

[0041] 3. Entity Extraction and Task Decomposition Objective: Extract key information (entities) from the text, such as the group chat name, number of red envelopes, amount, and type of red envelope, and break down the complex task into sub-tasks.

[0042] method: Regular expression extraction: using regular expressions to match numbers, units, and specific words.

[0043] Named Entity Recognition (NER): Extracts entity information using existing Chinese NER models (such as LTP and HanLP).

[0044] Task decomposition rules: Based on intent, the extracted entities are logically combined to form a sequence of task steps.

[0045] 4. Generate structured task descriptions The intent and extracted entities are combined into structured data, which facilitates subsequent mapping to specific RPA instructions: The RPA module generates a corresponding RPA instruction list based on the task description and mapping rules. The RPA execution engine then parses the instructions sequentially, and the task is completed on the mobile phone using UI automation tools to simulate user actions. The specific process is as follows: 1. Mapping rule design Establish a mapping table to map each task intent and parameter to a specific RPA operation instruction sequence. For example, for the "send_red_packet" task, the following operation steps can be predefined: A. Open the WeChat application.

[0046] B. Enter the group chat (based on chat_name).

[0047] C. Click the "Send Red Envelope" button.

[0048] D. Fill in the amount and count of the red envelopes.

[0049] E. Click "Send".

[0050] 2. Instruction Generation Module The structured task description is converted into a list of instructions, which can be in JSON format, so that they can be passed to the RPA execution layer.

[0051] 3. RPA Execution Module Function: Receives a list of RPA instructions and sequentially calls the underlying RPA execution tool (UiAutomator) to simulate user operations.

[0052] Implementation method: The generated instructions are converted into corresponding function calls or commands, which are then executed via API calls.

[0053] For example, you can call the Python library to control UiAutomator to complete corresponding clicks, inputs, and other operations.

[0054] This invention also includes a feedback display interface module, which provides clear feedback information regardless of whether the task succeeds or fails, and allows users to choose to continue or re-initiate the task. This not only enhances the user experience but also significantly improves the system's reliability and interactivity.

[0055] Feedback display interface module, such as Figure 2 As shown, the specific process functions include the following: 1. Results monitoring and log collection Implementation method: During the operation of the RPA execution module, the system will actively report the execution result of the current operation through the internal event mechanism or callback interface. In order to improve the standardization of feedback and processing efficiency, the system has designed a unified result feedback format, which includes the following four core elements: (1) Status identifier: used to indicate whether the task execution is successful, usually "success" or "failure"; (2) Message information: when the task is successful, the corresponding success message is returned; when it fails, the specific error information is returned for subsequent diagnosis; (3) Execution log content: records the detailed execution process of this task, as a basis for debugging and auditing; (4) Timestamp information: identifies the accurate time of task completion, used for process tracking and timing analysis.

[0056] The execution module will generate the structured results described above at the end of the task and pass them to the log collector. The log collector is responsible for uniformly archiving the result data and further passing it to the feedback processing module to support subsequent functions such as user prompts, anomaly monitoring, and task backtracking.

[0057] 2. Feedback Processor Implementation method: The feedback processing module receives and parses the data returned by the RPA execution module. The module first identifies and judges the status field in the returned result: When the status is "success", the system will automatically format the success message and output a standardized success message so that the user can confirm the task execution result; When the status is "failure", the system will extract the error message description, combine it with the error type and context, show the user the clear reason for the failure, and provide corresponding recommended operation prompts to help the user troubleshoot or correct the operation.

[0058] This module can further improve the efficiency of problem localization by combining log information, and is a key feedback node in the task execution closed loop.

[0059] 3. User Interface (UI) Interface design: Design a dedicated feedback page, including a feedback information display area, a progress bar, and operation buttons.

[0060] Example of a success feedback interface: "Task successfully completed! Time taken: XX seconds" Display detailed execution logs (optional).

[0061] Provide a "Confirm" or "Back" button.

[0062] Example of a failure feedback interface: "Task execution failed: Target control could not be found." Two operation buttons: "Retry" and "Re-describe the task with voice".

[0063] Front-end technologies: It is implemented using the native Android UI framework.

[0064] 4. Error retry and task reschedule mechanism Implementation method: Define a unified interface so that the corresponding function is triggered when the UI button is clicked.

[0065] The "Retry" interface: Recalls the RPA execution module, passing in the current task description and parameters, waits for the return result, and updates the UI.

[0066] The "Redescribe the task with voice" interface exits the current task execution state, switches to the voice input module interface, resets the task state, and waits for the user to enter a new voice command.

[0067] Based on the technical solution of this invention, the implementation process of this invention in practical applications will be illustrated through the following case scenarios. The specific application implementation scheme is as follows: Automating mobile tasks based on voice commands: The user's voice command was: "Please open WeChat for me and send 10 lucky red envelopes to the 'Cute Girls' group chat. The total amount is 10 yuan."

[0068] 1-1: When a user launches the App and clicks the "Start Voice Input" button on the main interface, the system requests and obtains microphone access permission.

[0069] 1-2: Utilize the phone's built-in microphone to collect user voice signals in real time using the AudioRecord class on the Android platform.

[0070] 1-3: Perform noise reduction, speech activity detection (VAD), and format conversion (convert to 16kHz, 16-bit mono PCM data) on the acquired audio data.

[0071] 1-4: The preprocessed audio data is passed to the ASR module, which calls the cloud-based speech recognition service to convert the user's speech into text in real time. After conversion by the ASR module, the generated text is: "Please help me open WeChat and send a lucky draw red envelope to the 'Little Cuties' group chat. There are 10 red envelopes, and the total amount is 10 yuan."

[0072] 1-5: After obtaining the text, the system sends it to the Natural Language Processing (NLP) module for semantic analysis and intent recognition. In this example, the user's intent is identified as "send a red envelope". 1-6: Using regular expressions and named entity recognition technology, extract key information from the text, such as application name, group chat participants, red envelope type, number of red envelopes and amount, and structure it into a standard task description.

[0073] 1-7: The system maps predefined RPA operation processes, breaking down tasks into specific instructions, including launching the specified application "WeChat", locating the specific chat window "Little Cutie Group", simulating clicking the red envelope button, filling in the amount "10" and quantity "10" fields, and finally executing the send operation.

[0074] 1-8: Call the API based on UiAutomator to parse and execute the above instructions in sequence, simulating the user's real operation on the mobile phone screen to realize the execution of automated tasks.

[0075] 1-9: After all RPA commands are executed successfully, the system collects execution logs and time data, and displays a prompt message on the UI interface: "Task completed successfully! Time taken 15 seconds," while providing options to "Return to homepage" or "View logs."

[0076] The innovative aspects of this invention are as follows: This invention integrates Automatic Speech Recognition (ASR), Natural Language Processing (NLP), and Robotic Process Automation (RPA) technologies to construct a mobile RPA automation system based on natural language parsing. Its core innovations and technical protection points are mainly reflected in the following aspects: 1. Voice-driven natural interaction and task triggering Key technologies: The system achieves real-time conversion of voice input by embedding a high-precision ASR engine (supporting both local offline recognition and cloud-based recognition modes). Combined with algorithms such as adaptive noise suppression and voice activity detection (VAD), it ensures accurate capture of user voice in various complex environments.

[0077] Innovation: Breaking away from the traditional manual operation mode, users can complete complex tasks through natural voice commands without touching the screen; voice input and real-time text feedback form a closed loop, improving the user interaction experience and ease of operation.

[0078] 2. Intelligent semantic understanding and task decomposition Key Technology: This invention employs a dual parsing strategy combining local fast regular expression matching with advanced cloud-based NLP models (such as those based on Transformer or BERT). The system first performs rapid parsing of common commands using predefined rules; for ambiguous or complex statements, it invokes cloud-based semantic understanding services to accurately identify user intent and extract key entities (such as application name, group chat name, red envelope type, quantity, amount, etc.).

[0079] Innovation: This module enables automatic conversion from natural language to structured task descriptions, breaking down complex tasks into sub-tasks and automatically generating execution parameters for each sub-task. It addresses the issues of hard-coded task flows and insufficient flexibility in existing technologies, providing intelligent and dynamic task planning and scheduling capabilities for automated operations.

[0080] 3. Fully automated generation and execution of RPA operation processes Key technologies: Based on the parsed structured task description, the system builds a predefined operation template library and automatically generates corresponding RPA operation instructions through mapping rules. Utilizing underlying automation tools such as UiAutomator and Appium, the system achieves fully automated execution of various interactive operations such as "clicking, swiping, and inputting" on mobile devices.

[0081] Innovation: This solution innovatively combines intelligent voice input with RPA technology, enabling the automatic generation and execution of natural language commands into executable code, significantly lowering the technical barrier to operation configuration. It not only supports automated processes across applications and tasks but also boasts excellent scalability, allowing for flexible adjustments to operation templates for different application scenarios.

[0082] 4. Real-time feedback and exception handling mechanism throughout the entire process Key technologies: The system incorporates complete result monitoring, log collection, and feedback processing modules during RPA task execution. Regardless of whether the task succeeds or fails, the system can promptly display the operation results, execution time, detailed logs, and error messages through the user interface, and provide operation options such as "retry" or "re-describe the task by voice".

[0083] Innovation: Through transparent operation process display and interactive design, the system not only enhances users' trust in the system, but also enables the system to have self-diagnosis and rapid recovery capabilities, thereby improving the overall system stability and user experience.

[0084] 5. System integration and scalability Key technologies: This invention organically integrates ASR, NLP, RPA, and feedback modules through modular design, constructing a complete closed-loop automation system. Seamless collaboration between modules is achieved through a unified data interface and calling protocol, and interfaces are reserved to support future expansion (e.g., support for more languages, processing more complex tasks, and multi-platform compatibility).

[0085] Innovation: The system not only achieves breakthroughs at a single technical level, but also realizes high integration and flexible expansion in its overall architecture, meeting the needs of different users and scenarios. This modular and pluggable design concept significantly enhances the system's commercial application prospects and competitiveness.

[0086] In summary, this invention, in realizing a natural voice-driven automated operating system for mobile phones, achieves efficient and stable execution of automated tasks by organically integrating speech recognition, intelligent semantic parsing, automatic task decomposition and RPA instruction generation, as well as real-time feedback and fault tolerance mechanisms throughout the entire process.

[0087] This invention can parse and recognize complex voice commands, which requires deep learning and model training. It can also break down complex tasks, unlike Siri and other similar products that can only understand simple commands. It can automate execution on mobile devices, while RPA technology is widely used on PCs but less so on mobile devices. It features a user-friendly UI that allows users to adjust commands and identify problems.

[0088] The above description is merely a detailed explanation of preferred embodiments and principles of the present invention. For those skilled in the art, there may be changes in specific implementation methods based on the ideas provided by the present invention, and these changes should also be considered within the scope of protection of the present invention.

Claims

1. A mobile RPA automation system based on natural language parsing, characterized in that, include: The voice input interface module provides an interactive interface for users and converts the user's voice input into text. The natural language processing module is used to preprocess, classify intents, extract entities, decompose tasks, and generate structured task descriptions from the text converted by the voice input interface module. The RPA module is used to generate a corresponding RPA instruction list based on the structured task description and mapping rules. At the same time, it calls the RPA execution engine to parse the RPA instructions in sequence and completes the task by simulating user operations on the mobile phone through UI automation tools. The feedback display interface module is used to provide feedback information based on the task completion status and allow users to choose to continue the operation or re-initiate the task.

2. The mobile RPA automation system based on natural language parsing according to claim 1, characterized in that, The voice input interface module includes a voice acquisition module, an audio preprocessing module, an automatic speech recognition module, and a result display and feedback module. The voice acquisition module is used to record audio using audio software; The audio preprocessing module is used to perform noise reduction and echo cancellation, voice activity detection and format conversion on the recording data to obtain preprocessed audio data. The automatic speech recognition module adopts local offline recognition or cloud recognition; after receiving audio data, the cloud or local automatic speech recognition module uses a deep learning model to perform speech recognition, outputs the corresponding text, and returns the output text to the client application. The results display and feedback module is used to display the recognized text, user feedback, and real-time display effects; The "displaying recognized text" refers to displaying the returned text on the user interface, allowing users to confirm, edit, or directly perform subsequent processing; the "user feedback" refers to providing error prompts and retry functions, allowing users to re-enter voice or perform manual correction if the recognition result does not meet user requirements; the "real-time display effect" refers to using a progress bar or waveform graph to provide real-time feedback on the recording status, allowing users to understand the recording and recognition process.

3. The mobile RPA automation system based on natural language parsing according to claim 2, characterized in that, In the audio preprocessing module, the noise reduction and echo cancellation refer to the use of a noise reduction method based on Fast Fourier Transform (FFT) to denoise the recording data. The speech activity detection refers to using a speech activity detection method to detect whether there is speech in the audio stream; if so, the speech segment is effectively extracted. The format conversion refers to converting the raw PCM data into the format required by the speech recognition engine.

4. The mobile RPA automation system based on natural language parsing according to claim 3, characterized in that, In the automatic speech recognition module, if cloud-based recognition is used, the preprocessed audio data is encapsulated into a request and sent to the cloud API; the request includes authentication information, audio format, and sampling rate.

5. The mobile RPA automation system based on natural language parsing according to claim 1, characterized in that... In the natural language processing module, intent classification refers to identifying the user's operational intent, and the specific process is as follows: Rule matching: Quickly identify common intents using predefined keywords and regular expressions; Deep learning: uses pre-trained models for semantic understanding and classifies complex or ambiguous expressions.

6. The mobile RPA automation system based on natural language parsing according to claim 5, characterized in that, In the natural language processing module, entity extraction and task decomposition refer to extracting key information from text and breaking down complex tasks into sub-tasks. The specific process is as follows: Regular expression extraction: using regular expressions to match numbers, units, and specific words; Named Entity Recognition (NER): Extracts entity information using existing Chinese NER models; Task decomposition rules: Based on intent, the extracted entity information is logically combined to form a sequence of task steps.

7. The mobile RPA automation system based on natural language parsing according to claim 6, characterized in that, In the natural language processing module, generating a structured task description refers to combining the intent and extracted entity information into structured data for subsequent mapping to specific RPA instructions.

8. The mobile RPA automation system based on natural language parsing according to claim 1, characterized in that, In the RPA module, the mapping rules are designed as follows: Establish a mapping table to map each task intent and parameter to a specific RPA operation instruction sequence.

9. The mobile RPA automation system based on natural language parsing according to claim 8, characterized in that, The RPA module includes: The instruction generation module is used to convert structured task descriptions into a list of RPA instructions; The RPA execution module receives a list of RPA instructions and sequentially calls the underlying RPA execution tools to simulate user operations; the specific implementation process is as follows: The generated instructions are converted into corresponding function calls or commands, which are then executed via the Application Programming Interface (API).

10. The mobile RPA automation system based on natural language parsing according to claim 9, characterized in that, The feedback display interface module includes: The results monitoring and log collection module is used to report operation results through internal events or callback interfaces and store result data using a log collector during the execution of the RPA execution module, and then pass it to the feedback processing module. The feedback processing module is used to determine the status field in the received return data; if the status field is determined to be "success", a success message is output in a formatted manner. If the status field indicates failure, extract the error message and add recommended action suggestions. The user interface module includes a feedback information display area, a progress bar, and operation buttons.

Images