Mobile phone rpa automation system based on natural language parsing

Abstract

The application belongs to the technical field of artificial intelligence, and particularly relates to a mobile phone RPA automation system based on natural language analysis. The system comprises a voice input interface module, which is used for providing an interactive interface for a user and converting voice input by the user into text; a natural language processing module, which is used for pre-processing, intent classification, entity extraction and task decomposition of the text converted by the voice input interface module and generating a structured task description; an RPA module, which is used for generating a corresponding RPA instruction list according to the structured task description, calling an RPA execution engine to sequentially analyze the RPA instructions, and simulating user operation on a mobile phone through a UI automation tool to complete a task; and a feedback display interface module, which is used for providing feedback information according to the completion of the task and allowing the user to select to continue operation or re-initiate the task.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and particularly relates to a mobile phone RPA automation system based on natural language parsing. BACKGROUND

[0002] With the rapid development of artificial intelligence and automation technology, speech recognition (ASR), natural language processing (NLP) and robotic process automation (RPA) are widely applied in various fields. At present, although voice assistants on smartphones (such as Siri, Google Assistant, etc.) can complete predefined functions such as making calls, sending messages, opening applications, etc. through voice commands, there are still great limitations in dealing with complex, multi-step automation tasks, and often require user manual intervention.

[0003] At the same time, although RPA technology has significantly improved the efficiency of enterprise process automation on the PC side, most solutions still rely on user manual configuration or recording of operation processes in the application of mobile phones, lack intelligent semantic understanding and task decomposition, and cannot allow users to conveniently define and execute automation tasks through natural language.

[0004] Although existing mobile RPA technology can implement multi-step automation tasks, it has the following shortcomings:

[0005] 1. Manual configuration is complex and has high code threshold

[0006] Traditional RPA technology requires users to have a high level of technical expertise and need to write code or record automation scripts to implement specific tasks. Users often need to set up each operation node step by step, such as clicking position, inputting text, sliding screen, etc. The configuration process of the entire process is tedious and prone to errors.

[0007] 2. Lack of natural interaction ability

[0008] Existing mobile RPA tools cannot automatically convert voice commands into executable operations, and users cannot complete complex cross-application automation operations through a natural language instruction. The interaction is rigid and lacks flexibility.

[0009] 3. Lack of intelligent semantic understanding and task decomposition

[0010] Current voice assistants (such as Siri, Google Assistant) support certain voice commands, but their task range is limited and mostly relies on a fixed command set. Existing mobile RPA tools mostly use hard coding, which means that the operation process is predefined and fixed, cannot be flexibly adjusted according to real-time environmental changes, cannot extract task intent from user natural language instructions, and cannot automatically generate operation steps according to these intents.

[0011] 4. Execution process visualization is not available, and user experience is poor

[0012] Most mobile RPA tools run in the background when performing tasks, and users cannot directly see the execution process of automated operations. Due to the lack of visual feedback of the execution process, users may be concerned about the transparency and accuracy of the system when performing complex multi-step tasks. Moreover, if there is a problem with the execution of the automated task, the user cannot intervene and correct it in time

[0013] Therefore, it is very important to design a mobile RPA automation system based on natural language analysis that combines ASR, NLP and RPA technologies, automatically analyzes user voice instructions and intelligently generates executable operation processes, thereby breaking through the limitations of existing technologies and providing users with more efficient and convenient mobile automation operation experience. SUMMARY

[0014] The present application is to overcome the problems of existing mobile RPA technology, such as complex manual configuration, high code threshold, lack of natural interaction ability, lack of intelligent semantic understanding and task decomposition, and lack of execution process visualization and poor user experience. A mobile RPA automation system based on natural language analysis is provided, which combines ASR, NLP and RPA technologies, automatically analyzes user voice instructions and intelligently generates executable operation processes, thereby breaking through the limitations of existing technologies and providing users with more efficient and convenient mobile automation operation experience.

[0015] In order to achieve the above-mentioned application purposes, the following technical solutions are adopted in the present application:

[0016] The mobile RPA automation system based on natural language analysis comprises:

[0017] A voice input interface module for providing an interactive interface for users and converting user input voice into text;

[0018] A natural language processing module for pre-processing, intent classification, entity extraction, task decomposition and generating structured task description of the text converted by the voice input interface module;

[0019] An RPA module for generating a corresponding RPA instruction list according to the structured task description, calling an RPA execution engine to sequentially analyze the RPA instructions, and simulating user operations on the mobile phone through a UI automation tool to complete the task;

[0020] A feedback display interface module for providing feedback information according to the task completion status and allowing the user to choose to continue operation or re-initiate the task.

[0021] As preferred, the voice input interface module comprises a voice collection module, an audio preprocessing module, an automatic speech recognition module and a result display and feedback module.

[0022] The voice collection module is configured to use audio software to record voice.

[0023] The audio preprocessing module is configured to perform noise reduction and echo cancellation, voice activity detection and format conversion on the recorded voice data to obtain preprocessed audio data.

[0024] The automatic speech recognition module adopts local offline recognition or cloud recognition. After receiving the audio data, the automatic speech recognition module uses a deep learning model to perform speech recognition, outputs corresponding text, and returns the output text to the client application.

[0025] The result display and feedback module is configured to display the recognized text, user feedback and real-time display effect. The display of the recognized text refers to displaying the returned text on the user interface, allowing the user to confirm, edit or directly perform subsequent processing. The user feedback refers to providing error prompts and retry functions, allowing the user to re-enter voice or manually correct if the recognition result does not meet the user's requirements. The real-time display effect refers to using a progress bar or waveform graph to real-time feedback the recording state, allowing the user to understand the recording and recognition process.

[0026] As preferred, in the audio preprocessing module, the noise reduction and echo cancellation refers to using a noise reduction method based on fast Fourier transform (FFT) to perform de-noising processing on the recorded voice data.

[0027] The voice activity detection refers to using a voice activity detection method to detect whether there is voice in the audio stream. If there is, the voice segment is effectively intercepted.

[0028] The format conversion refers to converting the original PCM data into a format required by the speech recognition engine.

[0029] As preferred, in the automatic speech recognition module, if cloud recognition is adopted, the preprocessed audio data is packaged into a request and transmitted to the cloud API. The request includes authentication information, audio format and sampling rate.

[0030] As preferred, in the natural language processing module, the specific process of preprocessing the text is as follows:

[0031] Cleaning the text: removing noise characters, extra spaces and punctuation marks;

[0032] Word segmentation: using the Chinese word segmentation tool Jieba to perform word segmentation processing to prepare for subsequent analysis;

[0033] Standardization: unify different expressions into a standard.

[0034] As preferred, in the natural language processing module, the intent classification refers to identifying the user's operation intent, and the specific process is as follows:

[0035] Rule matching: quickly identify common intents using predefined keywords and regular expressions;

[0036] Deep learning: use pre-trained models for semantic understanding and classify complex or ambiguous expressions.

[0037] As preferred, in the natural language processing module, the entity extraction and task decomposition refer to extracting key information from text and decomposing complex tasks into subtasks, and the specific process is as follows:

[0038] Regular expression extraction: use regular expression to match numbers, units and specific words;

[0039] Named Entity Recognition (NER): use existing Chinese NER models to extract entity information;

[0040] Task decomposition rules: based on intent, logically combine extracted entity information to form a task step sequence;

[0041] In the natural language processing module, the structured task description generation refers to combining intent and extracted entity information into structured data for subsequent mapping to specific RPA instructions.

[0042] As preferred, in the RPA module, the mapping rule design is as follows:

[0043] Establish a mapping table to map each task intent and parameter to a specific RPA operation instruction sequence.

[0044] As preferred, the RPA module includes:

[0045] Instruction generation module for converting structured task description into RPA instruction list;

[0046] RPA execution module for receiving RPA instruction list and sequentially calling underlying RPA execution tools to simulate user operations; the specific implementation process is as follows:

[0047] Convert the generated instructions into corresponding function calls or commands and execute through Application Programming Interface (API) calls.

[0048] As preferred, the feedback display interface module includes:

[0049] A result monitoring and log collection module is configured to report operation results through internal events or callback interfaces during process execution by the RPA execution module, store result data using a log collector, and deliver the result data to the feedback processing module.

[0050] The feedback processing module is configured to determine the status field in the received return data, format an output success message if the status field is determined to be success, and extract error information and attach a recommended operation prompt if the status field is determined to be failure.

[0051] The user interaction interface module includes a feedback information display area, a progress bar, and operation buttons.

[0052] The error retry and task reissue mechanism module is configured to define a unified interface to trigger corresponding functions when the operation buttons in the user interaction interface module are clicked.

[0053] The unified interface includes:

[0054] A retry interface is configured to re-call the RPA execution module, input the current task description and parameters, wait for a return result, and update the user interaction interface (UI).

[0055] A re-voice task description interface is configured to exit the current task execution state, switch to a voice input interface, reset the task state, and wait for a user to input a new voice command.

[0056] Compared with the prior art, the present application has the following beneficial effects: (1) the user only needs to download and install the App to enjoy the full intelligent, voice-driven automation operation experience without any tedious manual configuration; (2) the system can intelligently analyze complex voice instructions, automatically disassemble tasks, generate and execute operation codes, greatly reducing the technical threshold; (3) the present application realizes natural and efficient automation processes through voice interaction, significantly improving user experience and operation efficiency; (4) the present application is widely applicable to the field of smartphone operation automation, such as social application automation (automatic red packet sending, automatic message sending), mobile payment, online shopping, smart home control, and various other scenarios; (5) the present application is suitable for enterprise-level process automation needs and can also be used as a personal intelligent assistant to provide convenient and efficient automation tools for the general non-technical user. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 FIG. 1 is a schematic diagram of a framework of a smartphone RPA automation system based on natural language analysis according to the present application;

[0058] Figure 2 FIG. 2 is a flowchart of a smartphone RPA automation system based on natural language analysis according to the present application in actual application implementation. DETAILED DESCRIPTION

[0059] In order to more clearly illustrate the embodiments of the present application, the specific embodiments of the present application will be described below with reference to the accompanying drawings. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained from these drawings and other embodiments can be obtained by those skilled in the art without creative labor.

[0060] As Figure 1 shown, the present application provides a mobile phone RPA automation system based on natural language analysis, which mainly includes the following modules:

[0061] The present application designs a voice input interface module to provide a friendly interactive interface for users, and converts the voice input by the user into text. The interface includes the following modules:

[0062] 1. Voice collection module:

[0063] Hardware and permissions: (1) Use the built-in microphone of the mobile phone. (2) The application must request the microphone use permission before starting (Android uses the RECORD_AUDIO permission, and iOS uses the recording permission of AVAudioSession).

[0064] Implementation: (1) Use the AudioRecord class for recording. (2) Configure the sampling rate (for example, 16 kHz), the sampling format (16-bit PCM), and the single-channel input.

[0065] 2. Audio preprocessing module:

[0066] Noise reduction and echo cancellation: (1) Use the FFT-based noise reduction algorithm. (2) Perform noise reduction processing on the recorded data to ensure that the audio signal is clear.

[0067] Voice activity detection (VAD): Apply the VAD algorithm to detect whether there is voice in the audio stream, effectively intercept the voice segment, and avoid the influence of blank data on recognition.

[0068] Format conversion: Convert the original PCM data to the format required by the ASR engine (such as WAV, FLAC, or compressed MP3 format), and perform sampling rate or encoding format conversion on the audio data if necessary.

[0069] 3. Automatic speech recognition (ASR) module:

[0070] ASR engine selection:

[0071] Local offline recognition: Use the PocketSphinx offline recognition library, which is suitable for scenarios with low network requirements.

[0072] Cloud-based recognition: Integrates with Google Speech-to-Text service, sends audio data to the cloud for processing via network request, and returns the recognition results.

[0073] Implementation process:

[0074] Data transmission: If cloud-based recognition is used, the preprocessed audio data is encapsulated into a request (such as an HTTP POST request) and sent to the cloud API. The request needs to include parameters such as authentication information, audio format, and sampling rate.

[0075] ASR processing: After receiving audio data, the cloud-based or local ASR module uses a deep learning model (the Transformer model) to perform speech recognition and output the corresponding text.

[0076] Result return: The recognition results are returned to the client application in text format.

[0077] 4. Results Display and Feedback Module:

[0078] Display the recognized text: Display the text returned by the ASR module on the user interface, allowing users to confirm, edit, or directly perform subsequent processing (such as triggering the NLP module for further semantic analysis).

[0079] User feedback: Provides error messages and retry functions. If the recognition result is not ideal, it allows users to re-enter their voice or perform manual correction.

[0080] Real-time display: Progress bars or waveform graphs can be used to provide real-time feedback on the recording status, allowing users to understand the recording and recognition process.

[0081] The Natural Language Processing (NLP) module preprocesses, classifies, extracts entities, and decomposes the text returned by the ASR module to generate a structured task description. The specific process is as follows:

[0082] 1. Text preprocessing

[0083] step:

[0084] Cleanse text: Remove noisy characters, extra spaces, and punctuation marks.

[0085] Word segmentation: The Chinese word segmentation tool Jieba was used for word segmentation to prepare for subsequent parsing.

[0086] Standardization: unifying different expressions, such as standardizing "10 yuan" and "10 yuan" into "10 yuan".

[0087] 2. Intent Classification

[0088] Objective: Identify the user's primary operational intent, such as "open application", "send red packet", "send message", etc.

[0089] Method:

[0090] Rule Matching: Quickly identify common intents using predefined keywords and regular expressions.

[0091] Deep Learning: Use pre-trained models for semantic understanding to classify complex or ambiguous expressions.

[0092] 3. Entity Extraction and Task Decomposition

[0093] Objective: Extract key information (entities) from text, such as group chat name, red packet quantity, amount, red packet type, etc., and decompose complex tasks into subtasks.

[0094] Method:

[0095] Regular Expression Extraction: Use regular expressions to match numbers, units, and specific words.

[0096] Named Entity Recognition (NER): Use existing Chinese NER models (such as LTP, HanLP) to extract entity information.

[0097] Task Decomposition Rules: Based on intent, logically combine extracted entities to form a sequence of task steps.

[0098] 4. Generate Structured Task Description

[0099] Combine intent and extracted entities into structured data for subsequent mapping to specific RPA instructions:

[0100] RPA module generates corresponding RPA instruction list based on task description using mapping rules. Call RPA execution engine to parse instructions one by one, simulate user operations on the phone through UI automation tools to complete tasks, specific process as follows:

[0101] 1. Mapping Rule Design

[0102] Establish a mapping table to map each task intent and parameter to a specific sequence of RPA operation instructions. For example, for the "send_red_packet" task, the following operation steps can be predefined:

[0103] A. Open WeChat application.

[0104] B. Enter group chat (locate according to chat_name).

[0105] C. Click the "Send Red Packet" button.

[0106] D. Fill in the red envelope amount (amount) and quantity (count).

[0107] E. Click "Send".

[0108] 2. Instruction generation module

[0109] Convert the structured task description into a list of instructions, which can be in JSON format to pass to the RPA execution layer.

[0110] 3. RPA execution module

[0111] Function: Receive the RPA instruction list and call the underlying RPA execution tool (UiAutomator) in turn to simulate user operations.

[0112] Implementation:

[0113] Convert the generated instructions into corresponding function calls or commands and execute them through API calls.

[0114] For example, call the Python library to control UiAutomator to complete corresponding clicks, inputs, etc.

[0115] The invention also includes a feedback display interface module, whether the task is successful or failed, the system can provide clear feedback information, and allow the user to choose to continue operation or re-initiate the task. This not only improves the user experience, but also greatly improves the reliability and interactivity of the system.

[0116] The feedback display interface module, as shown in Figure 2 , specifically includes the following process functions:

[0117] 1. Result monitoring and log collection

[0118] Implementation:

[0119] During the running of the RPA execution module, the system will actively report the execution results of the current operation through internal event mechanisms or callback interfaces. In order to improve the standardization and processing efficiency of feedback, the system designs a unified result feedback format, which contains the following four core elements: (1) status identifier status: used to indicate whether the task execution is successful, usually "success" or "failure"; (2) information description message: when the task is successful, return the corresponding success prompt; when it fails, return specific error information to facilitate subsequent diagnosis; (3) execution log content log: records the detailed execution process of this task as a basis for debugging and auditing; (4) timestamp information timestamp: identifies the accurate time of task completion, used for process tracking and time sequence analysis.

[0120] The execution module will generate the structured results mentioned above at the end of the task and pass them to the log collector. The log collector is responsible for uniformly archiving the result data and further passing it to the feedback processing module to support subsequent user prompts, abnormality monitoring, task backtracking, etc.

[0121] 2. Feedback processor

[0122] Implementation:

[0123] The feedback processing module is used to receive and analyze the data content returned by the RPA execution module. The module first identifies and judges the status field (status) in the returned results:

[0124] When the status is "success", the system will automatically format the success information and output standardized success prompts for the user to confirm the task execution results;

[0125] When the status is "failure", the system will extract the error information description content, combine the error type and context environment, and show the user the specific failure reason and provide the corresponding recommended operation prompt to assist the user in troubleshooting or operation correction.

[0126] This module can further improve the problem positioning efficiency in combination with log information, and is a key feedback node in the task execution closed loop.

[0127] 3. User interaction interface (UI)

[0128] Interface design:

[0129] Design a special feedback page, including feedback information display area, progress bar, and operation buttons.

[0130] Example of successful feedback interface:

[0131] "Task completed successfully! Time consumed: XX seconds"

[0132] Display detailed execution logs (optional).

[0133] Provide "Confirm" or "Return" buttons.

[0134] Example of failure feedback interface:

[0135] "Task execution failed: Cannot find target control."

[0136] Two operation buttons: "Retry" and "Describe the task with voice again".

[0137] Front-end technology:

[0138] Implemented using the native Android UI framework.

[0139] 4. Error retry and task reissue mechanism

[0140] Implementation:

[0141] Define a unified interface to trigger the corresponding function when the UI button is clicked.

[0142] "Retry" interface: re-call the RPA execution module, pass in the current task description and parameters, wait for the return result and update the UI.

[0143] "Re-describe task with voice" interface: exit the current task execution state, switch to the voice input module interface, reset the task state, and wait for the user to enter a new voice command.

[0144] Based on the technical solution of the application, the following case scenarios are used to illustrate the implementation process of the application in practical application, and the specific application implementation scheme is as follows:

[0145] Mobile phone automation task implementation based on voice instructions:

[0146] The user's voice instruction is "please help me open WeChat and send a lucky draw red packet to the group chat group of small lovely group, with 10 red packets and a total amount of 10 yuan".

[0147] 1-1: When the user starts the App, click the "start voice input" button on the main interface, and the system requests and obtains the microphone usage permission.

[0148] 1-2: Use the built-in microphone of the mobile phone to collect user voice signals in real time through the AudioRecord class of the Android platform.

[0149] 1-3: The collected audio data is denoised, voice activity detection (VAD) and format conversion (converted to 16kHz, 16-bit single-channel PCM data).

[0150] 1-4: The pre-processed audio data is transmitted to the ASR module, and the cloud voice recognition service is called to convert the user's voice into text in real time. After conversion by the ASR module, the generated text is: "Please help me open WeChat and send a lucky draw red packet to the group chat group of small lovely group, with 10 red packets and a total amount of 10 yuan".

[0151] 1-5: After obtaining the text, the system sends the text to the natural language processing module (NLP) to analyze the semantics and identify the intent. In this example, the user's intent is identified as "send red packet"

[0152] 1-6: Extract key information such as application name, group chat object, red envelope type, red envelope quantity and amount from the text using regular expressions and named entity recognition techniques, and structure them into standard task descriptions.

[0153] 1-7: The system maps the predefined RPA operation flow, breaking down the task into specific instructions, including starting the specified application "WeChat", locating the specific chat window "Xiaokeai Group", simulating clicking the red envelope button, filling in the amount "10" and quantity "10" fields, and finally executing the sending operation.

[0154] 1-8: Call the UiAutomator-based API to parse and execute the above instructions one by one, simulating the user's real operation on the mobile phone screen, and realizing the execution of the automation task.

[0155] 1-9: When all RPA instructions are executed successfully, the system collects execution logs and time consumption data, and displays a prompt message "Task completed successfully! Time consumption 15 seconds" on the UI interface, and provides options to "Return to home page" or "View logs".

[0156] The innovations of the present application are as follows:

[0157] The present application integrates speech recognition (ASR), natural language processing (NLP) and mobile robot process automation (RPA) technology to build a mobile RPA automation system based on natural language analysis. The core innovation and technical protection points mainly lie in the following aspects:

[0158] 1. Voice-driven natural interaction and task triggering

[0159] Technical key: The system realizes real-time conversion of voice input by embedding a high-precision ASR engine (supporting local offline recognition and cloud recognition dual mode). Combined with adaptive noise suppression, voice activity detection (VAD) and other algorithms, it can accurately capture user voice in various complex environments.

[0160] Innovation: Breakthrough traditional manual operation mode, users can trigger complex tasks through natural voice instructions without touching the screen; voice input and real-time text feedback form a closed loop, improving user interaction experience and operation convenience.

[0161] 2. Intelligent semantic understanding and task decomposition

[0162] Technical key: The invention adopts a dual analysis strategy of local fast regular matching combined with cloud advanced NLP model (such as Transformer or BERT model). The system first quickly analyzes common instructions through predefined rules; for ambiguous or complex sentences, it calls the cloud semantic understanding service to accurately identify user intent and extract key entities (such as application name, group chat name, red packet type, quantity, amount, etc.).

[0163] Innovation: It realizes the automatic conversion from natural language to structured task description, can decompose complex tasks into several sub-tasks, and automatically generate the execution parameters of each sub-task. This module solves the problem of task flow hard coding and lack of flexibility in existing technology, providing intelligent and dynamic task planning and scheduling capabilities for automated operations.

[0164] 3. Fully automatically generate and execute RPA operation process

[0165] Technical key: Based on the parsed structured task description, the system constructs a predefined operation template library and automatically generates corresponding RPA operation instructions through mapping rules. Using UiAutomator, Appium and other underlying automation tools, the system realizes the full automatic execution of various interactive operations such as "click, slide, input" on mobile phones.

[0166] Innovation: It innovatively combines intelligent voice input with RPA technology, realizes the automatic generation and execution of executable code from natural language instructions, and completely reduces the technical threshold of operation configuration. This solution not only supports cross-application and cross-task automation process, but also has good scalability, which can be flexibly adjusted for different application scenarios.

[0167] 4. Full-process real-time feedback and exception handling mechanism

[0168] Technical key: The system designs a complete result monitoring, log collection and feedback processing module during RPA task execution. Whether the task is successful or failed, the system can display the operation results, execution time, detailed logs and error prompts through the user interface in a timely manner, and provide "retry" or "re-describe the task with voice" operation options.

[0169] Innovation: Through transparent operation process display and interaction design, not only enhances the user's trust in the system, but also makes the system have self-diagnosis and rapid recovery capabilities, improves the stability and user experience of the overall system.

[0170] 5. System integration and scalability

[0171] Technical key: The application integrates ASR, NLP, RPA and feedback modules organically through modular design, and builds a complete closed-loop automation system. The seamless collaboration between modules is realized through unified data interface and calling protocol, and the interface is reserved for subsequent expansion (such as supporting more languages, more complex task processing and multi-platform compatibility).

[0172] Innovation: The system not only realizes breakthrough in a single technical level, but also realizes high integration and flexible expansion in the overall architecture, meeting the needs of different users and scenarios. This modular and plug-in design concept greatly improves the commercial application prospects and competitiveness of the system.

[0173] In summary, in the implementation of the natural speech driven mobile phone automation operation system, through the organic integration of voice recognition, intelligent semantic analysis, automatic task decomposition and RPA instruction generation, as well as real-time feedback and fault tolerance mechanism in the whole process, the operation complexity and technical threshold are greatly reduced, and the efficient and stable execution of automatic tasks is realized.

[0174] The application can realize the analysis and intention recognition of complex voice instructions, which requires deep learning and model training; The application can realize the task decomposition of complex content, which is different from Siri and other products in the market, which can only understand simple task instructions; The application can perform automation on the mobile phone end, and RPA technology is widely used on the PC end, but it is less used on the mobile phone end; The application has a friendly UI feedback, which is convenient for users to adjust instructions and locate problems.

[0175] The above only describes the preferred embodiments and principles of the application in detail, and for ordinary skilled persons in the art, according to the idea provided by the application, there will be changes in specific implementation, and these changes should be regarded as the protection scope of the application.

Claims

1. A mobile phone RPA automation system based on natural language parsing, characterized in that, Comprise: a voice input interface module for providing an interactive interface for a user while converting the voice input by the user into text; a natural language processing module for pre-processing, intent classification, entity extraction, task decomposition and generating a structured task description of the text converted by the voice input interface module; an RPA module for generating a corresponding RPA instruction list according to the structured task description by using mapping rules, while calling an RPA execution engine to sequentially analyze the RPA instructions and simulating user operations on a mobile phone to complete a task through a UI automation tool; a feedback display interface module for providing feedback information according to the completion of the task and allowing the user to select to continue operating or re-initiate the task; In the natural language processing module, the intent classification refers to identifying the operation intention of the user, and the specific process is as follows: Rule matching: quickly identify common intentions by using predefined keywords and regular expressions; Deep learning: use pre-trained models for semantic understanding to classify complex or ambiguous expressions; In the natural language processing module, the entity extraction and task decomposition refer to extracting key information from the text and decomposing complex tasks into subtasks, and the specific process is as follows: Regular expression extraction: use regular expression matching to extract numbers, units and specific words; Named Entity Recognition (NER): use existing Chinese NER models to extract entity information; Task decomposition rules: based on the intention, logically combine the extracted entity information to form a task step sequence; In the natural language processing module, the generation of a structured task description refers to combining the intention and the extracted entity information into structured data for subsequent mapping to specific RPA instructions; In the RPA module, the design of the mapping rules is as follows: Establish a mapping table to correspond each task intention and parameter to a specific sequence of RPA operation instructions; The RPA module comprises: an instruction generation module for converting the structured task description into an RPA instruction list; an RPA execution module for receiving the RPA instruction list and sequentially calling underlying RPA execution tools to simulate user operations; the specific implementation process is as follows: Convert the generated instructions into corresponding function calls or commands and execute them through an Application Programming Interface (API).

2. The natural language parsing based mobile RPA automation system of claim 1, wherein, The voice input interface module comprises a voice collection module, an audio preprocessing module, an automatic speech recognition module, and a result display and feedback module; The voice collection module is used to record audio using audio software; The audio preprocessing module is used to perform noise reduction and echo cancellation, speech activity detection, and format conversion processing on the recorded audio data to obtain pre-processed audio data; The automatic speech recognition module uses local offline recognition or cloud recognition; after receiving the audio data, the cloud or local automatic speech recognition module performs speech recognition using a deep learning model, outputs the corresponding text, and returns the output text to the client application; The result display and feedback module is used to display the recognized text, user feedback, and real-time display effects. The display of the recognized text refers to displaying the returned text on the user interface, allowing the user to confirm, edit or directly proceed to subsequent processing; the user feedback refers to providing error prompts and retry functions, allowing the user to re-enter the voice or manually correct if the recognition result does not meet the user's requirements; the real-time display effect refers to using a progress bar or waveform chart to provide real-time feedback on the recording status, allowing the user to understand the recording and recognition process.

3. The natural language parsing based mobile RPA automation system of claim 2, wherein, In the audio preprocessing module, the noise reduction and echo cancellation refer to using a noise reduction method based on fast Fourier transform (FFT) to perform noise reduction processing on the recorded data; The voice activity detection refers to using a voice activity detection method to detect whether there is voice in the audio stream, and if so, effectively intercepting the voice segment; The format conversion refers to converting the original PCM data into the format required by the speech recognition engine.

4. The natural language parsing based mobile RPA automation system of claim 3, wherein, In the automatic speech recognition module, if cloud recognition is used, the preprocessed audio data is packaged into a request and transmitted to the cloud API; the request includes authentication information, audio format and sampling rate.

5. The natural language parsing based mobile RPA automation system of claim 1, wherein, The feedback display interface module includes; The result monitoring and log collection module is used to report the operation results and store the result data using a log collector during the execution of the RPA execution module through internal events or callback interfaces, and deliver them to the feedback processing module; The feedback processing module is used to determine the status field in the received return data; if the status field is determined to be success, a success message is formatted and output; If the status field is determined to be failure, the error information is extracted and a recommended operation prompt is attached; The user interaction interface module includes a feedback information display area, a progress bar and operation buttons.

Images