Voice interaction method and device, and storage medium

Abstract

The invention discloses a voice interaction method and device and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining a target voice instruction of a user, and carrying out the analysis of the target voice instruction according to a preset target large language model, and obtaining a control intention; determining a position sequence of a target interface element corresponding to the control intention according to the control intention and a UI interface corresponding to a controlled device; determining a trigger action sequence according to the control intention; the controlled device is controlled to respond to the target voice instruction based on the position sequence and the trigger action sequence, and the quantity of component elements of the position sequence is equal to that of component elements of the trigger action sequence. Interaction with the user interface can be automatically completed according to the voice instruction input by the user, and the interaction efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a voice interaction method, device, and storage medium. Background Art

[0002] With the development of mobile internet and the Internet of Things (IoT), users are increasingly interacting with mobile devices (such as phones and tablets running Android, iOS, Linux, Windows, Sailfish, and other operating systems) and new retail devices (such as vending machines). To improve user experience and automation, simulating user operations through voice commands is becoming increasingly common.

[0003] Currently, user voice commands are usually bound to specific device operations or application functions. The device operation location information (such as button position, text box position, etc.) corresponding to the user voice command is hard-coded in the script. When the device receives the user voice command, the script will parse the command and simulate the user to perform the corresponding operation based on the preset location information. However, the hard-coded location information often depends on the specific device and application version. When the device or application is updated, the original hard-coded location information may change, causing the hard-coded script to become invalid and unable to accurately simulate the user operation.

[0004] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a voice interaction method, device and storage medium, aiming to solve the technical problem of how to automatically complete the interaction with the user interface based on the voice instructions input by the user.

[0006] To achieve the above objectives, the present application proposes a voice interaction method, which includes:

[0007] Obtaining the user's target voice command and parsing the target voice command according to a preset target large language model to obtain the control intention;

[0008] Determining a position sequence of target interface elements corresponding to the control intent based on the control intent and the UI interface corresponding to the controlled device; and

[0009] Determining a trigger action sequence according to the control intention;

[0010] The controlled device is controlled to respond to the target voice instruction based on the position sequence and the trigger action sequence, wherein the position sequence and the trigger action sequence have the same number of components.

[0011] In one embodiment, the step of obtaining a target voice instruction from a user and parsing the target voice instruction according to a preset target large language model to obtain a control intention includes:

[0012] When the user inputs the wake-up word, the target voice command is obtained;

[0013] Converting the target voice command into target text according to a preset voice recognition engine;

[0014] Calling the target large language model, parsing the target text according to the target large language model, and obtaining an intent distribution;

[0015] Matching is performed based on the probability information in the intention distribution and the preset intention label, and the control intention is determined based on the matching result.

[0016] In one embodiment, the step of matching the probability information in the intention distribution with a preset intention label and determining the control intention based on the matching result includes:

[0017] Obtaining the highest probability of the intent label in the intent distribution, and comparing the highest probability with a preset probability threshold;

[0018] If the highest probability is greater than or equal to the probability threshold, determining the intention label corresponding to the highest probability as the control intention;

[0019] If the highest probability is less than the probability threshold, it is determined that the target voice instruction is ambiguous, and a re-input prompt is output.

[0020] In one embodiment, before the step of acquiring the user's target voice instruction and parsing the target voice instruction according to a preset target large language model to obtain the control intention, the following steps are included:

[0021] Detecting user operation behaviors and obtaining operation data and instruction information that triggers the operation behaviors;

[0022] Training a preset instruction action model according to the operation data and the instruction information triggering the operation behavior to obtain a target instruction action model;

[0023] The target instruction action model is called to generate a control instruction to obtain a control instruction set.

[0024] In one embodiment, before the step of acquiring the user's target voice instruction and parsing the target voice instruction according to a preset target large language model to obtain the control intention, the method further includes:

[0025] Get UI screenshots for multiple devices;

[0026] extracting element features of each element from the user interface screenshot according to an image recognition algorithm;

[0027] An element feature library is established according to the element features, and a preset image matching algorithm is trained according to the element library to obtain a target image matching algorithm.

[0028] In one embodiment, the step of determining a position sequence of target interface elements corresponding to the control intent based on the control intent and the UI interface corresponding to the controlled device includes:

[0029] Acquire feature information of the target interface element in the control intention, and determine a feature type corresponding to the feature information according to the feature information;

[0030] Determine a corresponding preset positioning algorithm according to the feature type;

[0031] The feature information is matched according to the preset positioning algorithm to determine the target interface element.

[0032] In one embodiment, the step of controlling the controlled device to respond to the target voice command based on the position sequence and the trigger action sequence includes:

[0033] Matching the control intention with the control instruction set to obtain a target instruction template;

[0034] Fill the position sequence and the trigger action sequence into the corresponding placeholder positions in the target instruction template to obtain the target control instruction;

[0035] The target control instruction is called and the target voice instruction is responded to.

[0036] In one embodiment, after the step of controlling the controlled device to respond to the target voice command based on the position sequence and the trigger action sequence, the method further includes:

[0037] Determining whether the target voice command is executed successfully;

[0038] If the execution is successful, a successful voice prompt will be generated;

[0039] If the execution fails, the preset instruction set is matched again to obtain and execute the second target control instruction.

[0040] In addition, to achieve the above-mentioned purpose, the present application also proposes a voice interaction device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the voice interaction method described above.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the voice interaction method described above are implemented.

[0042] The present application provides a voice interaction method, which can more accurately understand the user's intention by parsing the voice instructions issued by the user through a large language model. According to the control intention and the UI interface corresponding to the controlled device, the position sequence of the target interface element corresponding to the control intention is determined; and the trigger action sequence is determined according to the control intention; the trigger action in the user's voice instruction and the interface element corresponding to the trigger action are dynamically obtained through the control intention, rather than through hard-coded element locators (such as XPath, CSS selectors) to determine the position of the interface element, which can be flexibly adapted to different devices and different application versions. By responding to the target voice instruction through the position sequence and the trigger action sequence, the user's operation is simulated, and the interaction with the user interface can be automatically completed according to the voice instruction input by the user, thereby improving the efficiency of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 A flowchart of the first embodiment of the voice interaction method of this application is provided;

[0046] Figure 2 A detailed flow chart of the first embodiment of the voice interaction method of this application is provided;

[0047] Figure 3 Another detailed flow chart provided for the first embodiment of the voice interaction method of this application;

[0048] Figure 4This is an example diagram provided for the first embodiment of the voice interaction method of this application;

[0049] Figure 5 A flowchart of the second embodiment of the voice interaction method of this application is provided;

[0050] Figure 6 A flowchart of the fourth embodiment of the voice interaction method of this application is provided;

[0051] Figure 7 A flowchart of the fifth embodiment of the voice interaction method of this application is provided;

[0052] Figure 8 This is a schematic diagram of the device structure of the hardware operating environment involved in the voice interaction method in the embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: obtaining the user's target voice command, and parsing the target voice command according to a preset target large language model to obtain a control intention; determining the position sequence of the target interface elements corresponding to the control intention according to the control intention and the UI interface corresponding to the controlled device; and determining a trigger action sequence according to the control intention; controlling the controlled device to respond to the target voice command based on the position sequence and the trigger action sequence, wherein the number of constituent elements of the position sequence and the trigger action sequence is equal.

[0056] With the rapid development of mobile internet and IoT technologies, users are interacting more frequently with mobile devices (such as phones and tablets) and new retail devices (such as vending machines). To improve user experience and automation, simulating user operations through voice commands is becoming increasingly common.

[0057] Currently, user voice commands are usually bound to specific device operations or application functions. The device operation location information (such as button position, text box position, etc.) corresponding to the user voice command is hard-coded in the script. When the device receives the user voice command, the script will parse the command and simulate the user to perform the corresponding operation based on the preset location information. However, the hard-coded location information often depends on the specific device and application version. When the device or application is updated, the original hard-coded location information may change, causing the hard-coded script to become invalid and unable to accurately simulate the user operation.

[0058] The present application provides a voice interaction method, which can more accurately understand the user's intention by parsing the voice instructions issued by the user through a large language model. According to the control intention and the UI interface corresponding to the controlled device, the position sequence of the target interface element corresponding to the control intention is determined; and the trigger action sequence is determined according to the control intention; the trigger action in the user's voice instruction and the interface element corresponding to the trigger action are dynamically obtained through the control intention, rather than through hard-coded element locators (such as XPath, CSS selectors) to determine the position of the interface element, which can be flexibly adapted to different devices and different application versions. By responding to the target voice instruction through the position sequence and the trigger action sequence, the user's operation is simulated, and the interaction with the user interface can be automatically completed according to the voice instruction input by the user, thereby improving the efficiency of the interaction.

[0059] It should be noted that the execution subject of this embodiment can be a computing service device with network communication and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or an electronic device capable of implementing the above functions, a program process abnormality monitoring device, etc. The following uses a voice interaction device as an example to illustrate this embodiment and the following embodiments.

[0060] Based on this, the present application embodiment provides a voice interaction method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the voice interaction method of the present application.

[0061] In this embodiment, the voice interaction method includes steps S100 to S400:

[0062] Step S100: Acquire the user's target voice instruction, and parse the target voice instruction according to a preset target large language model to obtain the control intention.

[0063] It should be noted that large language models refer to large neural network models based on deep learning, which are used to process and understand natural language text. They are able to capture the complexity and diversity of language. Intent information refers to the user's desired goal or expected outcome.

[0064] In this embodiment, the user issues a voice command through the microphone of a device (such as a mobile phone, smart home device, etc.), and the device captures the user's voice input and converts it into a digital signal for storage and processing. The captured voice signal is converted into text form using speech recognition technology, and the recognized text is preprocessed, such as noise removal, punctuation normalization, and case conversion. The preprocessed text is input into a preset target large language model, and the large language model uses its own language knowledge and reasoning ability to deeply analyze the text to identify the user's intention. By interacting through voice commands, the user does not need to manually input or click on the screen, thereby improving the efficiency of interaction. The analysis of the large language model can more accurately understand the user's intention, reducing the possibility of misunderstanding and misoperation.

[0065] In one possible implementation, please refer to Figure 2 , step S100 may further include steps S110 to S140:

[0066] Step S110: After monitoring the user inputting the wake-up word, the target voice command is obtained.

[0067] It should be noted that the wake-up word refers to a short word or phrase used to activate voice interaction between the device and the user. Users can set the wake-up word according to their personal habits.

[0068] It's important to note that an agent is an entity that can act autonomously, perceive its environment, make decisions, and potentially interact with other agents. An agent can make decisions based on its internal state, environmental input, and pre-set goals or rules. It perceives information about its environment through various sensors (such as microphones and cameras), and based on this information and its internal knowledge base, it makes decisions and takes appropriate actions.

[0069] In this embodiment, before the intelligent agent is awakened, the device continuously monitors the sounds in the environment and captures the sound signals in the surrounding environment through the microphone. When the energy of the sound signal exceeds a certain threshold or meets specific frequency characteristics, it is judged as a possible wake-up word or wake-up sound. And extract the characteristic information in the sound signal, such as short-time energy, zero-crossing rate, and Mel-frequency cepstral coefficients. After the feature extraction is completed, the extracted sound signal features are compared with the pre-stored wake-up word features. If the match is successful, it is considered that the user has entered the wake-up word, and the device will further activate the intelligent agent in the device and wait for the user to issue subsequent instructions.

[0070] Optionally, when comparing the extracted sound signal features with pre-stored wake-up word features, the distance between feature vectors can be calculated (such as Euclidean distance, cosine similarity, etc.), or classification judgment can be performed through a classifier (such as a support vector machine, neural network, etc.).

[0071] Step S120: converting the target voice instruction into target text according to a preset voice recognition engine.

[0072] In this embodiment, the intelligent agent calls a preset speech recognition engine to process the captured speech signal. The preset speech recognition engine can be an independent software module or a part integrated into the intelligent agent or device operating system. The speech recognition engine extracts features from the speech signal and converts it into a series of mathematical features that can represent the speech content. These features may include spectrum, pitch, speaking speed, etc. The extracted features are then matched with the speech model pre-stored in the speech recognition engine. When the best matching speech model is found, the speech recognition engine will output the corresponding text to obtain the target file. The intelligent agent will pass the converted target text to the target large language model.

[0073] Optionally, the agent can also perform post-processing on the output text, such as error correction and formatting, to ensure its accuracy and readability. For example, the agent can utilize a built-in spell-checking algorithm or call a specialized spell-checking library to perform spelling and language checking on the output text, identifying and correcting spelling errors and ensuring grammatical correctness. The agent can also apply preset text templates to format the text to ensure consistency and standardization, thereby improving text processing efficiency.

[0074] Step S130 : calling the target large language model, parsing the target text according to the target large language model, and obtaining an intent distribution.

[0075] Step S140 , matching is performed according to the probability information in the intention distribution and the preset intention label, and the control intention is determined according to the matching result.

[0076] In this embodiment, the intelligent agent inputs the text information into the preset target large language model for parsing to understand the user's needs and uses. After the text information is input into the target large language model, the target large language model encodes the input text and generates a series of outputs, including the embedding representation of each word, possible entity labels, intent distribution, etc. Entity labels are usually specific names of people, places, times, etc. mentioned in the text. The intent distribution is usually a probability vector, where each probability vector corresponds to a preset intent label, indicating the probability that the model believes that the input text belongs to the corresponding intent. Finally, the intent distribution output by the model is matched with the preset intent label to determine the control intent, and the control intent label is combined with entity recognition to generate a complete user request representation.

[0077] In one possible implementation, please refer to Figure 3 , step S140 may further include steps S141 to S143:

[0078] Step S141: Obtain the highest probability of the intent label in the intent distribution, and compare the highest probability with a preset probability threshold.

[0079] Step S142: If the highest probability is greater than or equal to the probability threshold, the intention label corresponding to the highest probability is determined as the control intention.

[0080] Step S143: If the highest probability is less than the probability threshold, it is determined that the target voice instruction is unclear, and a re-enter prompt is output.

[0081] In this implementation, a probability threshold can also be set. Only when the highest probability exceeds this threshold is the model's output considered reliable. If the highest probability is lower than the threshold, the judgment can be rejected or the user can be asked to re-enter. By setting a probability threshold, the impact of noise or ambiguity can be reduced, increasing the accuracy of judgments.

[0082] Optionally, when the target voice instruction input by the user contains multiple intentions at the same time, a threshold can be set in advance, and all labels with probabilities exceeding the threshold are selected as the user's intentions.

[0083] For example, in a smart home application, possible intent labels include turning on the lights (Intent A), turning off the air conditioner (Intent B), setting the alarm (Intent C), and checking the weather (Intent D). For each input text, the target language model outputs a vector containing the probabilities of these four intents: Intent A (turning on the lights): 0.85, Intent B (turning off the air conditioner): 0.05, Intent C (setting the alarm): 0.08, and Intent D (checking the weather): 0.02. Assuming we set a threshold of 0.7, we only consider the intent output by the model to be reliable when the highest probability exceeds this threshold. The highest probability of 0.85 exceeds the threshold of 0.7, so the intent output by the model (Intent A) is considered reliable.

[0084] Step S200 : determining a position sequence of target interface elements corresponding to the control intention according to the control intention and the UI interface corresponding to the controlled device.

[0085] Step S300: determining a triggering action sequence according to the control intention.

[0086] It's important to note that a UI (User Interface) refers to the part of a computer system or application that users interact with, including elements like graphics, text, buttons, and menus, as well as how these elements respond to user input. Interface elements refer to, but are not limited to, the representation of UI elements within an application, including buttons and text boxes. Interface elements can also be XML descriptions of the user interface, web-based DOM (Document Object Model), or the pixel positions of widgets on the screen.

[0087] In this embodiment, a control intent may include multiple sub-intentions, and the position of the target interface element corresponding to each sub-intention and the trigger action are combined into a sequence in the order of execution. By parsing the control intent, the target interface element and the execution action in the control intent can be determined. The target page element can be a button, text box, etc., and the trigger action can be click, long press, slide, input text, etc. The position of the target interface element in the UI interface can include the coordinates of the element, hierarchical information or other location information that can uniquely identify the element. By determining the position sequence and trigger action sequence of the target interface element corresponding to the control intent, a basis is provided for subsequent automated control operations to guide the controlled device to perform corresponding operations.

[0088] In this embodiment, when the interface elements are the pixel locations of widgets on the screen, screenshots of the application or webpage are captured and analyzed using image recognition technology integrated into an automated testing framework, such as libraries like OpenCV and PyAutoGUI. The screenshots are then used to identify specific interface elements in the screenshots, such as buttons, text boxes, and icons, and to determine their location, size, shape, and other characteristics. When the interface elements are XML descriptions of user interfaces or web-based DOM, the hierarchical structure, attributes, and location of the interface elements can be determined by parsing the XML or HTML files or traversing the DOM tree.

[0089] Step S400 : Controlling the controlled device to respond to the target voice instruction based on the position sequence and the trigger action sequence, wherein the position sequence and the trigger action sequence have the same number of constituent elements.

[0090] In this embodiment, the number of elements in the position sequence and the trigger action sequence should be equal, and each position corresponds to a trigger action. The position can be a coordinate on the screen, a button identifier, etc.; the trigger action can be a click, a slide, etc.

[0091] In a feasible implementation, step S400 includes steps S410 to S430:

[0092] Step S410, matching the control intention with the control instruction set to obtain a target instruction template;

[0093] Step S420, filling the position sequence and the trigger action sequence into the corresponding placeholder positions in the target instruction template to obtain the target control instruction;

[0094] Step S430: calling the target control instruction and responding to the target voice instruction.

[0095] In this embodiment, the control intent is further refined or standardized to ensure that it can accurately match the instructions in the preset instruction set. The control instruction set is a database or knowledge base containing various possible control instructions. These instructions are usually associated with the functions, operations or tasks of the device. When performing instruction matching, various technologies such as keyword matching, semantic similarity calculation, and context understanding can be used. The control instruction set should contain a series of predefined instruction templates, each template corresponding to a control intent. After finding the matching instruction template, the position sequence (such as application icons, buttons, etc.) and trigger action sequence (such as click, slide, etc.) that need to be operated are determined according to the control intent. In the matched target control instruction template, the corresponding placeholders are found (these placeholders are used to identify the position and action information that need to be dynamically filled). The position sequence and trigger action sequence are filled into these placeholders to generate the target control instruction. Use an execution engine (such as a UI automation framework, a script interpreter, etc.) to call the target control instruction, parse and execute the instructions and actions in the target control instruction, and simulate the user's operation.

[0096] Optionally, when refining intent information, you can define a clear intent hierarchy based on the application scenario and device capabilities. This structure can include high-level intents (such as "play music"), mid-level intents (such as "play music by a specific singer"), and low-level intents (such as "play Jay Chou's latest album"). For complex intents, break them down into more specific and easier-to-handle sub-intents. For example, break down "query tomorrow's weather and set a reminder" into two sub-intents: "query tomorrow's weather" and "set a weather reminder."

[0097] Optionally, the vocabulary in intent information can be standardized, such as removing synonyms, abbreviations, and colloquial expressions, to ensure the accuracy and consistency of the intent information. The intent format can also be standardized, with a standard intent format defined, including intent type, parameters, and contextual information. This ensures that the intent information accurately matches the instructions in the preset instruction set.

[0098] For example, Figure 4 As shown in the figure, when the user issues the target voice command "Use AutoNavi Map to navigate home," the location sequences are the location of the AutoNavi Map app icon, the location of the "Home" button on the AutoNavi Map homepage, and the location of the "Start" button in the navigation interface. Each location sequence corresponds to a click action.

[0099] For example, in an enterprise IM (Instant Messaging) application, such as WeChat, when a user needs to interact with a contact, he or she enters the contact's keywords and uses a fuzzy search and matching algorithm to match contacts that meet the user's needs based on substring matching, pinyin matching, or near-string matching. The similarity calculated based on the fuzzy search and matching algorithm is obtained, and the matching results are sorted in descending order of similarity, and the matching result with the highest similarity is selected as the contact that meets the user's needs.

[0100] In this embodiment, the automated matching process reduces the potential for human error by automatically matching and identifying target elements and accurately locating them based on feature information, rather than relying on potentially inaccurate visual recognition or manual positioning. The predefined instruction code contains common structure and logic, while specific location and action information is provided through dynamic fill-in. This avoids the need for repetitive hard-coding of large amounts of location information, thereby reducing code size and memory usage.

[0101] The present application provides a voice interaction method, which can more accurately understand the user's intention by parsing the voice instructions issued by the user through a large language model. According to the control intention and the UI interface corresponding to the controlled device, the position sequence of the target interface element corresponding to the control intention is determined; and the trigger action sequence is determined according to the control intention; the trigger action in the user's voice instruction and the interface element corresponding to the trigger action are dynamically obtained through the control intention, rather than through hard-coded element locators (such as XPath, CSS selectors) to determine the position of the interface element, which can be flexibly adapted to different devices and different application versions. By responding to the target voice instruction through the position sequence and the trigger action sequence, the user's operation is simulated, and the interaction with the user interface can be automatically completed according to the voice instruction input by the user, thereby improving the efficiency of the interaction.

[0102] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 5 , before step S100, steps S01 to S03 are also included:

[0103] Step S01: detecting the user's operation behavior, obtaining operation data and instruction information triggering the operation behavior;

[0104] In this embodiment, user behavior data is collected in different scenarios through logging, screen capture, and sensor data. Different usage scenarios are distinguished, such as browsing the web, watching videos, and editing documents. After data collection is complete, data preprocessing is performed. Data cleaning is performed to remove invalid, redundant, or erroneous data. Data standardization is performed to unify data from different sources and formats into an analyzable format.

[0105] In this embodiment, by capturing user operational behavior, basic behavioral data, contextual data, session data, device data, and interaction frequency data can be obtained. Basic behavioral data may include the identifier or name of the UI element (such as a button, link, or image) clicked by the user, the timestamp of the click (i.e., the specific time the user performed the click operation), and the location information of the click, such as the coordinates on the screen. The starting and ending positions of the slide, the direction and distance of the slide, and the timestamp of the slide operation. The text content entered by the user in the input box, the timestamp of the input operation, and the identifier or name of the input box. Contextual data may include the identifier or name of the page or interface the user is currently on, the timestamp of the page load, and other UI elements on the page and their status (such as whether a button was clicked or whether the input box is empty). Session data may include the start and end time of the user session, the sequence of pages or interfaces visited by the user during the session, and the sequence of operational behaviors performed by the user during the session. Device data may include the type of device used by the user (such as a mobile phone, tablet, or computer), the device's operating system and version, and the device's screen size and resolution. Interaction frequency data may include the frequency with which the user performs a certain operational behavior within a specific time period and the frequency with which the user interacts with specific UI elements. By analyzing the user's stay time and operation behavior on different pages or interfaces, we can infer the user's preferences and interests.

[0106] Step S02: training a preset instruction action model according to the operation data and the instruction information triggering the operation behavior to obtain a target instruction action model;

[0107] Step S03: calling the target instruction action model to generate control instructions to obtain a control instruction set.

[0108] It should be noted that the command action model refers to a model that can generate a reasonable control command script based on the input operation data and command information. This model can be a large language model or other types of artificial intelligence models, such as neural network models and rule-based models.

[0109] In this embodiment, the preprocessed data is divided into a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used to adjust the model parameters and monitor the training process, and the test set is used to evaluate the model performance. Label the data set. The label can be the classification of the operation behavior (such as click, input, etc.), the type of operation object (such as button, text box, etc.), or the type of trigger instruction (such as voice instruction, shortcut key, etc.). The preset instruction action model is trained using the training data set so that the model can generate reasonable control instructions based on the input operation data and instruction information. The model performance is evaluated on the validation set, and the model parameters and architecture are adjusted according to the evaluation results, including adjusting the learning rate, increasing the number of training rounds, improving the feature extraction method, etc. The model performance is verified on the test set. Ensure that the model can also perform well on unseen data.

[0110] In this embodiment, during the long-term learning process, the model can gradually accumulate the user's preferences and behavioral habits, and can automatically generate reasonable control instructions based on the user's instruction information, thereby improving operational efficiency and accuracy.

[0111] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to above and will not be described in detail. On this basis, before step S100, the following steps may be further included:

[0112] Get UI screenshots for multiple devices;

[0113] extracting element features of each element from the user interface screenshot according to an image recognition algorithm;

[0114] An element feature library is established according to the element features, and a preset image matching algorithm is trained according to the element library to obtain a target image matching algorithm.

[0115] In this embodiment, screenshots of user interfaces of devices of different brands, models and operating systems are collected, and the screenshots are preprocessed as necessary, such as scaling, cropping, denoising, etc., to improve the accuracy of the image recognition algorithm. Image recognition algorithms are selected, such as convolutional neural networks (CNN), edge detection, template matching, etc. Element features are extracted according to the image recognition algorithm, and features of each interface element, such as shape, color, texture, position, etc. are extracted. An element feature library is constructed, and a database containing various interface element features is constructed based on the extracted element features. The element feature library is used to train the preset image matching algorithm so that it can accurately match and identify elements in the user interface. After training and optimization, a target image matching algorithm that can accurately match and identify user interface elements is obtained.

[0116] In this embodiment, through training, the target image matching algorithm can accurately match and identify elements in different device user interfaces, improving the efficiency of test automation and verification. It can adapt to devices of different brands, models, and operating systems, as well as different user interface layouts and elements.

[0117] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 6 , step S200 may include steps S210 to S230:

[0118] Step S210, acquiring feature information of the target interface element in the control intention, and determining a feature type corresponding to the feature information according to the feature information;

[0119] Step S220, determining a corresponding preset positioning algorithm according to the feature type;

[0120] Step S230: Match the feature information according to the preset positioning algorithm to determine the target interface element.

[0121] In this embodiment, the control intent describes the target interface element to be located and its feature information. The feature information includes the target element's attributes (such as ID and class name), image template, text content extracted by OCR, and relative position description. Based on the parsed target element feature information, the most appropriate positioning algorithm is selected from a preset positioning algorithm library.

[0122] In the present embodiment, a variety of positioning algorithms are predefined, such as image matching algorithms, positioning algorithms based on layout or relative coordinates, OCR text recognition algorithms, and attribute matching algorithms using an automated testing framework (such as the find_element_by_* method of the Selenium framework, the uiautomator2 framework). If the target element feature type is an image recognition feature (such as the shape, color, texture, etc. of the element), an image matching algorithm is selected to compare the similarity between the actual image of the element and the image template. If the target element feature type is layout information or relative coordinates, matching is performed according to the positional relationship of the element. If the target element feature type is an image comprising text, then according to the OCR text recognition algorithm (Optical Character Recognition, optical character recognition), the text content in the identification element is compared with the target text. If the target element feature type is an attribute type (such as ID, class name), the attribute matching algorithm using the automated testing framework is used to match.

[0123] Optionally, when there are multiple matching elements, pre-defined rules can be applied to further determine the final positioning result. Pre-set custom rules can be defined based on specific business logic or user preferences. For example: commonly used elements can be filtered according to the user's usage habits. For example, in automated office software, users may often use certain specific functions or commands. By recording the user's usage habits, the automation process can be optimized to improve the user experience. It is also possible to filter based on user needs. When there are multiple matching UI elements, the user can choose multiple selection or single selection mode according to their needs. For example, in automated file processing, the user may need to select multiple files for batch processing, in which case it can be set to multiple selection mode; in some cases, the user may only need to select one file for processing, in which case it can be set to single selection mode.

[0124] Based on the first embodiment of the present application, in the fifth embodiment of the present application, the same or similar contents as those in the above embodiment 1 can be referred to the above introduction and will not be described in detail later. Figure 7 , after step S400, steps S500 to S700 may be further included:

[0125] Step S500, determining whether the target voice command is executed successfully;

[0126] Step S600: If the execution is successful, a success voice prompt is generated;

[0127] Step S700: If the execution fails, rematch with the preset instruction set to obtain and execute the second target control instruction.

[0128] In this embodiment, after the target control instruction is sent to the corresponding application for execution, an execution result is generated. The execution result includes a success / failure flag, returned data, an error code, etc. The success / failure flag is used to determine whether the instruction was successfully executed. If the instruction is successfully executed, a success voice prompt is generated and played to inform the user that the operation is complete. If the instruction fails to execute, the target voice instruction or parsed text instruction is re-matched with the preset instruction set to obtain and execute a second target control instruction. The second target control instruction can be a revised instruction or a backup instruction.

[0129] Optionally, if the second target control instruction still fails to execute, a retry threshold can be set. When the number of retries is less than the retry threshold, a retry is performed; when the number of retries is greater than or equal to the retry threshold, an error prompt is issued to the user.

[0130] In this embodiment, when the target voice command is successfully executed, the user is given accurate feedback in a timely manner through voice. When the command fails to execute, the command matching is repeated, thereby improving the efficiency of the automation process.

[0131] The present application provides a voice interaction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the voice interaction method in the above-mentioned embodiment one.

[0132] Reference below Figure 8 , which shows a schematic diagram of the structure of a voice interaction device suitable for implementing the embodiments of the present application. The voice interaction device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (portable Android devices), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 8 The voice interaction device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0133] like Figure 8 As shown, the voice interaction device may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the voice interaction device. Processing device 1001, ROM 1002, and RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input device 1007 including, for example, a touch screen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output device 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003 including, for example, a magnetic tape, hard disk, etc.; and communication device 1009. The communication device 1009 can allow the voice interaction device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a voice interaction device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0134] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0135] The voice interaction device provided in this application, employing the voice interaction method of the aforementioned embodiment, can solve the technical problem of automatically completing interaction with a user interface based on user input voice commands. Compared to the prior art, the beneficial effects of the voice interaction device provided in this application are the same as those of the voice interaction method provided in the aforementioned embodiment. Other technical features of this voice interaction device are the same as those disclosed in the aforementioned embodiment and are not further elaborated here.

[0136] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0137] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0138] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer program) stored thereon, and the computer-readable program instructions are used to execute the voice interaction method in the above-mentioned embodiment.

[0139] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0140] The computer-readable storage medium may be included in the voice interaction device, or may exist independently without being incorporated into the voice interaction device.

[0141] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the voice interaction device, the voice interaction device: obtains the user's target voice command, and parses the target voice command according to the preset target large language model to obtain the control intention; determines the position sequence of the target interface elements corresponding to the control intention according to the control intention and the UI interface corresponding to the controlled device; and determines the trigger action sequence according to the control intention; controls the controlled device to respond to the target voice command based on the position sequence and the trigger action sequence, wherein the number of constituent elements of the position sequence and the trigger action sequence is equal.

[0142] Computer program code for performing the operations of the present application may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0143] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0144] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0145] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., a computer program) for executing the above-mentioned voice interaction method. This computer-readable storage medium can solve the technical problem of how to automatically complete interaction with a user interface based on voice instructions input by a user. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the voice interaction method provided in the above-mentioned embodiment, and will not be elaborated here.

[0146] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A voice interaction method, characterized in that: The method includes: Acquire the user's target voice command, and parse the target voice command according to a preset target large language model to obtain the control intention; Determine, according to the control intention and the UI interface corresponding to the controlled device, a position sequence of target interface elements corresponding to the control intention; and Determine a trigger action sequence according to the control intention; The controlled device is controlled to respond to the target voice instruction based on the position sequence and the trigger action sequence, wherein the number of component elements of the position sequence and the trigger action sequence is equal.

2. The method according to claim 1, characterized in that The step of obtaining the user's target voice command and parsing the target voice command according to a preset target large language model to obtain the control intention includes: When the user inputs a wake-up word, the target voice command is obtained; Converting the target voice command into target text according to a preset voice recognition engine; Calling the target large language model, parsing the target text according to the target large language model, and obtaining intent distribution; Matching is performed based on the probability information in the intention distribution and the preset intention label, and the control intention is determined based on the matching result.

3. The method according to claim 2, characterized in that The step of matching the probability information in the intention distribution with the preset intention label and determining the control intention according to the matching result includes: Obtaining the highest probability of the intent label in the intent distribution, and comparing the highest probability with a preset probability threshold; If the highest probability is greater than or equal to the probability threshold, determining the intention label corresponding to the highest probability as the control intention; If the highest probability is less than the probability threshold, it is determined that the target voice instruction is ambiguous, and a re-input prompt is output.

4. The method according to claim 1, characterized in that Before the step of acquiring the target voice instruction of the user and parsing the target voice instruction according to a preset target large language model to obtain the control intention, the method includes: Detecting the user's operation behavior and obtaining operation data and instruction information that triggers the operation behavior; According to the operation data and the instruction information triggering the operation behavior, a preset instruction action model is trained to obtain a target instruction action model; The target instruction action model is called to generate control instructions to obtain a control instruction set.

5. The method according to claim 1, characterized in that Before the step of acquiring the target voice instruction of the user and parsing the target voice instruction according to the preset target large language model to obtain the control intention, the method further includes: Get UI screenshots for multiple devices; Extracting element features of each element from the user interface screenshot according to an image recognition algorithm; An element feature library is established according to the element features, and a preset image matching algorithm is trained according to the element library to obtain a target image matching algorithm.

6. The method according to claim 1, characterized in that The step of determining the position sequence of the target interface elements corresponding to the control intention according to the control intention and the UI interface corresponding to the controlled device comprises: Acquire feature information of the target interface element in the control intention, and determine a feature type corresponding to the feature information according to the feature information; Determine a corresponding preset positioning algorithm according to the feature type; The feature information is matched according to the preset positioning algorithm to determine the target interface element and the position sequence.

7. The method according to claim 1, characterized in that The step of controlling the controlled device to respond to the target voice command based on the position sequence and the trigger action sequence includes: Matching the control intention with the control instruction set to obtain a target instruction template; Fill the position sequence and the trigger action sequence into the corresponding placeholder positions in the target instruction template to obtain the target control instruction; The target control instruction is called, and the target voice instruction is responded to.

8. The method according to claim 1, characterized in that After the step of controlling the controlled device to respond to the target voice instruction based on the position sequence and the trigger action sequence, the method further includes: Determining whether the target voice command is executed successfully; If the execution is successful, a successful voice prompt will be generated; If the execution fails, the preset instruction set is matched again to obtain and execute the second target control instruction.

9. A voice interaction device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the voice interaction method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the voice interaction method according to any one of claims 1 to 8 are implemented.

Images