Voice interaction method, apparatus, computer device and computer-readable storage medium

Abstract

A voice interaction method, apparatus, computer device, and computer-readable storage medium, related to the voice control technical field, includes: receiving a user's voice request through a first interaction page of a target application (S110); matching semantic text data corresponding to the user's voice request with first widget property data to determine an interaction target widget corresponding to the semantic text data (S120); generating a widget operation command based on a widget operation type of the interaction target widget (S130); and executing the widget operation command to output a response content to the user's voice request (S140).

Description

[Technical Field]

[0001] This application claims priority to a Chinese patent application entitled "Voice interaction method, apparatus, computer device and computer-readable storage medium," filed on December 1, 2022, with application number 202211539085.1, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of voice control technology, and in particular to a voice interaction method, apparatus, computer device and computer-readable storage medium. [Background technology]

[0003] Voice interaction technology is widely applied to various intelligent terminals, allowing users to control the intelligent terminals in a contactless manner. Using voice interaction technology to enable users to control the software in the center console system of an automobile is of great practical significance in ensuring vehicle driving safety.

[0004] In the related art, when controlling a target application in an automobile center console system using voice interaction, it is necessary to additionally develop a voice interaction operation interface for the target application, which increases the development cost of the target application. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure aims to solve at least one of the technical problems existing in the prior art. To this end, a first objective of the present disclosure is to provide a voice interaction method that decouples user voice recognition and semantic analysis from specific target application widgets, thereby reducing the development cost of target applications in automobile center console systems and avoiding the loss of voice interaction control functions due to the inability of third-party target applications installed in the automobile center console system to connect to a voice operation interface.

[0006] A second objective of this disclosure is to provide a voice interaction device.

[0007] A third object of the present disclosure is to provide a computer device.

[0008] A fourth object of the present disclosure is to provide a computer-readable storage medium.

[0009] To achieve the above object, a first embodiment of the present disclosure provides a voice interaction method applied to an automobile center console system, including: receiving a user's voice request through a first interaction page of a target application, where a target control widget is displayed on the first interaction page, and the target control widget is a controllable widget determined in the widget displayed on the first interaction page of the target application based on a widget type in a widget-related configuration file; first widget property data is associated with the target control widget, and the first widget property data is generated based on widget identification information, widget text, and widget number of the target control widget when monitoring switching from a second interaction page to the first interaction page; matching semantic text data corresponding to the user's voice request in the first widget property data to determine an interaction target widget corresponding to the semantic text data; generating a widget operation command based on a widget operation type of the interaction target widget; and executing the widget operation command to output a response content to the user's voice request.

[0010] According to one embodiment of the present disclosure, when the widget of the target application is updated, the widget-related configuration file can be modified to generate a new widget-related configuration file, and the new widget-related configuration file is used to determine the target control widget in the interaction page of the target application after the update.

[0011] According to one embodiment of the present disclosure, second widget property data corresponding to a target control widget on the second interaction page is stored in a buffer space, and the voice interaction method further includes at least one of the following steps: if the second widget property data and the first widget property data do not match, clearing the second widget property data in the buffer space and storing the first widget property data in the buffer space; if the second widget property data and the first widget property data match, reserving the second widget property data in the buffer space and discarding the first widget property data.

[0012] According to one embodiment of the present disclosure, the first widget property data corresponding to the widget shown in the first interaction page is obtained by adopting at least one of the following methods: obtaining first widget property data corresponding to the widget shown in the first interaction page based on a message queue; if a popup does not appear in the first interaction page, obtaining first widget property data corresponding to widgets other than a progress type widget on the first interaction page by calling an auxiliary service function of an operating system; if a popup appears in the first interaction page, obtaining first widget property data corresponding to the widget shown in the first interaction page by an Aspect interception manner; and if the progress type widget exists on the first interaction page, obtaining first widget property data corresponding to the progress type widget on the first interaction page by an Aspect interception manner.

[0013] According to one embodiment of the present disclosure, a behavior analysis module and a widget operation module are associated with the target application, and the method of generating the first widget property data includes: when monitoring switching from a second interaction page to the first interaction page, generating the first widget property data through the widget operation module based on widget identification information, widget text, and widget number of the target control widget; and invoking a widget registration interface of the behavior analysis module to register the first widget property data in the behavior analysis module.

[0014] According to one embodiment of the present disclosure, registering the first widget property data to the behavior analysis module includes registering the first widget property data to the behavior analysis module based on widget-related rules predefined in a widget-related configuration file.

[0015] According to one embodiment of the present disclosure, before executing the widget operation command, the voice interaction method further includes a step of receiving a widget operation command sent by the behavior analysis module through a command sending interface of the widget operation module, and the step of executing the widget operation command includes a step of calling a widget operation interface of the widget operation module and executing the widget operation command on the interaction target widget.

[0016] According to one embodiment of the present disclosure, the first widget property data includes widget text of the widget, and the step of matching semantic text data corresponding to the user's voice request in the first widget property data and determining an interaction goal widget corresponding to the semantic text data includes the step of determining an interaction goal widget corresponding to the semantic text data based on a matching result between the semantic text data corresponding to the user's voice request and the widget text of the widget.

[0017] To achieve the above object, a second embodiment of the present disclosure provides a voice interaction device applied to an automobile center console system, comprising: a receiving module used to receive a user's voice request through a first interaction page of a target application, wherein a target control widget is displayed on the first interaction page, and the target control widget is a controllable widget determined in the widget displayed on the first interaction page of the target application according to a widget type in a widget-related configuration file; a first widget property data is associated with the target control widget; and the first widget property data is associated with a second interaction page. a receiving module for generating a widget operation command based on the widget identification information, widget text, and widget number of the target control widget when monitoring switching from the previous page to the first interaction page; a matching module for matching semantic text data corresponding to the user's voice request in the first widget property data to determine an interaction target widget corresponding to the semantic text data; a command generating module for generating a widget operation command based on a widget operation type of the interaction target widget; and a command executing module for executing the widget operation command to output a response content to the user's voice request.

[0018] To achieve the above object, a third embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, implements the steps of the method described in any one of the above embodiments.

[0019] To achieve the above object, a fourth embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored therein, the computer program implementing the steps of the method described in any one of the above embodiments when executed by a processor.

[0020] According to the embodiments provided in the present disclosure, by adopting a method of cooperating the auxiliary service function of the operating system with the Aspect interception method, it is possible to realize operation control for any target application installed in the automobile center console system, and the target application does not need to connect to a voice operation interface, which greatly reduces the development costs of the target application and avoids the loss of voice interaction function due to the inability of a third-party target application to connect to a voice operation interface. Widget operations of a target application can be added or modified through a widget-related configuration file without the need to modify the widget acquisition and widget operation code. By using rule filtering and a message queue buffer, the number and frequency of acquiring widgets on an interaction page can be controlled, which can avoid the frequent acquisition of data such as widget identification information due to frequent changes in the content of the interaction page, thereby ensuring the performance of voice interaction in the automobile center console system.

[0021] Additional aspects and advantages of the present disclosure will be set forth in part in the description that follows, and in part will be obvious from the description, or may be learned by practice of the present disclosure.

[0022] The above and / or additional aspects and advantages of the present disclosure will become apparent and be readily understood from the following description of the embodiments taken in conjunction with the drawings. [Brief explanation of the drawings]

[0023] [Figure 1a]FIG. 1 is a schematic diagram of an application scenario of the voice interaction method provided by one embodiment of the present specification; [Figure 1b] FIG. 1 is a schematic diagram of the process of a voice interaction method provided by one embodiment of the present specification. [Figure 2] FIG. 1 is a schematic diagram of the process of a voice interaction method provided by one embodiment of the present specification. [Figure 3] 1 is a schematic diagram of a first interaction page on which a pop-up provided by one embodiment of the present disclosure appears. [Figure 4] 1 is a schematic diagram of a process for obtaining first widget property data provided by an embodiment of the present specification; [Figure 5a] 1 is a schematic diagram of a page including a list widget provided by one embodiment of the present disclosure. [Figure 5b] FIG. 2 is a schematic diagram of the process of interaction between a widget operation module and a behavior analysis module provided by one embodiment of the present specification; [Figure 5c] FIG. 2 is a schematic diagram of an interaction process between a behavior analysis module and a widget operation module provided by one embodiment of the present specification; [Figure 6] FIG. 2 is a schematic diagram of the interaction process of a semantics analysis module, a behavior analysis module, and a widget operation module provided by one embodiment of the present specification; [Figure 7] 1 is a block diagram of a voice interaction device provided by an embodiment of the present specification; [Figure 8] FIG. 1 is a block diagram of a computer device provided by one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024] The following describes in detail the embodiments of the present disclosure, and examples of the embodiments are shown in the drawings, where the same or similar designations throughout refer to the same or similar elements, or elements having the same or similar functions. The embodiments described below with reference to the drawings are illustrative and are used only for the purpose of explaining the present disclosure, and should not be understood as limiting the present disclosure.

[0025]

[0003] With the development of intelligence and voice technology, voice interaction technology has been widely applied to various intelligent terminals, allowing users to control the intelligent terminals in a contactless manner. In the automotive industry, since users need to hold the steering wheel with both hands while driving a vehicle, there is a certain risk if users manually control the software in the vehicle center console display while driving. Therefore, it is of great practical significance to enable users to control the software in the automotive center console system through voice interaction technology, and voice interaction technology has attracted particular attention in the automotive industry.

[0026] Some automobile manufacturers have already conducted some research into voice interaction control technology. Related technologies typically use voice control software to enable voice interaction between users and target applications in the automobile's center console system. In an interaction scenario, the voice control software first continuously receives user-uttered speech through a voice module, analyzes the speech to obtain semantic text, and then analyzes the semantic text to obtain the user's desired behavioral operation. Finally, it invokes the target application's operation interface and implements the required functions in the interface. For example, when a navigation application is running, the user utters, "Search for Oriental Pearl." The voice module analyzes the received speech and determines that the user needs to search for the location of Oriental Pearl in the navigation application. Based on the analysis, the voice module invokes the navigation application's interface, pops up a search box, and the user can enter the search parameter "Oriental Pearl" in the search box and click the search button to obtain directions to Oriental Pearl. This method requires that the target application installed in the automobile's center console system be customizable and capable of additionally developing an interface for implementing voice interaction operations. When a new function is introduced in a target application, a new interface needs to be developed, which results in high development costs, lacks flexibility, and makes it impossible to realize voice operation and control for target applications that do not support customization. Related technologies also adopt Aspect-based widget acquisition and operation methods, which can realize the function of target application widget operation based on voice interaction at low cost, but require the target application to integrate an Aspect Jar package. If a third-party target application does not integrate the relevant Jar package, it cannot connect to the widget operation function based on voice interaction.

[0027] However, users expect the voice interaction function of the automobile center console system to enable any operation function on the automobile center console display to be controlled through voice interaction, that is, to realize the function of being able to speak anything that can be seen. Therefore, it is necessary to provide a voice interaction method, apparatus, computer device, and computer-readable storage medium, which adopts a mode in which the auxiliary service function of the operating system cooperates with the Aspect interception mode to obtain and operate widgets on the current interaction page, and realizes operation control for any target application installed in the automobile center console system, without the need for the target application to connect to a voice operation interface, which greatly reduces the development cost of the target application, and avoids the case where a third-party target application cannot connect to a voice operation interface and therefore does not support the voice interaction function. Through the widget-related configuration file, it is not necessary to modify the widget acquisition and widget operation code, and it is possible to add or modify the widget operation of the target application. Through the mode of rule filtering and message queue buffer, it is possible to control the number and frequency of widget acquisition on the current interaction page, and avoid the case where widget acquisition is too frequent due to the content of the current interaction page changing frequently, thereby ensuring the performance of voice interaction.

[0028] 1A is a schematic diagram of an application scenario of the voice interaction method, apparatus, computer device, and computer-readable storage medium provided herein. Taking the voice interaction method in an automobile center console system as an example, for a target application running on the automobile center console display, a widget operation module monitors the current interaction page of the target application. After monitoring whether the content of the current interaction page has changed, the widget operation module obtains the widget identification information, widget text, and widget number corresponding to the widget in the current interaction page after the change, and arranges these data according to the predefined widget-related rules in the widget-related arrangement file. The arranged content is assembled into widget property data, and a widget registration interface defined by a behavior analysis module is registered with the behavior analysis module for subsequent generation of corresponding widget operation commands.

[0029] When a user makes a voice request to the current interaction page, the semantics analysis module receives the user's voice request, converts the voice request into text, and performs semantic analysis on the voice request to obtain corresponding semantic text data, which is then sent to the behavior analysis module. The behavior analysis module receives the semantic text data, matches the text included in the semantic text data with the widget text in the widget property data corresponding to the widget on the current interaction page, finds the widget corresponding to the matching widget text, determines it as the interaction target widget, and identifies the widget operation type corresponding to the interaction target widget, and generates a widget operation command for the interaction target widget. The behavior analysis module sends the generated widget operation command to the widget operation module through a command sending interface. After receiving the widget operation command through the widget operation module, the widget operation interface is called to execute the corresponding widget operation command for the interaction target widget, and the execution result can be displayed on the automobile center console display. In this way, the decoupling of semantic analysis, behavioral analysis, and specific widget operation can be realized through the three modules of semantic analysis, behavioral analysis, and widget operation, which are divided according to the process based on voice interaction. Furthermore, the target application does not need to develop additional interfaces. After the target application is installed in the automobile center console system and configured according to the widget-related configuration file, it can realize the function of being able to speak everything that is visible.

[0030] In this example, the widget operation module monitors the interaction page, obtains data such as widget identification information corresponding to the widget, and executes corresponding widget operation commands for the widget based on the AccessibilityService+Aspect method, which can meet the operation needs of all types of widgets. At the same time, it controls the number of data such as widget identification information corresponding to the widget after monitoring changes in the content of the interaction page through the widget type in the widget-related configuration file, and controls the frequency of obtaining data such as widget identification information corresponding to the widget through the message queue method, thereby reducing the wear and tear on the automobile center console system when the content of the interaction page on the automobile center console display changes too frequently.

[0031] The embodiment of the present specification provides a voice interaction method applied to an automobile center console system, and referring to FIG. 1b, the method may include the following steps:

[0032] S110: receiving a user's voice request through a first interaction page of a target application, where a target control widget is displayed on the first interaction page, the target control widget is a controllable widget determined in the widget displayed on the first interaction page of the target application based on the widget type in the widget-related configuration file, and first widget property data is associated with the target control widget, and the first widget property data is generated based on the widget identification information, widget text, and widget number of the target control widget when monitoring switching from a second interaction page to the first interaction page.

[0033] Here, the first interaction page is the current interaction page when the target application in the automobile center console system is running. The widget-related configuration file can be used to select and filter widget types. The first widget property data can be used to generate corresponding widget operation commands based on the user's voice request.

[0034] Specifically, when the target application is executed and a first interaction page is displayed, it can be used to receive a voice request made by a user for a widget shown in the first interaction page, and subsequently perform text conversion and semantic analysis on the voice request.

[0035] Before responding to a user's voice request, through the first interaction page of the target application, the widget text, widget identification information, widget number, etc. corresponding to the widget displayed on the first interaction page are obtained and assembled to generate corresponding first widget property data, which can be used to subsequently generate corresponding widget operation instructions. For example, for a search widget on the top page of the navigation, its widget text may be "search", its widget identification information may be 01, and its widget number may be 1. The widget text, widget identification information, widget number, etc. of the search widget are obtained and assembled to obtain the first widget property data corresponding to the search widget, which can be used to subsequently generate corresponding widget operation instructions when the user needs to operate the widget.

[0036] In some cases, the widgets displayed on the first interaction page may include non-click event widgets that do not respond to control events, such as label widgets, and it is generally not possible to directly generate corresponding widget operation instructions based on the first widget property data corresponding to such widgets. Therefore, when obtaining the widget text, widget identification information, and widget number corresponding to the widget, rule filtering can first be performed to select the widget type on the interaction page and filter out the non-click event widgets, thereby reducing the amount of data such as widget identification information corresponding to the obtained widgets and reducing the execution load of the voice interaction method.

[0037] Specifically, the widget type that needs to be obtained in advance can be configured in a widget-related configuration file. Before obtaining the widget identification information, widget text, and widget number corresponding to the widget displayed on the first interaction page, the widget type corresponding to the widget displayed on the first interaction page is matched with the widget type in the widget-related configuration file based on the widget type previously configured in the widget-related configuration file, and the widget that matches the widget type is selected as the target control widget, and the widget that does not match the widget type is filtered.

[0038] Furthermore, since the object of user voice interaction is usually the content on the current interaction page of the target application, the current interaction page can be monitored, and when the content of the current interaction page changes, data such as widget identification information of the target control widget on the current interaction page after the change can be obtained in real time and used to subsequently generate a corresponding widget operation command, thereby realizing the operation of the widget on the current interaction page when the target application is running for any target application installed in the automobile center console system, without the need to develop an additional interface for connecting to voice interaction control in the target application.

[0039] Specifically, the current interaction page monitors changes in window and page content that occur when switching from the second interaction page to the first interaction page, then selects and filters the widgets displayed on the first interaction page, and obtains the widget identification information, widget text, and widget number corresponding to the selected target control widget from the first interaction page. The widget identification information, widget text, and other data are assembled to generate corresponding first widget property data. In some embodiments, the current interaction page can be monitored through an auxiliary service function and / or an interception method of the operating system to obtain the widget identification information, widget text, and widget number corresponding to the widget on the current interaction page after the change occurs.

[0040] For example, the current interaction page is monitored through an auxiliary service function of the operating system to monitor changes in the window and page content. Then, the widgets displayed on the converted interaction page are selected and filtered based on the widget types previously configured in the widget-related configuration file to determine the target control widget on the page. Then, the widget identification information, widget text, and widget number corresponding to the target control widget on the converted page are obtained through a monitoring callback method, and corresponding first widget property data is generated and assembled, which is subsequently used to match the semantic text data, thereby identifying the interaction target widget that the user needs to operate and the corresponding widget operation type. In some embodiments, the auxiliary service function of the operating system may be an AccessibilityService service function supported by the Android system, and the interception method may be an Aspect interception method.

[0041] In some embodiments, after obtaining the widget text, widget identification information, and widget number corresponding to the target control widget on the first interaction page, these data can be arranged and assembled based on the widget-related arrangement file to generate corresponding first widget property data.

[0042] S120, matching semantic text data corresponding to the user's voice request in the first widget property data, and determining an interaction target widget corresponding to the semantic text data.

[0043] Wherein, the semantic text data may include the widget text of the interaction target widget, and the first widget property data may be a data collection including the widget text of all target control widgets on the first interaction page.

[0044] Specifically, after receiving a user's voice request, first, a semantic analysis is performed on the voice request to obtain corresponding semantic text data, and then, based on the widget text of the interaction target widget included in the semantic text data, a matching is performed on the widget text of all target control widgets on the first interaction page to find a target control widget corresponding to the widget text, and determine it as the interaction target widget.

[0045] For example, on the navigation homepage, a user's voice request "search for Oriental Pearl" is received, and the user's voice request "search for Oriental Pearl" is subjected to text conversion and analysis to obtain corresponding semantic text data, which includes "search" and "Oriental Pearl." Based on the semantic text data, matching is performed among the widget texts corresponding to all target control widgets on the navigation homepage to find widget text that is the same as "search." Thus, a search widget corresponding to the widget text "search" is found, and the widget is determined as the interaction target widget.

[0046] S130, generating a widget operation command according to the widget operation type of the interaction target widget.

[0047] Here, the widget operation type corresponds to the widget type. For example, if the widget type is a button, the corresponding widget operation type may be click, double click, long tap, etc.; if the widget type is a selection box, the corresponding widget operation type may be check, etc.; if the widget type is a slider, the corresponding widget operation type may be drag, etc.; if the widget type is a text box, the corresponding widget operation type may be character input, etc.

[0048] Specifically, after determining the interaction target widget corresponding to the semantic text data, the corresponding widget operation type can be identified based on the type of the interaction target widget, and behavioral relationship data corresponding to the control operation behavior for the interaction target widget can be obtained by analyzing based on the semantic text data, thereby generating a widget operation command for operating the interaction target widget.

[0049] For example, after finding the search widget based on the semantic text data, the type of the search widget can be identified as a composite widget of a text widget and a button widget, and the corresponding widget operation type can be identified as inputting text into the text widget and clicking the button widget. The semantic text data "Search" and "Oriental Pearl" and the widget operation type of the search widget can be combined to generate an instruction to operate the search widget, which can include inputting "Oriental Pearl" into the search text widget and then clicking the search button widget.

[0050] S140: Execute the widget operation command and output the response content to the user's voice request.

[0051] Here, the response content may be the result of executing a widget operation command.

[0052] Specifically, according to the widget operation command, a widget operation interface is called to perform an operation corresponding to the interaction target widget on the first interaction page, and the operation result is fed back and displayed on the current interaction page. In some embodiments, the widget operation interface may be provided by an auxiliary service function of the automobile center console system in which the target application is installed. In other embodiments, the widget operation interface may be obtained through an interception manner.

[0053] For example, based on the command to operate the generated search widget, the command may include invoking a widget operation interface provided by the auxiliary service function of the automobile center console system in which the target application is installed, performing a search operation corresponding to the search widget on the navigation homepage, inputting "Oriental Pearl" in the search box, and clicking a search button. The search result may be obtaining a route from the current location to Oriental Pearl, and the route may be fed back and displayed on the current interaction page.

[0054] In the above embodiment, rule filtering is implemented through widget-related configuration files to reduce the amount of data obtained, such as widget identification information corresponding to widgets on the current interaction page, thereby improving the efficiency of voice interaction. Changes in the window and page content of the current interaction page are monitored, and after monitoring the changes, data such as widget identification information corresponding to widgets on the current interaction page after the change is obtained, thereby realizing real-time acquisition of data such as widget identification information on the current interaction page. By decoupling the user's voice recognition and semantic analysis from specific target application widgets, the isolation of semantic behavior analysis from widget operation is realized. Compared with the conventional method of achieving voice interaction control for a target application in an automobile center console system by connecting the target application to a voice operation interface, this method does not require the target application to develop an interface for connecting voice interaction control, which significantly reduces the development cost of the target application and avoids a situation where a third-party target application installed in the automobile center console system does not support voice interaction due to its inability to connect to a voice operation interface.

[0055] The embodiment of the present specification provides a voice interaction method, and referring to FIG. 2, the method may include the following steps:

[0056] S210: When monitoring switching from the second interaction page to the first interaction page, determine a target control widget that can be controlled on the first interaction page among the widgets displayed on the first interaction page based on the widget type in the widget-related configuration file.

[0057] S220, obtaining first widget property data corresponding to a target control widget on a first interaction page.

[0058] S230, receiving a user's voice request through a first interaction page of the target application.

[0059] S240, matching semantic text data corresponding to the user's voice request in the first widget property data, and determining an interaction target widget corresponding to the semantic text data.

[0060] S250, generating a widget operation command based on the widget operation type of the interaction target widget.

[0061] S260, executing the widget operation command and outputting the response content to the user's voice request.

[0062] Specifically, when monitoring switching from the second interaction page to the first interaction page, the device selects and filters the widget types of the widget targets on the first interaction page based on the widget types previously configured in the widget-related configuration file, and determines the corresponding widget that matches the widget type in the widget-related configuration file as the target control widget. The device obtains the widget identification information, widget text, and widget number corresponding to the target control widget from the first interaction page, assembles data such as the widget identification information and widget text, and generates corresponding first widget property data.

[0063] A voice request issued by a user for a widget displayed on a first interaction page is received, and a semantic analysis is performed on the voice request to obtain corresponding semantic text data. Based on the widget text of the interaction target widget included in the semantic text data, matching is performed among the widget texts in the first widget property data of all target control widgets on the first interaction page, and a target control widget corresponding to the widget text is found and determined as the interaction target widget.

[0064] A corresponding widget operation type is identified based on the type of the interaction target widget, and behavioral relationship data corresponding to the control operation behavior for the interaction target widget is obtained by analyzing based on the semantic text data, thereby generating a widget operation command for operating the interaction target widget.

[0065] According to the widget operation command, a widget operation interface is invoked to perform an operation corresponding to the interaction target widget on the first interaction page, and the operation execution result is fed back and displayed on the current interaction page.

[0066] In some embodiments, when a widget of a target application is updated, the widget-associated configuration file can be modified to generate a new widget-associated configuration file, which is used to determine the target control widget in the interaction page of the target application after the update.

[0067] The widget-related configuration file may include a widget-related rule, which is used to associate and configure the widget identification information and widget text of the target control widget. The target control widget may have a relationship with other widgets.

[0068] In some cases, some widgets in the target application are displayed as icons, or the widgets are separated from the widget text content, and the corresponding widget text associated with such widgets cannot be directly obtained, making it impossible to accurately determine the interaction target widget corresponding to the semantic text data. Because the widget identification information is unique, the widget text corresponding to the widget identification information is located, and the corresponding widget identification information can be found through the identified widget text, thereby finding the corresponding widget.

[0069] Specifically, the widget identification information, widget text, etc. corresponding to the acquired target control widget can be arranged based on the widget-related rules in the widget-related configuration file. When the widget of the target application is updated, the widget-related rules in the widget-related configuration file can be appropriately modified to generate a new widget-related configuration file. After the new widget-related configuration file is published and the interaction page of the target application is updated, when it is monitored that the second interaction page has been switched to the first interaction page, the new widget-related configuration file can be acquired and used to determine the target control widget on the first interaction page after the widget has been updated, and to arrange the widget identification information, widget text, etc. of the target control widget to generate new first widget property data. The new first widget property data can be matched with semantic text data to generate corresponding widget operation commands. Compared with unconfigured first widget property data, the configured first widget property data can more appropriately match the logic for performing behavior analysis on the semantic text data. When a new function appears in the target application, the widget-related configuration file can be modified accordingly to realize the configuration modification to the first widget property data, thereby realizing the addition or modification to the widget operation. In some embodiments, the widget-related configuration file can adopt a JSON file.

[0070] In the above embodiment, configuration modifications to widget property data can be realized by modifying the widget-related configuration file, and additions or modifications to widget operations of a target application can be realized without modifying the code implemented by the widget operation. For a new target application, there is no need to connect to a voice interaction control interface, and there is no need to add corresponding logic in the code implemented by the widget operation. By modifying the widget-related configuration file, operations can be performed on the widgets of the new target application. The widget-related configuration file supports online modifications. After modifying and publishing the widget-related configuration file, the new widget-related configuration file can be obtained when it is necessary to obtain widget property data corresponding to a widget.

[0071] In some embodiments, the second widget property data corresponding to the target control widget on the second interaction page is stored in the buffer space. The voice interaction method further includes at least one of the following:

[0072] If the second widget property data and the first widget property data do not match, clearing the second widget property data in the buffer space and storing the first widget property data in the buffer space.

[0073] If the second widget property data and the first widget property data match, reserving the second widget property data in a buffer space and discarding the first widget property data.

[0074] Here, the widget property data in the buffer space can be matched with the semantic text data to determine the interaction target widget.

[0075] In some cases, when monitoring switching from the second interaction page to the first interaction page, the page content of the current interaction page may change, but the target control widget may remain the same. For example, during video playback, the first widget property data acquired on the current video playback page and the second widget property data acquired 5 seconds ago may differ in the video content, but the other target control widgets remain unchanged, and the corresponding first widget property data and second widget property data are the same. In this case, to save the performance of voice interaction, there is no need to update the second widget property data with the first widget property data. Therefore, a buffer space can be set up to compare the first widget property data with the second widget property data to determine whether the target control widgets on the first interaction page and the second interaction page are the same.

[0076] In some embodiments, it can be determined whether the second widget property data and the first widget property data are consistent through the MD5 information digest value. Specifically, the MD5 information digest values ​​of all target control widgets corresponding to the second widget property data in the buffer space and the acquired first widget property data are calculated and compared. If the MD5 information digest value corresponding to the second widget property data does not match the MD5 information digest value corresponding to the first widget property data, the buffer space is cleared and the first widget property data is stored in the buffer space. If the MD5 information digest value corresponding to the second widget property data matches the MD5 information digest value corresponding to the first widget property data, the second widget property data in the buffer space is reserved and the first widget property data is discarded.

[0077] In some embodiments, the voice interaction method employs at least one of the following methods to obtain first widget property data corresponding to a widget displayed on the first interaction page:

[0078] Obtain first widget property data corresponding to the widget shown on the first interaction page based on the message queue.

[0079] Here, the message queue can be used to monitor switching from the second interaction page to the first interaction page and control the frequency of acquiring the first widget property data.

[0080] In some cases, the window and page content of the current interaction page may change continuously, and the corresponding change frequency may be high. If the current interaction page is monitored for changes and then the widget property data of the widget on the new page is immediately obtained, the frequency of the data acquisition may be too high, which may affect the performance of voice interaction. Therefore, a message queue can be set up and the frequency control function of the message queue can be used to control the frequency of widget property data acquisition, thereby preventing a sudden increase in frequency from affecting the performance of the automobile center console system and reducing wear and tear on the automobile center console system.

[0081] Specifically, after the first widget property data is obtained by the method of any of the above embodiments, the first widget property data is sent to a message queue. The message queue processes the first widget property data in the queue at regular time intervals, for example, sends it to a buffer space. In some embodiments, the message queue can process the first widget property data in the queue every 10 seconds.

[0082] If no popup appears on the first interaction page, first widget property data corresponding to other widgets than the progress type widget is obtained on the first interaction page by calling an auxiliary service function of the operating system.

[0083] Here, the operating system is an automobile center console system. The auxiliary service function can be used to provide a user with auxiliary services such as replacement and feedback enhancement, support retrieval of widgets other than a progress bar widget on an interaction page, and provide a widget operation interface with the ability to perform various controls on the widget, such as clicking, double-clicking, checking, long-pressing, and dragging. The first widget property data can be generated by assembling widget text, widget identification information, widget number, etc. corresponding to the widget.

[0084] Specifically, if no popup appears on the first interaction page, all widgets on the first interaction page are displayed to the auxiliary service function of the operating system. The auxiliary service function of the operating system can obtain widget text, widget identification information, widget numbers, etc. corresponding to widgets other than the progress type widget on the first interaction page, and assemble them according to certain rules to generate the first widget property data. In some embodiments, the auxiliary service function of the operating system can be the AccessibilityService auxiliary service function supported by the Android system.

[0085] When the popup appears on the first interaction page, obtain first widget property data corresponding to the widget shown on the first interaction page through an Aspect interception method.

[0086] In some cases, as shown in FIG. 3, a popup Dialog component is displayed on the first interaction page, and the toolbox widgets under the popup Dialog component are visible to the user, but are not visible to the auxiliary service functions of the operating system because they are covered by the Dialog component. That is, the auxiliary service functions of the operating system cannot obtain widget property data corresponding to the toolbox widgets under the Dialog component. Therefore, the widget property data corresponding to such widgets can be obtained through the Aspect interception method.

[0087] Specifically, the widget object under the popup on the first interaction page is hooked through Aspect interception, and the widget property data corresponding to the hooked widget object is obtained. When the corresponding widget operation command needs to be executed on the hooked widget object, the corresponding widget operation interface can be called through reflection.

[0088] If there is a progress type widget on the first interaction page, the first widget property data corresponding to the progress type widget on the first interaction page is obtained by an Aspect interception method.

[0089] In some cases, the auxiliary service functions of the operating system support many operations on widgets, such as clicking, long pressing, double clicking, and checking, but for progress bar type widgets, the auxiliary service functions of the operating system cannot invoke corresponding methods such as setting values, etc. Therefore, this embodiment uses Aspect interception to obtain widget property data corresponding to the progress bar type widget and realizes operations on the progress bar type widget.

[0090] Specifically, the progress bar type widget object on the first interaction page is hooked using Aspect interception to obtain widget property data corresponding to the hooked widget object. When a corresponding widget operation command needs to be executed on the hooked progress bar type widget, the widget operation method of the progress bar type widget is invoked using reflection to realize the operation on the progress bar type widget. For example, the setRate method of the progress bar type widget is invoked using reflection to realize setting a specific value on the progress bar type widget.

[0091] In some cases, the Aspect interception method can be used to fully obtain widgets on the first interaction page and execute corresponding widget operation commands on the widgets, but the target application must integrate the corresponding function package, i.e., the Jar package corresponding to the Aspect function. If the target application does not integrate the Jar package, it cannot connect to the speak function for everything it sees. In the above embodiment, a method combining the operating system's auxiliary service function and the Aspect interception method can be used to obtain widget property data corresponding to widgets on the interaction page and execute widget operation commands on the widgets to control them. If the target application does not integrate a Jar package, it can connect to the speak function for everything it sees through the operating system's auxiliary service function. If there is no need to control progress bar widgets and widgets covered by popups through voice interaction, most voice interaction controls on the interaction page can be achieved through the operating system's auxiliary service function, without affecting the target application. If a target application needs to control progress bar widgets and widgets covered by popups through voice interaction, it can integrate a Jar package that supports the Aspect function into the target application and combine the operating system's auxiliary service function with Aspect interception to more fully support the "Speak What You See" function. By combining the operating system's auxiliary service function with Aspect interception, it is possible to avoid situations where the target application cannot connect to the voice interaction function.

[0092] In some cases, the target application is associated with a behavior analysis module and a widget operation module. Referring to Figure 4, the first widget property data generating method can include the following steps:

[0093] S310, when monitoring switching from the second interaction page to the first interaction page, generate first widget property data according to the widget identification information, widget text, and widget number of the target control widget through the widget operation module.

[0094] Here, the widget identification information may be an ID value that uniquely identifies a widget, and the widget number may be a number value that serves as an auxiliary identifier for the same widget. The first widget property data may be data used to describe one widget and distinguish it from other widgets.

[0095] The target application can connect to the widget operation module through a widget operation interface, and the widget operation module can be used to connect to the behavior analysis module and the widget in the target application. Specifically, the widget operation module monitors the content of the current interaction page and monitors whether the second interaction page has been switched to the first interaction page. After that, the widget operation module can determine the target control widget on the first interaction page based on the widget type in the widget-related configuration file. The widget operation module obtains the widget identification information, widget text, and widget number corresponding to the target control widget on the first interaction page, and arranges and assembles these data based on the widget-related configuration file to generate first widget property data.

[0096] In some embodiments, the first widget property data may be a widget message. Monitoring the interaction page through the widget operation module and obtaining the widget identification information, widget text, and widget number corresponding to the widget can be achieved by using the auxiliary service function of the operating system or Aspect interception through the widget operation module, or can be achieved by using the auxiliary service function of the operating system plus Aspect interception.

[0097] In addition, the widget operation module can be used to obtain widget identification information, widget text, and widget number corresponding to all widgets on the first interaction page, and can also be used to obtain widget identification information, widget text, and widget number corresponding to the target control widget on the first interaction page based on the widget type in the widget-related configuration file.

[0098] S320, calling a widget registration interface of the behavior analysis module, and registering the first widget property data in the behavior analysis module.

[0099] Here, the behavior analysis module matches the semantic text data in the first widget property data, and determines an interaction target widget corresponding to the semantic text data.

[0100] Specifically, the widget operation module calls a widget registration interface defined by the behavior analysis module, and registers the first widget property data generated by the widget operation module with the behavior analysis module. In some embodiments, the widget operation module registers the first widget property data in a widget buffer space, and the behavior analysis module matches the semantic text data with the first widget property data in the widget buffer space.

[0101] In the above embodiment, an interface is adopted to connect the behavior analysis module and the widget operation module, and each module is responsible for a corresponding part of the functions and processes. The technology choices of the behavior analysis module and the widget operation module do not affect each other.

[0102] In some embodiments, registering the first widget property data with the behavioral analysis module includes registering the first widget property data with the behavioral analysis module based on widget-related rules predefined in a widget-related configuration file.

[0103] Here, the widget association rules are used to associate widget identification information with widget text and place it there, so that the interaction target widget can be accurately determined.

[0104] In some cases, some widgets in a target application are displayed as icons, or widgets are separated from their widget text content, and the corresponding widget text associated with such widgets cannot be directly obtained. Because the widget identification information is unique, widget association rules are predefined to locate the widget text corresponding to the widget identification information, and the corresponding widget identification information can be found through the identified widget text, thereby enabling the corresponding widget to be found.

[0105] In some other cases, for example, in a list widget, the widget identification information corresponding to elements of the same type in each item is the same, i.e., the identification information of the list widget, and the corresponding widget text may also be the same. In this case, it is not possible to accurately identify which element in the list widget is the interaction target widget through the widget text alone. Since some widgets have a certain association relationship, widget association rules can be defined in advance, and the widget identification information and widget text for uniquely determining the widget can be associated and arranged, so that the corresponding widget can be accurately found through the identified associated text.

[0106] Specifically, the widget operation module performs corresponding arrangement of the acquired widget identification information, widget text, and widget number based on the widget-related rules predefined in the widget-related arrangement file, assembles the arranged contents into first widget property data, and registers the first widget property data in the behavior analysis module by calling a widget registration interface defined by the behavior analysis module.

[0107] For example, referring to FIG. 5a, on the navigation top page, the results of searching for addresses are displayed in the form of a list, and each item in the list has a button whose widget text is "Route." If "Route" is directly used as the widget text of the identified button and the process of matching semantic text data is performed, it is not possible to accurately determine which item in the list is the "Route" button that needs to be identified. Since each "Route" button can be determined by associating it with the title text (i.e., address text) of the list item to which it belongs, the association between the "Route" button and the title text of the list item to which it belongs can be written in a widget association rule and used to arrange the widget text of the "Route" button.

[0108] In some embodiments, the widget-related rules may be JSON rules. Illustratively, with continued reference to FIG. 5a, the corresponding widget-related rules may be expressed as follows:

[0109] { “page”: “Navigation top page” “com.zeekr.automap:id / slv_search_button”:{ “des”:“search”, “itemType”:“button” } “com.zeekr.automap:id / slv_search_list”:{ “des”:“root”, “itemid”:“com.zeekr.automap:id / cl_auto_search_listview_item”, “itemdes”:“com.zeekr.automap:id / cl_auto_search_listview_title”, “itemType”:“list” } }

[0110] The above widget-related rule specifies that there are two widgets on the navigation top page, the widget identification information id of the first widget is com.zeekr.automap:id / slv_search_button and the corresponding widget text is "Search", and the widget identification information id of the second widget is com.zeekr.automap:id / slv_search_list and the corresponding widget text is "Route". itemid is used to define the widget operation type identifier corresponding to each item element in the second widget, and itemdes is used to define the text corresponding to each item element in the second widget. Through the identifier itemid+itemdes, the element and corresponding operation type of each item in the list can be defined, for example, "No. 100 Haike Road, Pudong New Area, Shanghai" (this text is the text of the id corresponding to itemdes, i.e., com.zeekr.automap:id / cl_auto_search_listview_title), and the text "Root" defined by the des of the second widget is used as the identification text of the route button of the first item, and the widget operation type identifier of the route button of the first item is generated by the button widget (i.e., com.zeekr.automap:id / cl_auto_search_listview_item).

[0111] The widget operation module assembles the above-mentioned arranged content into widget property data, and registers the widget property data in the behavior analysis module by calling the widget registration interface defined by the behavior analysis module. When the user says "No. 100 Haike Road, Pudong New Area, Shanghai" and "route", the behavior analysis module matches the semantic text data and determines that the user needs to click the route button corresponding to an item such as No. 100 Haike Road, Pudong New Area, Shanghai, and then generates a widget operation command for the button widget. The widget operation module clicks the route button of the first item based on the widget operation command, thereby selecting the route to No. 100 Haike Road, Pudong New Area, Shanghai.

[0112] For example, Figure 5b is a schematic diagram of a widget message registration process between a widget operation module and a behavior analysis module. Referring to Figure 5b, in the widget message registration process, the widget operation module uses the AccessibilityService+Aspect method to obtain widget identification information, widget text, and widget number corresponding to the widget in the current interaction page, and then arranges and assembles this data according to predefined widget-related rules in the widget-related configuration file to obtain first widget property data. The widget operation module then calls the widget registration interface defined by the behavior analysis module and registers the first widget property data in the widget buffer space.

[0113] In the above embodiment, widget-related rules are adopted to configure widgets, and widget controls in the target application can be added or modified without modifying the code of the widget operation module. Furthermore, the widget-related configuration file supports online configuration and publishing, eliminating the need to upgrade the functions of the associated modules through over-the-air technology.

[0114] In some embodiments, before executing the widget operation command, the voice interaction method includes receiving a widget operation command issued by the behavior analysis module through a command issuing interface of the widget operation module.

[0115] Specifically, the behavior analysis module searches the first widget property data for a widget text of the interaction target widget included in the user's semantic text data, finds a widget corresponding to the widget text, and determines it as the interaction target widget. Then, the behavior analysis module identifies a corresponding widget operation type based on the type of the interaction target widget, analyzes the semantic text data to obtain behavioral relationship data corresponding to a control operation behavior on the interaction target widget, and generates a widget operation command for operating the interaction target widget. The behavior analysis module invokes the command sending interface of the widget operation module and sends the generated widget operation command to the widget operation module. The widget operation module receives the widget operation command and performs a corresponding operation on the interaction target widget.

[0116] In some embodiments, the semantic text data may be obtained by performing text conversion and semantic analysis on a user's voice request received by the semantic analysis module, and a behavior analysis module may be used to connect the semantic analysis module and the widget operation module.

[0117] The step of executing the widget operation command may include the step of invoking a widget operation interface of a widget operation module to execute the widget operation command on the interaction target widget.

[0118] Illustratively, Figure 5c is a schematic diagram of a widget operation interaction process between a behavior analysis module and a widget operation module. Referring to Figure 5c, the widget operation interaction process may include: the behavior analysis module matches semantic text data with the widget text of the first widget property data in the widget buffer space, determines an interaction target widget and a corresponding widget operation type, and generates a corresponding widget operation command. The behavior analysis module invokes a command issuing interface of the widget operation module to issue a widget operation command to the widget operation module. After the widget operation module receives the widget operation command, it invokes a widget operation interface to perform a corresponding operation on the interaction target widget on the current interaction page.

[0119] In some embodiments, the widget operation interface can be provided by an auxiliary service function of the operating system, which can be the AccessibilityService service function of the Android system. Specifically, after receiving a widget operation command through the widget operation module, the widget operation interface provided by the AccessibilityService service function can be called to perform various widget operations on the interaction target widget, including clicking, double-clicking, checking, long-pressing, dragging, etc.

[0120] In some other embodiments, the widget operation interface can be obtained through Aspect interception. Specifically, the widget object displayed on the first interaction page is hooked through the widget operation module through Aspect interception. When a corresponding widget operation command needs to be executed on the hooked interaction target widget object, a reflection method is used to invoke the widget operation method of the interaction target widget object, thereby realizing the operation on the interaction target widget. The widget operation interface obtained through Aspect interception can be used to realize complete widget operation.

[0121] Executing a widget operation command by calling a widget operation interface obtained through Aspect interception requires integrating a Jar package corresponding to the Aspect function corresponding to the target application, and therefore, a method combining an auxiliary service function of the operating system with the Aspect interception method can be used to execute a widget operation command. If there is no need to operate a progress bar type widget or a widget covered by a popup, widget operation for a widget on an interaction page can be achieved by calling a widget operation interface provided by the auxiliary service function of the operating system. If there is a need to operate a progress bar type widget or a widget covered by a popup, a Jar package corresponding to the Aspect function can be integrated into the target application and the widget operation interface obtained through the Aspect interception method can be called to achieve the corresponding widget operation.

[0122] In some embodiments, the first widget property data includes a widget text of the widget, and matching semantic text data corresponding to the user's voice request in the first widget property data, and determining an interaction target widget corresponding to the semantic text data includes determining an interaction target widget corresponding to the semantic text data based on a matching result between the semantic text data corresponding to the user's voice request and the widget text of the widget.

[0123] Here, the semantic text data may be obtained by performing text conversion and semantic analysis on the user's voice request using a semantic analysis module.

[0124] Specifically, referring to FIG. 6, the semantic analysis module receives a user's voice request, converts the user's voice request into text, performs semantic analysis on the semantic analysis module, and obtains corresponding semantic text data, which is then sent to the behavior analysis module. The semantic text data is received through the behavior analysis module, and the behavior analysis module matches the semantic text data with the widget text included in the first widget property data. The widget corresponding to the widget text matching the text in the semantic text data is determined as the interaction target widget, and the corresponding widget operation type is identified based on the type of the interaction target widget. The interaction target widget and the corresponding widget operation type are associated, and the corresponding behavioral relationship data is analyzed. The behavior analysis module generates a widget operation command for the interaction target widget, and sends it to the widget operation module through a command sending interface. The widget operation module receives the widget operation command, invokes the widget operation interface, and executes the corresponding widget operation command for the interaction target widget to complete controlling the widget of the target application through voice interaction.

[0125] In addition, the semantic analysis module can adopt mature semantic analysis technology, and the behavior analysis module can adopt mature behavior analysis technology, and the technology selection of the semantic analysis module and the behavior analysis module does not affect each other.

[0126] In the above embodiment, the process of controlling the widget of a target application through voice interaction is divided into three modules: semantic analysis, behavioral analysis, and widget operation. The semantic analysis module, behavioral analysis module, and widget operation module are each responsible for a specific part of the process, and the technology selection of the three modules does not affect each other, thereby realizing the decoupling of semantic analysis, behavioral analysis, and specific widget operation. Whether the target application is independently developed by the service provider that provides the ``Speak What You See'' function or a target application provided by a third party, there is no need to develop an interface to connect to the ``Speak What You See'' function of the automobile center console system, and all of them can be connected to the ``Speak What You See'' function.

[0127] An embodiment of the present specification provides a voice interaction device applied to an automobile center console system. Referring to FIG. 7, the voice interaction device 400 may include a receiving module 410, a matching module 420, a command generation module 430, and a command execution module 440.

[0128] The receiving module 410 is used for receiving a user's voice request through a first interaction page of a target application, where a target control widget is displayed on the first interaction page, the target control widget is a controllable widget determined in the widget displayed on the first interaction page of the target application according to the widget type in the widget-related configuration file, and first widget property data is associated with the target control widget, and the first widget property data is generated according to the widget identification information, widget text, and widget number of the target control widget when monitoring switching from the second interaction page to the first interaction page.

[0129] The matching module 420 is used to match the semantic text data corresponding to the user's voice request in the first widget property data, and determine the interaction target widget corresponding to the semantic text data.

[0130] The command generation module 430 is used to generate a widget operation command based on the widget operation type of the interaction target widget.

[0131] The command execution module 440 is used to execute widget operation commands and output responses to user voice requests.

[0132] For specific limitations of the voice interaction device, please refer to the limitations of the voice interaction method described above and will be omitted here. Each module in the voice interaction device can be realized in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be integrated into or stand-alone in a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0133] An embodiment of the present invention further provides a computer device, as shown in FIG. 8, wherein the computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable by the processor 520, and when the processor 520 executes the computer program 530, the computer device 500 realizes the voice interaction method of any one of the above embodiments.

[0134] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, the computer program realizing the voice interaction method of any one of the preceding embodiments when executed by a processor.

[0135] It should be noted that the logic and / or steps depicted in process charts or otherwise described herein may be viewed, for example, as an executable sequential listing of instructions for implementing logical functions, and may be tangibly embodied in any computer-readable medium for use with an instruction-executing system, apparatus, or device (including, for example, a computer-based system, a processor-based system, or a system that retrieves and executes instructions from other instruction-executing systems, apparatus, or devices), or for use in conjunction with such instruction-executing systems, apparatus, or devices. For purposes of this specification, a "computer-readable medium" may include any program capable of containing, storing, communicating, propagating, or transmitting an instruction-executing system, apparatus, or device, or a device for use in conjunction with such an instruction-executing system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include electrical connections having one or more wires (electronic devices), portable computer disk enclosures (magnetic devices), random access memory (RAM), read-only memory (ROM), programmable-erasable read-only memory (EPROM or flash memory), fiber optic devices, and portable CD-ROMs (CD-ROMs). It may also be a computer readable medium, such as paper or other suitable medium onto which the program may be printed, since the program may be obtained in electronic form, for example by optically scanning the paper or other medium and editing, interpreting or otherwise processing as needed in an appropriate manner, and then storing it in computer memory.

[0136] It should be noted that each part of the present application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described implementation, several steps or methods can be implemented in software or firmware stored in a memory and executed by an appropriate instruction execution system. For example, when implemented in hardware, as in other implementations, the hardware can be implemented in any one or combination of discrete logic circuits having logic circuits for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic circuits, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), etc., as are known in the art.

[0137] In the description herein, reference to the terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that the specific features, structures, materials, or characteristics described in the relevant embodiment or example are included in at least one embodiment or example of the present disclosure. Exemplary expressions of the terms herein do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any embodiment or example.

[0138] It should be noted that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or to specify the number of technical features being indicated. Thus, a feature qualified as "first" or "second" may indicate or imply the inclusion of at least one of that feature. In the description of this disclosure, unless otherwise clearly and specifically limited, "plurality" means at least two, e.g., two, three, etc.

[0139] In this disclosure, unless otherwise clearly specified or limited, the terms "attached," "connected," "coupled," "fixed," etc. shall be interpreted broadly, and may refer to, for example, a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a direct connection, an indirect connection via an intermediate medium, internal communication between two elements, or an interactive relationship between two elements. Those skilled in the art can understand the specific meanings of the above terms in this disclosure according to specific circumstances.

[0140] Although the embodiments of the present disclosure have been shown and described above, the embodiments are merely illustrative and should not be understood as limiting the present disclosure, and those skilled in the art may make modifications, alterations, substitutions and variations to the embodiments within the scope of the present disclosure.

Claims

1. A voice interaction method applied to an automobile center console system, comprising: The method comprises: receiving a user's voice request through a first interaction page of a target application, wherein a target control widget is displayed on the first interaction page, the target control widget being a controllable widget determined in the widget displayed on the first interaction page of the target application based on a widget type in a widget-related configuration file; and a first widget property data is associated with the target control widget, the first widget property data being generated based on widget identification information, widget text, and widget number of the target control widget when monitoring switching from a second interaction page to the first interaction page; Matching semantic text data corresponding to the user's voice request in the first widget property data, and determining an interaction target widget corresponding to the semantic text data; generating a widget operation command based on a widget operation type of the interaction target widget; and executing the widget operation command to output a response to the user's voice request.

2. 2. The voice interaction method of claim 1, wherein when a widget of the target application is updated, the widget-related configuration file can be modified to generate a new widget-related configuration file, and the new widget-related configuration file can be used to determine a target control widget in an interaction page of the updated target application.

3. Second widget property data corresponding to the target control widget on the second interaction page is stored in a buffer space, and the method further includes: If the second widget property data and the first widget property data do not match, clearing the second widget property data in the buffer space and storing the first widget property data in the buffer space; 2. The voice interaction method according to claim 1, further comprising at least one of the steps of: reserving the second widget property data in the buffer space and discarding the first widget property data when the second widget property data and the first widget property data match.

4. Obtaining first widget property data corresponding to the widget shown on the first interaction page based on a message queue; If no popup appears on the first interaction page, obtain first widget property data corresponding to other widgets than the progress type widget on the first interaction page by calling an auxiliary service function of an operating system; When a popup appears on the first interaction page, obtain first widget property data corresponding to the widget displayed on the first interaction page through an aspect interception manner; 2. The voice interaction method of claim 1, wherein when the progress type widget is present on the first interaction page, at least one of: acquiring first widget property data corresponding to the progress type widget on the first interaction page using an aspect interception method to acquire first widget property data corresponding to the widget shown on the first interaction page.

5. The target application corresponds to a behavior analysis module and a widget operation module, and the first widget property data generation method is: generating the first widget property data through the widget operation module according to the widget identification information, widget text, and widget number of the target control widget when monitoring switching from the second interaction page to the first interaction page; 2. The voice interaction method according to claim 1, further comprising: invoking a widget registration interface of the behavior analysis module and registering the first widget property data with the behavior analysis module.

6. Registering the first widget property data in the behavior analysis module includes: The voice interaction method according to claim 5, further comprising: registering first widget property data with the behavior analysis module based on widget-related rules predefined in a widget-related configuration file.

7. Before executing the widget operation command, the method includes: The method further includes receiving a widget operation command sent by the behavior analysis module through a command sending interface of the widget operation module; The step of executing the widget operation command includes: The voice interaction method according to claim 5, further comprising: invoking a widget operation interface of the widget operation module to execute the widget operation command on the interaction target widget.

8. The first widget property data includes a widget text of the widget, and the step of matching semantic text data corresponding to the user's voice request in the first widget property data and determining an interaction target widget corresponding to the semantic text data includes: The voice interaction method according to any one of claims 1 to 7, further comprising: determining an interaction target widget corresponding to the semantic text data based on a matching result between the semantic text data corresponding to the user's voice request and the widget text of the widget.

9. A voice interaction device applied to an automobile center console system, the device comprising: a receiving module used for receiving a user's voice request via a first interaction page of a target application, wherein a target control widget is displayed on the first interaction page, the target control widget is a controllable widget determined in the widget displayed on the first interaction page of the target application based on a widget type in a widget-related configuration file, and first widget property data corresponds to the target control widget, and the first widget property data is generated based on widget identification information, widget text, and widget number of the target control widget when monitoring switching from a second interaction page to the first interaction page; a matching module used for matching semantic text data corresponding to the user's voice request in the first widget property data and determining an interaction target widget corresponding to the semantic text data; a command generating module for generating a widget operation command based on a widget operation type of the interaction target widget; a command execution module used for executing the widget operation command to output a response content to the user's voice request.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor, when executing the computer program, performs the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.

Images