Voice interaction method and device, and storage medium

Abstract

The application discloses a voice interaction method and device and a storage medium, and the method comprises the following steps: if a voice instruction of a user is received, target information is extracted from a target text corresponding to the voice instruction, and the target information comprises at least one of the following information: target content role information and target scene role information; based on N interface elements contained in a current user interface, structured metadata corresponding to each interface element is generated, N is a positive integer; the target information is matched and calculated with the structured metadata of each interface element, and a target interface element hit by the voice instruction is determined from the N interface elements based on a matching calculation result; and an interaction operation with the target interface element is triggered based on the voice instruction. The scheme can accurately locate the target interface element of the current interaction of the user, and improve the accuracy of voice interaction.

Description

Technical Field

[0001] This invention relates to the field of voice interaction, and in particular to a voice interaction method, apparatus and storage medium. Background Technology

[0002] With the increasing development of human-computer interaction technology, the "Visible-and-Speakable" concept has received more and more attention and application. "Visible-and-Speakable" allows users to trigger interactive operations by speaking any visible text on the interface. For example, in a driving scenario, a user can speak the content displayed on the vehicle's central control screen, such as "open news about celebrity A," to display a specific news item. However, in related technologies, "Visible-and-Speakable" usually requires a pre-built static keyword library, which cannot cover the latest terms related to current events, leading to difficulties in completing the interaction process smoothly. Summary of the Invention

[0003] This invention provides a voice interaction method, device, medium, and storage medium to more accurately realize the visual and verbal interaction process.

[0004] In a first aspect, embodiments of the present invention provide a voice interaction method, including: If a user's voice command is received, target information is extracted from the target text corresponding to the voice command. The target information includes at least one of the following: target content role information and target scene role information, wherein the target content role information is used to characterize the named entity information of the voice command, and the target scene role information is used to characterize the business category to which the interactive intent of the target text belongs. Based on the N interface elements contained in the current user interface, generate structured metadata corresponding to each interface element, where N is a positive integer. The structured metadata includes the display text information of the corresponding interface element for matching with content role information, and the business tag information to which the corresponding interface element belongs for matching with scene role information. The target information is matched and calculated with the structured metadata of each interface element, and the target interface element hit by the voice command is determined from the N interface elements based on the matching calculation results. The voice command triggers an interactive operation with the target interface element.

[0005] In some implementations, the target information further includes target referential role information, and the extraction of target information from the target text corresponding to the voice command includes: The text used to represent indicative semantic information is extracted from the target text and used as the target referential role information.

[0006] In some implementations, generating structured metadata corresponding to each of the N interface elements contained in the current user interface includes: Obtain the target rendering tree corresponding to the current user interface, wherein the target rendering tree contains N nodes corresponding to the N interface elements, and each node corresponds to the original text and control type of the node; For each node, the original text of the node is used as the display text information of the node, and the business tag information of the node is determined based on the control type of the node; and the structured metadata of the interface element corresponding to the node is constructed based on the display text information and business tag information of the node.

[0007] In some implementations, after obtaining the target rendering tree corresponding to the current user interface, the method further includes: If a missing field is detected in the target rendering tree, obtain the temporary screen frame corresponding to the current user interface; The temporary screen frame is input into a preset visual language model, and the display text information and business tag information of each element contained in the temporary screen frame are output. Based on the display text information and business tag information of each element contained in the temporary screen frame, the missing fields of the target rendering tree are filled in, so as to determine the structured metadata of each interface element based on the filled target rendering tree.

[0008] In some implementations, if the target information includes target content role information, target scene role information, and target referential role information, the matching calculation based on the target information and the structured metadata of each interface element includes: For the structured metadata of each interface element, the target content role information is matched with the display text information corresponding to the interface element to obtain a first matching result; the target scene role information is matched with the business tag information corresponding to the interface element to obtain a second matching result; and the target referential role information is matched with a preset referential rule lexicon to obtain a third matching result. The first matching result, the second matching result, and the third matching result are then fused and calculated, and the fused calculation score is used as the matching result between the interface element and the target information.

[0009] In some implementations, matching the target scene role information with the business tag information corresponding to the interface element to obtain a second matching result includes: Based on the preset correspondence between scene role information and standard tag information, the target standard tag information corresponding to the target scene role information is determined; For each interface element, the target standard tag information is matched with the business tag information corresponding to that interface element to obtain the second matching result.

[0010] In some implementations, determining the target interface element hit by the voice command from the N interface elements based on the matching calculation result includes: Based on the fusion calculation score between each interface element and the target information, the target score with the highest fusion calculation score that is greater than the preset score is selected, and the interface element corresponding to the target score is taken as the target interface element.

[0011] In some embodiments, the method further includes: If multiple candidate user interfaces are currently displayed, obtain the name of the application to which each candidate user interface belongs; If the target information contains the target scene role information, the target scene role information is matched with the name of the application to which each candidate user interface belongs, and the candidate user interface with the highest matching degree is taken as the current user interface.

[0012] In some embodiments, the method further includes: If a change event for the current user interface is detected, determine whether the voice command interaction has been completed; If the voice command interaction has been completed, extract the updated structured metadata corresponding to the change event, and perform the next user voice interaction based on the updated structured metadata.

[0013] In some implementations, triggering the interaction with the target interface element based on the voice command includes: Extract the target operation type of the voice command from the target text; Based on the target operation type and the target interface element, generate and execute interactive instructions.

[0014] Secondly, embodiments of the present invention provide a voice interaction device, the device comprising: The target information extraction module is used to extract target information from the target text corresponding to the voice command when a user's voice command is received. The target information includes at least one of the following: target content role information and target scene role information, wherein the target content role information is used to characterize the named entity information of the voice command, and the target scene role information is used to characterize the business category to which the interactive intent of the target text belongs. The structured metadata generation module is used to generate structured metadata corresponding to each of the N interface elements contained in the current user interface, where N is a positive integer. The structured metadata includes the display text information of the corresponding interface element for matching with content role information, and the business tag information to which the corresponding interface element belongs for matching with scene role information. The processing module is used to perform matching calculations between the target information and the structured metadata of each interface element, and determine the target interface element hit by the voice command from the N interface elements based on the matching calculation results. The triggering module is used to trigger interactive operations with the target interface elements based on the voice command.

[0015] Thirdly, embodiments of the present invention provide a voice interaction device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the above-described voice interaction method.

[0016] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described voice interaction method.

[0017] The above-described one or more technical solutions in the embodiments of this application have at least the following technical effects: In the voice interaction method provided in the embodiments of this specification, if a user's voice command is received, target information is extracted from the target text corresponding to the voice command. The target information includes at least one of the following: target content role information and target scene role information. The target content role information is used to represent the named entity information of the voice command, and the target scene role information is used to represent the business category corresponding to the target text. Based on the N interface elements contained in the current user interface, structured metadata corresponding to each interface element is generated. The structured metadata of each interface element includes display text information and business tag information. The display text information is used to match the content role information, and the business tag information is used to match the scene role information. N is a positive integer. Matching calculations are performed based on the target information and the structured metadata of each interface element, and the target interface element hit by the voice command is determined from the N interface elements based on the matching calculation results. An interactive operation with the target interface element is triggered based on the voice command. In this solution, by constructing structured metadata for each interface element in the current user interface, new words and hot words appearing on the current user interface can be captured and instantly referenced, providing a foundation for realizing the visual and eloquent nature of the current user interface. Furthermore, when a user interacts with any interface element in the current user interface, the semantics of the user's voice command are accurately perceived by extracting the target content role information and / or target scene role information from the user's voice command. The target information is then matched with the structured metadata of each interface element to accurately locate the target interface element that the user is currently interacting with, complete the final interaction, and improve the accuracy of voice interaction. Attached Figure Description

[0018] Figure 1 A flowchart illustrating a voice interaction method provided in an embodiment of this specification; Figure 2 This is a schematic diagram of the functional modules of a voice interaction device provided in the embodiments of this specification; Figure 3 This is a schematic diagram of the structure of a voice interaction device provided in an embodiment of this specification. Detailed Implementation

[0019] The overall technical solution of this specification embodiment is as follows: If a user's voice command is received, target information is extracted from the target text corresponding to the voice command. The target information includes at least one of the following: target content role information and target scene role information, wherein the target content role information is used to represent the named entity information of the voice command, and the target scene role information is used to represent the business category corresponding to the target text; based on N interface elements contained in the current user interface, structured metadata corresponding to each interface element is generated, where N is a positive integer. The structured metadata includes the display text information of the corresponding interface element for matching with the content role information, and the business tag information to which the corresponding interface element belongs for matching with the scene role information; based on the target information and the structured metadata of each interface element, a matching calculation is performed, and based on the matching calculation result, the target interface element hit by the voice command is determined from the N interface elements; based on the voice command, an interactive operation with the target interface element is triggered.

[0020] The solution described in this specification constructs structured metadata for each interface element in the current user interface, enabling the capture and immediate reference of new and trending words appearing on the current user interface, thus providing a foundation for achieving visual and eloquent communication of the current user interface. Furthermore, when a user interacts with any interface element in the current user interface, the semantics of the user's voice commands are accurately perceived by extracting target content role information and / or target scene role information from the user's voice commands. This target information is then matched with the structured metadata of each interface element to precisely locate the target interface element currently being interacted with, completing the final interaction and improving the accuracy of voice interaction.

[0021] To better understand the above technical solutions, the technical solutions of the embodiments of this specification will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this specification and the specific features in the embodiments are detailed descriptions of the technical solutions of the embodiments of this specification, rather than limitations on the technical solutions of this specification. Unless otherwise specified, the embodiments of this specification and the technical features in the embodiments can be combined with each other.

[0022] First, it should be clarified that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0023] This specification provides an embodiment of a voice interaction method, such as... Figure 1 As shown, the method includes the following steps: Step S101: If a user's voice command is received, extract target information from the target text corresponding to the voice command. The target information includes at least one of the following: target content role information and target scene role information, wherein the target content role information is used to characterize the named entity information of the voice command, and the target scene role information is used to characterize the business category to which the interactive intent corresponding to the target text belongs. Step S102: Based on the N interface elements contained in the current user interface, generate structured metadata corresponding to each interface element, where N is a positive integer. The structured metadata includes the display text information of the corresponding interface element for matching with the content role information, and the business tag information to which the corresponding interface element belongs for matching with the scene role information. Step S103: Perform matching calculations based on the target information and the structured metadata of each interface element, and determine the target interface element hit by the voice command from the N interface elements based on the matching calculation results; Step S104: Trigger an interactive operation with the target interface element based on the voice command.

[0024] The solutions described in this specification can be applied to terminal devices equipped with a display screen and capable of voice control, such as vehicles, user smart devices, and home appliances. It should be noted that, considering the need to analyze user voice commands in the process of achieving "visual communication," uploading voice commands to a cloud server for analysis could potentially leak user privacy. Furthermore, the transmission of voice commands and the feedback of analysis results both rely on the network, resulting in significant latency and a poor user experience. Therefore, the methods provided in this specification can be executed on a terminal. For ease of explanation, an application to a vehicle is used as an example to illustrate the methods provided in this specification. Of course, in other embodiments, the methods can also be applied to the cloud or a system consisting of the cloud and a terminal.

[0025] Taking its application in vehicles as an example, vehicles are equipped with a central control screen, on which various applications can be installed, such as browsers, navigation, music apps, news apps, etc. Each application has its own user interface after being launched, and users in the car can trigger interactive operations by speaking any visible content on the user interface.

[0026] In some embodiments, the see-and-say function in a vehicle can be activated through means such as a user's voice control or a physical touch function start key, or the see-and-say function automatically runs by default when the vehicle is started. In step S101, the user's voice instruction may be any voice instruction issued by the user. In some embodiments, the user's voice instruction may be a voice instruction directed at any visible content on the vehicle's central control screen. For example, when multiple news headlines are currently displayed on the central control screen, the user's voice instruction is "Click on that news about celebrity A". After receiving the user's voice instruction, the voice instruction is converted into a corresponding target text, and target information is extracted from the target text.

[0027] In some embodiments, the text conversion of the voice instruction can be implemented through a preset automatic speech recognition model, that is, the user's voice instruction is input into the preset automatic speech recognition model, and the corresponding text is output. The preset automatic speech recognition model may be a Hidden Markov Model, a Transformer speech model, or the like, which is not limited herein. When training the preset automatic speech recognition model, the training data may include voice audios and corresponding text annotations. The initial automatic speech recognition model is iteratively trained based on the training data, and model parameters are adjusted until a preset number of iterations is reached or the accuracy of the model output is greater than a threshold, so as to complete the model training.

[0028] The voice instruction is recognized through the preset automatic speech recognition model to obtain the original text corresponding to the voice instruction. In the embodiments of this specification, the original text may be directly used as the target text, or the original text may be subjected to standardization processing, and the text after the standardization processing is used as the target text.

[0029] In some embodiments, performing standardization processing on the original text of the voice instruction may include, but is not limited to, lowercase conversion, function word filtering, and the like. The lowercase conversion refers to converting the numbers in the original text into lowercase, for example, converting "three ten" into "3:10".

[0030] In some embodiments, for the function word filtering process, a function word filtering dictionary may be pre-configured, and the dictionary may cover common semantically meaningless components in the Chinese context, including but not limited to: auxiliary words, prepositions, interjections, and conjunctions. Examples of these function word types are as follows: auxiliary words: such as "de", "di", and "de"; prepositions: such as "about", "for", and "in"; interjections: such as "ah", "oh", and "um"; conjunctions: such as "and", "with", and "as well as". By removing these interference items, the user's colloquial and redundant expressions can be refined into core semantic units, and only the target information such as content and scenarios for locating interface elements is retained, which greatly simplifies the subsequent matching processing logic and improves the matching accuracy.

[0031] In the example provided in this manual, in order to achieve accurate positioning of interface elements through target text, a semantic role hierarchical representation system was constructed, which decouples the user's voice commands into one or more semantic role dimensions.

[0032] Specifically, after obtaining the target text, target information is extracted. Target information includes at least one of the following: target content role information and target scenario role information. The target content role information carries the specific named entity or unique identifier mentioned in the user's command. Words in the text that have a unique name or specifically refer to a particular matter can be used as named entities, including names of people, places, and organizations. For example, if the user's voice command is "Click on the news about celebrity A," the extracted target content role information would be "celebrity A." The target scenario role information describes the business category or functional context to which the interactive intent of the target text belongs. For example, business categories can include different interactive service categories such as news, music, video, and advertising. Again, using the voice command "Click on the news about celebrity A" as an example, the extracted target content role information would be "news."

[0033] In some embodiments, the target information may further include target referential information, that is, text extracted from the target text to represent indicative semantic information is used as the target referential information. The target referential information is used to provide indicative or spatial semantics in the interaction, such as "this," "that," "first," "top left," etc.

[0034] In the embodiments of this specification, the aforementioned target information can be extracted in various ways, such as through rule matching or a preset semantic recognition model. Taking a preset semantic recognition model as an example, the target text can be input into the preset semantic recognition model, processed by the model, and the target information can be output. The preset semantic recognition model can be a BERT model, a large language model, etc. During the training process of the model, the training data can include the voice command text and the corresponding target information, including content role information, scene role information, and referential role information. Based on the training data, the initial semantic recognition model is iteratively trained, and the model parameters are adjusted until a preset number of iterations is reached or the accuracy of the model output is greater than a threshold, thus completing the model training.

[0035] Taking the target information as including content role information, target scene role information and target referent role information as an example, if the input target text is "click on the news about celebrity A", then the output target information is ["celebrity A", "that", "news"].

[0036] In step S102, the N interface elements included in the current user interface can be all currently visible elements. For example, if the current user interface is the interface of a news app, then all visible elements such as news items, advertisements, and images in the interface belong to the interface elements of that interface. Since users can interact with any interface element, in order to cover all interactive interface elements, in this embodiment of the specification, corresponding structured metadata is generated for each interface element. In some embodiments, the structured metadata includes at least the display text information and business tag information of each interface element, wherein the display text information is used to match the target content role information, and the business tag information is used to match the target scene role information.

[0037] The generation of structured metadata can be achieved in various ways. In some embodiments, structured metadata can be obtained through the following steps: obtaining the target rendering tree corresponding to the current user interface, wherein the target rendering tree contains N nodes corresponding to the N interface elements, and each node corresponds to the original text and control type of the node; for each node, using the original text of the node as the display text information of the node, and determining the business tag information of the node based on the control type of the node; and constructing the structured metadata of the interface element corresponding to the node based on the display text information and the business tag information of the node.

[0038] Specifically, the target rendering tree is a tree structure corresponding to the current user interface. It includes multiple nodes, each corresponding to a UI element. Each node has its own node data, such as the original text, control type, and unique identifier. The original text of a node is the original text content of the UI element it corresponds to. For example, if a UI element is a news item, then the title text of the news item is the original text of the corresponding node. The control types of nodes can include text controls, toggle controls, slider controls, image controls, button controls, icon controls, etc. The unique identifier of a node can be its ID.

[0039] In this embodiment, the original text of each node can be directly extracted from the target rendering tree and used as the display text information (title) of that node. Additionally, the control type of each node can be extracted from the target rendering tree, and the business tag information (category) of that node can be determined based on the control type. The business tag information can be used to characterize the business type to which the interface element belongs, and can include tags such as "news," "advertisement," and "music."

[0040] In some embodiments, a mapping relationship between control types and business tag information can be established. After extracting the control type of a node in the target rendering tree, the corresponding business tag information can be obtained by querying the mapping relationship. In other embodiments, if a unique business tag information cannot be determined by querying the mapping relationship, the original text of the node can be comprehensively analyzed. The original text can be semantically classified to determine the business to which the original text belongs. Then, the final business tag information of the node can be determined by combining the business to which the original text belongs and the candidate business tag information corresponding to the control type.

[0041] For each node, the structured metadata of the corresponding UI element can include the node's display text information and business label information. Of course, the structured metadata can also include other information, such as node ID, control type, etc., which are not limited here.

[0042] In this embodiment of the specification, considering that the obtained target rendering tree may have missing fields, resulting in the inability to obtain the structured metadata of each interface element, the following steps can be performed in this embodiment of the specification to ensure that the structured metadata of all interface elements is obtained: if missing fields are detected in the target rendering tree, a temporary screen frame corresponding to the current user interface is obtained; the temporary screen frame is input into a preset visual language model, and the display text information and business tag information of each element contained in the temporary screen frame are output; based on the display text information and business tag information of each element contained in the temporary screen frame, the missing fields of the target rendering tree are completed, so as to determine the structured metadata of each interface element based on the completed target rendering tree.

[0043] Specifically, a temporary screen frame is a single image frame temporarily stored during screen display. The temporary screen frame can capture the current user interface, and in some embodiments, the content of the temporary screen frame can be the same as the content of the current user interface. A preset Vision-Language Model (VLM) can be deployed in the vehicle. This preset VLM can process the input image and output the display text information and business label information for each interface element in the image. In some embodiments, the preset VLM deployed in the vehicle can be a lightweight model to avoid occupying too much storage space on the vehicle and to reduce response latency.

[0044] Using a pre-defined visual language model, the system can output the display text and business tag information for each interface element contained in a temporary screen frame. Furthermore, for display text and / or business tag information for one or more interface elements that cannot be extracted from the target rendering tree, the output of the pre-defined visual language model can be used to complete the data. This allows the extraction of structured metadata for each interface element from the completed target rendering tree.

[0045] In addition, to ensure the real-time nature of structured metadata, temporary screen frames are used only in memory for a single inference of the preset visual language model. Once the preset visual language model has finished processing the temporary screen frame, it is immediately released and not written to any storage medium.

[0046] In the embodiments of this specification, the structured metadata of each interface element in the current user interface can be reused when the current user interface remains unchanged, avoiding repeated calculations and wasting computing resources.

[0047] In some embodiments, when extracting structured metadata using both a target rendering tree and a preset visual language model, the following situation may occur: the target rendering tree contains interface element A, and the structured metadata of interface element A extracted from the target rendering tree is data a. However, after processing a temporary screen frame using the preset visual language model, the extracted structured metadata of interface element A is data b. This means the structured metadata of the same interface element obtained through the two methods is different, indicating a data conflict. Considering that the output of the preset visual language model can tolerate a certain degree of error, when the metadata obtained by the two methods conflicts, the result obtained from the target rendering tree is given priority. Only when the target rendering tree has missing data is the output of the preset visual language model used.

[0048] In other embodiments, instead of using a target rendering tree, a preset visual language model can be directly selected to process temporary screen frames and obtain structured metadata for all interface elements.

[0049] It should be noted that the generation process of structured metadata can also be selected based on the needs of the vehicle system. For example, extracting structured metadata through a rendering tree can be applied to all vehicle system platforms, including those with low computing power; using a preset visual language model can be applied to vehicle system platforms with high computing power; and using a fusion mode of rendering tree and preset visual language model can be applied to high reliability scenarios (such as driver voice interaction scenarios).

[0050] In step S103, the target information is matched and calculated with the structured elements of each interface element. This can be done by matching the semantic roles of each dimension contained in the target information, thereby achieving a structured alignment between the semantic components of the user's voice command and the interface elements, and achieving accurate positioning of the interface elements.

[0051] In some embodiments, taking the target information including target content role information, target scene role information, and target referential role information as an example, the target information matching can be achieved through the following steps: For the structured metadata of each interface element, the target content role information is matched with the display text information corresponding to the interface element to obtain a first matching result; the target scene role is matched with the business tag information corresponding to the interface element to obtain a second matching result; the target referential role is matched with a preset referential rule lexicon to obtain a third matching result; the first matching result, the second matching result, and the third matching result are fused and calculated, and the fused calculation score is used as the matching result between the interface element and the target information.

[0052] Specifically, taking the voice command "click on the news article about celebrity A" as an example, the extracted target content character information, target scene character information, and target referent character information are "celebrity A", "that article", and "news article" respectively.

[0053] If the current user interface includes interface element C1 and interface element C2, where interface element C1 is news with the original headline "Celebrity A Delivers Takeout Late at Night" and interface element C2 is an advertisement with the original headline "Celebrity A's Same Model Electric Vehicle Limited-Time Offer," then construct the structured metadata for interface elements C1 and C2. This will give interface element C1 the display text "Celebrity A Delivers Takeout Late at Night," the business tag "News," and the interface element ID "001." Similarly, interface element C2 will have the display text "Celebrity A's Same Model Electric Vehicle Limited-Time Offer," the business tag "Advertisement," and the interface element ID "008."

[0054] When matching target information with interface element C1, the target content role information "Celebrity A" is matched with the display text information "Celebrity A delivers takeout late at night" of interface element C1. Since the display text information completely contains the target content role information, the first matching result is 1, i.e., a complete match. The target scene role information "News" is matched with the business tag information "News" of interface element C1. Since they are completely identical, the second matching result is 1. The target referential role information is matched with a preset referential rule dictionary. If the maximum similarity between the target referential role information and all referential words in the preset referential rule dictionary is 0.5, then the third matching result is determined to be 0.5. The preset referential rule dictionary can be a pre-constructed set containing general referential words. It should be noted that when matching each type of role information, either full-word matching or similarity matching can be used; there is no limitation here.

[0055] When merging the first, second, and third matching results, a weighted fusion can be selected. For each character information, a corresponding weight can be pre-set. For example, the weight for the first matching result is 0.6, the weight for the second matching result is 0.3, and the weight for the third matching result is 0.5. Of course, the specific values ​​of each weight can be selected according to actual needs and are not limited here. Taking the above weight values ​​of 0.6, 0.3, and 0.5 as an example, the final fusion score of the target information and interface element C1 is 0.6×1 + 0.3×1 + 0.1×0.5 = 0.95.

[0056] Accordingly, when matching the target information with interface element C2, the target content role information "Celebrity A" is matched with the display text information "Celebrity A's same model electric car limited-time special offer" of interface element C2. Since the display text information completely contains the target content role information, the first matching result is 1, i.e., a complete match. The target scene role information "News" is matched with the business tag information "Advertisement" of interface element C2. Since the two are completely different, the second matching result is 0. The target referential role information is matched with the preset referential rule dictionary. If the maximum similarity between the target referential role information and all referential words in the preset referential rule dictionary is 0.5, then the third matching result is determined to be 0.5. It should be noted that when performing each type of role information matching, either full-word matching or similarity matching can be used; there is no limitation here.

[0057] When fusing the first, second, and third matching results, the weight values ​​of 0.6, 0.3, and 0.5 mentioned above are still used. Therefore, the final fusion score of the target information and the interface element C2 is 0.6×1+0.3×0+0.1×0.5=0.65.

[0058] In this embodiment of the specification, based on the fusion calculation score between each interface element and the target information, a target score with the highest fusion calculation score that is greater than a preset score can be selected, and the interface element corresponding to the target score can be used as the target interface element. Continuing with the example above, the fusion calculation score between the target information and interface element C1 is greater than the fusion calculation score with interface element C2; therefore, interface element C1 can be determined as the target interface element hit by the voice command.

[0059] In addition, when multiple interface elements match the target information, further interaction with the user can be initiated. For example, the user can be asked to confirm the interface element they want to interact with again, and the interface element can be located based on the user's feedback.

[0060] In some embodiments, when matching the target scene role with the business tag information corresponding to the interface element, it can be achieved in the following way: based on the preset correspondence between scene role information and standard tag information, determine the target standard tag information corresponding to the target scene role information; for each interface element, match the target standard tag information with the business tag information corresponding to the interface element to obtain the second matching result.

[0061] Specifically, to improve adaptability to diverse expressions, embodiments of this specification can construct a preset correspondence between scene role information and standard tags, and periodically update this preset correspondence. Specifically, the standard tag information can include business tag information corresponding to all interface elements; that is, the business tag information corresponding to each interface element is standard tag information, such as "news," "music," and "advertisement." However, when users perform voice control, their voice commands are more colloquial and may not be standard tag information. Therefore, one or more scene role information can be set for each standard tag to construct its own mapping rules. For example, for the standard tag information "news," the corresponding scene role information could include "news," "information," and "headlines." In some embodiments, the scene role information in the preset correspondence can be obtained through statistical analysis of multiple voice data collected from real users. In some embodiments, the preset correspondence between scene role information and standard tag information can be updated in the cloud. For example, when the cloud detects that a new application has been launched or an existing application has undergone an interface redesign, it can update the preset correspondence based on the new interface. For example, it can add new standard tag information or add new scene role information to the existing standard tag information.

[0062] In the embodiments of this specification, after obtaining the target scene role information corresponding to the user's voice command, the corresponding target standard tag information can be obtained by searching the preset correspondence between the scene role information and the standard tag information. Then, the target standard tag information is matched with the business tag information corresponding to the interface element, so that the second matching result is more accurate.

[0063] Through the matching process above, it can be seen that the semantic validity of content role information is directly carried by the displayed text information of interface elements. A match is valid only if the content role information appears in the displayed text of the interface elements, thus achieving a completely consistent voice representation with the visual presentation. Scene role information is related to the business label information of the interface's original color, used to constrain the scope of the matched business and eliminate interference from inconsistent business information. Referential role information can provide indicative or spatial semantics in the interaction. Through the fusion of these three elements with multi-dimensional semantic coverage, precise positioning of interactive interface elements is achieved.

[0064] In this embodiment of the specification, if the display screen currently shows multiple candidate user interfaces, in order to quickly determine the current user interface, the following steps may also be performed: obtain the name of the application to which each candidate user interface belongs; if the target information contains the target scene role information, match the target scene role with the name of the application to which each candidate user interface belongs, and take the candidate user interface with the highest matching degree as the current user interface.

[0065] For example, if the screen currently displays Application 1 and User Interface 1, and Application 2 and User Interface 2, where Application 1 is named "XX News" and Application 2 is named "YY Music," and the user's voice command is "Click on the news article about celebrity A," the corresponding target scene role information is "news." The target scene information is matched with the name of each application to achieve initial filtering of the user interfaces. Since the target scene information perfectly matches "XX News," User Interface 1 of "XX News" can be selected as the current user interface.

[0066] In step S104, after determining the target interface element, an interactive operation with the target interface element can be initiated. In some embodiments, the target operation type of the voice command can be extracted from the target text. For example, if the user's voice command is "click on the news article about celebrity A," the operation intent is parsed from the target text, and the target operation type is "click." Then, based on the target operation type and the target interface element, an interactive command is generated and executed.

[0067] Using the example of interface elements C1 and C2 above, the target interface element obtained from the matching process is C1. We can extract the interface element ID "001" of C1, and generate an interaction instruction based on the target operation type "click" and the interface element ID "001": {action: "click", target: "001"}. Executing this interaction instruction will open the news details page corresponding to interface element C1.

[0068] The embodiments of this specification may also include the following steps: if a change event for the current user interface is detected, determine whether the voice command interaction has been completed; if the voice command interaction has been completed, extract the updated structured metadata corresponding to the change event, and perform the next user voice interaction based on the updated structured metadata.

[0069] Specifically, changes to the current user interface can include changes to any content on the interface (such as card loading, list scrolling, ad carousel), or opening a new user interface. In some embodiments, for Android systems, this can be achieved by listening to the TYPE WINDOW CONTENT CHANGED notification; for iOS systems, it can be achieved by listening to the ScreenChanged notification. If the user's voice command interaction has already been completed when the event change is detected, the event change will not affect the voice interaction process. Therefore, the updated structured metadata corresponding to the change event can be extracted again, and the next user voice interaction can be based on the updated structured metadata.

[0070] It should be noted that when an event change is detected, if the user's voice command has not yet been completed, since the user's voice command corresponds to the structured metadata of each interface element before the event change, if the structured metadata is updated immediately when the event change is detected, it may result in the voice command being unable to match the structured metadata of the interface elements before the change. Therefore, in the embodiments of this specification, the structured metadata of the event change is updated after the voice command interaction is completed.

[0071] In some embodiments, after a change event is detected, structured metadata is asynchronously extracted during non-voice interaction periods (such as idle periods after voice interaction is completed) and cached. If a change is detected in the cache when the next voice command arrives, a synchronous refresh is immediately performed (e.g., only for convenient interface changes or running a partial VLM) to ensure that the matching process is based on the latest interface state. The structured metadata cache is valid until the next interface element change event, ensuring strict consistency between semantic matching and visual presentation.

[0072] To better understand the voice interaction method provided in the embodiments of this specification, the overall process of the embodiments of this specification is described below, which includes the following steps: Step 1: Receive the user's voice command, such as "Click on the news article about celebrity A"; Step 2: Call the automatic speech recognition model to convert the voice command into raw text; Step 3: Standardize the original text, including converting it to lowercase and filtering out function words, to obtain the standardized text; Step 4: Extract semantic role information from the standardized text, extracting content role information, i.e. named entities, which are used to anchor target interface elements; Step 5: Extract scene role information, i.e., business category keywords, for context constraints; Step Six: Extract referential information, i.e., indicative words and phrases, supporting natural referential expression; Step 7: Perform semantic matching based on content role information, scene role information, referential role information, and structured metadata of interface elements in the current user interface, and perform fusion calculation based on the matching results to obtain the fusion calculation score; Step 8: Locate the target interface element based on the fusion calculation score; Step 9: Perform interactive operations with the target interface elements.

[0073] Additionally, it should be noted that during voice interaction, besides using the "See and Say" function to achieve interaction between the user and the device, a general voice engine can also be used. For example, a voice engine deployed on the device terminal or in the cloud can perform speech recognition, semantic parsing, and command issuance for the user's voice commands. In some cases, the processing result of the voice engine on the user's voice command may differ from the processing result under the "See and Say" function. This is because the user's voice command may correspond to an interaction with a specific interface element on the display screen, or it may also correspond to an interaction intent other than an interface element. If the two conflict, in some embodiments, when the "See and Say" function is enabled, the interaction operation corresponding to the "See and Say" function can be executed first. If the "See and Say" function fails to determine the interaction operation corresponding to the user's voice command, the voice engine can determine and execute the interaction operation corresponding to the user's voice command.

[0074] In summary, the solution implemented in this specification, under the "what you see is what you speak" function, eliminates the need for users to match all words to the target interface element. Users only need to speak a fragment or part of the interface element to accurately locate the interactive target element. Furthermore, by constructing structured metadata for each interface element in the current user interface and automatically updating the metadata after new content is rendered, any new or trending words appearing on the interface can be referenced in real time, eliminating the need for a pre-built keyword library and preventing issues with new trending words failing to match pre-built keywords. This solution, by extracting content roles, scene roles, and referential roles, achieves complete semantic understanding and high-confidence disambiguation of natural spoken language, better adapting to the flexible expression of user voice commands and accurately locating the interactive target interface element. Simultaneously, this solution can be completed on the device side without data upload, significantly reducing the latency of voice interaction.

[0075] Based on the same inventive concept, embodiments of this specification also provide a voice interaction device, such as... Figure 2 As shown, the device includes: The target information extraction module 201 is used to extract target information from the target text corresponding to the voice command when a user's voice command is received. The target information includes at least one of the following: target content role information and target scene role information, wherein the target content role information is used to characterize the named entity information of the voice command, and the target scene role information is used to characterize the business category to which the interactive intent of the target text belongs. The structured metadata generation module 202 is used to generate structured metadata corresponding to each interface element based on the N interface elements contained in the current user interface, where N is a positive integer. The structured metadata includes the display text information of the corresponding interface element for matching with content role information, and the business tag information to which the corresponding interface element belongs for matching with scene role information. The processing module 203 is used to perform matching calculations between the target information and the structured metadata of each interface element, and determine the target interface element hit by the voice command from the N interface elements based on the matching calculation results. Trigger module 204 is used to trigger interactive operations with the target interface elements based on the voice command.

[0076] Regarding the above-mentioned device, the specific implementation of each step has been described in detail in the embodiments of the voice interaction method provided in the specification, and will not be elaborated here.

[0077] Based on the same inventive concept, embodiments of the present invention also provide a vehicle electric suction door control device, such as... Figure 3The aforementioned includes a memory 304, a processor 302, and a computer program stored in the memory 304 and executable on the processor 302. When the processor 302 executes the program, it implements any of the above-described voice interaction methods.

[0078] Among them, Figure 3 In this document, a bus architecture (represented by bus 300) is used. Bus 300 may include any number of interconnected buses and bridges, linking various circuits including one or more processors represented by processor 302 and memory represented by memory 304. Bus 300 may also link various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. Bus interface 305 provides an interface between bus 300 and receiver 301 and transmitter 303. Receiver 301 and transmitter 303 may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices over a transmission medium. Processor 302 is responsible for managing bus 300 and general processing, while memory 304 can be used to store data used by processor 302 during operation.

[0079] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0080] Based on the same inventive concept, embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described voice interaction method.

[0081] Based on the same inventive concept, embodiments of this specification provide a computer program product, which includes a computer program that, when executed by a processor, is used to load and execute the steps of the above-described voice interaction method.

[0082] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored as one or more instructions or code on or transmitted via a computer-readable medium. Other examples and embodiments are within the scope of this invention and the appended claims. For example, due to the nature of software, the functions described above can be implemented using software executed by a processor, hardware, firmware, hardwired, or any combination thereof. Furthermore, the functional units can be integrated into a single processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0084] The units described as separate components may or may not be physically separate. Similarly, the components of the control device may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0086] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A voice interaction method, characterized in that, include: If a user's voice command is received, target information is extracted from the target text corresponding to the voice command. The target information includes at least one of the following: target content role information and target scene role information, wherein the target content role information is used to characterize the named entity information of the voice command, and the target scene role information is used to characterize the business category to which the interactive intent of the target text belongs. Based on the N interface elements contained in the current user interface, generate structured metadata corresponding to each interface element, where N is a positive integer. The structured metadata includes the display text information of the corresponding interface element for matching with content role information, and the business tag information to which the corresponding interface element belongs for matching with scene role information. The target information is matched and calculated with the structured metadata of each interface element, and the target interface element hit by the voice command is determined from the N interface elements based on the matching calculation results. The voice command triggers an interactive operation with the target interface element.

2. The method as described in claim 1, characterized in that, The target information also includes target referential role information, and the extraction of target information from the target text corresponding to the voice command includes: The text used to represent indicative semantic information is extracted from the target text and used as the target referential role information.

3. The method as described in claim 2, characterized in that, The step of generating structured metadata corresponding to each of the N interface elements contained in the current user interface includes: Obtain the target rendering tree corresponding to the current user interface, wherein the target rendering tree contains N nodes corresponding to the N interface elements, and each node corresponds to the original text and control type of the node; For each node, the original text of the node is used as the display text information of the node, and the business tag information of the node is determined based on the control type of the node; and the structured metadata of the interface element corresponding to the node is constructed based on the display text information and business tag information of the node.

4. The method as described in claim 3, characterized in that, After obtaining the target rendering tree corresponding to the current user interface, the method further includes: If a missing field is detected in the target rendering tree, obtain the temporary screen frame corresponding to the current user interface; The temporary screen frame is input into a preset visual language model, and the display text information and business tag information of each element contained in the temporary screen frame are output. Based on the display text information and business tag information of each element contained in the temporary screen frame, the missing fields of the target rendering tree are filled in, so as to determine the structured metadata of each interface element based on the filled target rendering tree.

5. The method as described in claim 3, characterized in that, If the target information includes target content role information, target scene role information, and target referential role information, the matching calculation based on the target information and the structured metadata of each interface element includes: For the structured metadata of each interface element, the target content role information is matched with the display text information corresponding to the interface element to obtain a first matching result; the target scene role information is matched with the business tag information corresponding to the interface element to obtain a second matching result; and the target referential role information is matched with a preset referential rule lexicon to obtain a third matching result. The first matching result, the second matching result, and the third matching result are then fused and calculated, and the fused calculation score is used as the matching result between the interface element and the target information.

6. The method as described in claim 5, characterized in that, The step of matching the target scene role information with the business tag information corresponding to the interface element to obtain a second matching result includes: Based on the preset correspondence between scene role information and standard tag information, the target standard tag information corresponding to the target scene role information is determined; For each interface element, the target standard tag information is matched with the business tag information corresponding to that interface element to obtain the second matching result.

7. The method as described in claim 5, characterized in that, The step of determining the target interface element hit by the voice command from the N interface elements based on the matching calculation result includes: Based on the fusion calculation score between each interface element and the target information, the target score with the highest fusion calculation score that is greater than the preset score is selected, and the interface element corresponding to the target score is taken as the target interface element.

8. The method as described in claim 1, characterized in that, The method further includes: If multiple candidate user interfaces are currently displayed, obtain the name of the application to which each candidate user interface belongs; If the target information contains the target scene role information, the target scene role information is matched with the name of the application to which each candidate user interface belongs, and the candidate user interface with the highest matching degree is taken as the current user interface.

9. The method as described in claim 1, characterized in that, The method further includes: If a change event for the current user interface is detected, determine whether the voice command interaction has been completed; If the voice command interaction has been completed, extract the updated structured metadata corresponding to the change event, and perform the next user voice interaction based on the updated structured metadata.

10. The method as described in claim 1, characterized in that, The interaction operation triggered by the voice command with the target interface element includes: Extract the target operation type of the voice command from the target text; Based on the target operation type and the target interface element, generate and execute interactive instructions.

11. A voice interaction device, characterized in that, include: The target information extraction module is used to extract target information from the target text corresponding to the voice command when a user's voice command is received. The target information includes at least one of the following: target content role information and target scene role information, wherein the target content role information is used to characterize the named entity information of the voice command, and the target scene role information is used to characterize the business category to which the interactive intent of the target text belongs. The structured metadata generation module is used to generate structured metadata corresponding to each of the N interface elements contained in the current user interface, where N is a positive integer. The structured metadata includes the display text information of the corresponding interface element for matching with content role information, and the business tag information to which the corresponding interface element belongs for matching with scene role information. The processing module is used to perform matching calculations between the target information and the structured metadata of each interface element, and determine the target interface element hit by the voice command from the N interface elements based on the matching calculation results. The triggering module is used to trigger interactive operations with the target interface elements based on the voice command.

12. A voice interaction device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The device contains a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-10.

Images