A voice interaction method, device, electronic device, storage medium, and program product

Abstract

This invention relates to the field of human-computer interaction technology, and more particularly to a voice interaction method, device, electronic device, storage medium, and program product. The method may include: in response to a user's voice command for a target device, obtaining context, wherein the context describes the semantic attributes of controls and / or the semantic relationships between controls, and the controls are those currently displayed on the screen of the target device; performing voice recognition on the user's voice command based on the context; and performing intent understanding on the voice recognition result based on the context, to generate control operation instructions based on the intent understanding result, wherein the control operation instructions instruct the target control to be operated among the controls and the operation action to be performed on the target control. The technical solution of this invention can improve the accuracy of voice interaction.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of human-computer interaction technology, and in particular to a voice interaction method, device, electronic device, storage medium and program product. Background Technology

[0002] With the increasing popularity of smart home appliances (such as smart ovens), smart home appliances with touch screens not only support touch interaction but also voice interaction, thereby meeting users' increasingly natural operating needs.

[0003] In the process of realizing this invention, the inventors discovered the following technical problems in the prior art: the accuracy of voice interaction is not high, which urgently needs to be solved. Summary of the Invention

[0004] This invention provides a voice interaction method, device, electronic device, storage medium, and program product, which solves the problem of low accuracy in voice interaction.

[0005] According to one aspect of the present invention, a voice interaction method is provided, which may include:

[0006] In response to a user voice command for the target device, obtain the context, where the context is used to describe the semantic attributes of the controls and / or the semantic relationships between the controls, and the controls are the controls currently displayed on the screen of the target device;

[0007] Based on context, perform speech recognition on user voice commands;

[0008] Based on the context, the speech recognition results are subjected to intent understanding, and control operation instructions are generated based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each control and the operation action to be performed on the target control.

[0009] According to another aspect of the present invention, a voice interaction device is provided, which may include:

[0010] The context acquisition module is used to acquire the context in response to user voice commands for the target device. The context is used to describe the semantic attributes of the controls and / or the semantic relationships between the controls. The controls are the controls currently displayed on the screen of the target device.

[0011] The speech recognition module is used to perform speech recognition on user voice commands based on context.

[0012] The intent understanding module is used to understand the intent of speech recognition results based on context, and to generate control operation instructions based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each control and the operation action to be performed on the target control.

[0013] According to another aspect of the present invention, an electronic device is provided, which may include:

[0014] At least one processor; and

[0015] A memory that is communicatively connected to at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by at least one processor, such that when the at least one processor executes the program, it implements the voice interaction method provided in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided having computer instructions stored thereon for causing a processor to execute and implement the voice interaction method provided in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the voice interaction method provided in any embodiment of the present invention.

[0019] The technical solution of this invention, in response to a user's voice command for a target device, obtains context, which describes the semantic attributes of controls and / or the semantic relationships between controls. The controls are those currently displayed on the target device's screen. Then, based on the context, the user's voice command is recognized. Based on the context, the voice recognition result is interpreted to generate control operation instructions, which instruct the target control to be operated and the operation action to be performed on the target control. This technical solution, by transforming screen controls into semantic context and injecting the context as a constraint into the voice recognition and intent understanding process, enables the system to perform voice recognition and intent understanding based on its ability to "understand" screen content. This achieves a precise "what you see is what you say, what you say is what you get" voice interaction experience, improving the accuracy of voice interaction.

[0020] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a voice interaction method provided according to an embodiment of the present invention;

[0023] Figure 2 This is a flowchart of another voice interaction method provided according to an embodiment of the present invention;

[0024] Figure 3 This is a flowchart of yet another voice interaction method provided according to an embodiment of the present invention;

[0025] Figure 4 This is a flowchart of another voice interaction method provided according to an embodiment of the present invention;

[0026] Figure 5 This is a schematic diagram of a visual and verbal interactive system based on dynamic screen perception, provided by an embodiment of the present invention.

[0027] Figure 6 This is a structural block diagram of a voice interaction device according to an embodiment of the present invention;

[0028] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the voice interaction method of this invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The same applies to "target," "original," etc., and will not be repeated here. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution of this invention all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to maintain user personal information security and network security.

[0032] Figure 1 This is a flowchart illustrating a voice interaction method provided in an embodiment of the present invention. This embodiment is applicable to voice interaction between a user and a target device, and particularly to voice interaction between a user and smart home appliances. The method can be executed by the voice interaction device provided in this embodiment, which can be implemented in software and / or hardware. The device can be integrated into an electronic device, which can be various user terminals or servers.

[0033] See Figure 1 The method of this invention specifically includes the following steps:

[0034] S110. In response to a user voice command for the target device, obtain the context, wherein the context is used to describe the semantic attributes of the controls and / or the semantic relationships between the controls, and the controls are the controls currently displayed on the screen of the target device.

[0035] The target device can be understood as an electronic device (such as a smart home appliance, wearable device, or smart robot) that the user instructs to interact with via voice. In this embodiment of the invention, optionally, the target device carries a screen (such as a touch screen), on which interactive controls (such as one or more of buttons, switches, sliders, list items, steppers, text labels, progress bars, and prompt boxes) are currently displayed.

[0036] User voice commands can be understood as instructions issued by the user via voice to operate the aforementioned controls. Taking smart home appliances as an example, such instructions could be, for example, "set the oven temperature to 200 degrees" or "continue," depending on the actual situation, and no specific limitation is made here.

[0037] Semantic attributes can be understood as objective data dimensions that characterize the interactive meaning and functional features of the corresponding control. In this embodiment of the invention, optional attributes may include one or more of the following: control type, text label, location information, function identifier, current state, and operable state, which are used to define the independent semantic identity of the control in the interactive scenario.

[0038] Semantic relationships can be understood as structured associations between controls due to one or more of spatial layout, hierarchical affiliation, logical subordination, and operational sequence dependence. In this embodiment of the invention, optional associations may include one or more connection types such as spatial adjacency, container inclusion, hierarchical nesting, operational sequence, and functional complementarity, which are used to characterize the position, role, and mutual constraints of controls in the overall semantic environment of the screen.

[0039] Based on this, the context can be understood as structured semantic information constructed in real time based on the screen-side user interface (UI), used to describe the semantic attributes of controls and / or the semantic relationships between controls. In this embodiment of the invention, optionally, the context can be represented based on one or more of the following methods: graph, natural language, logical rules, linear tags, and structured objects, without specific limitations; further optionally, as described above, the context can characterize screen semantics, and therefore can also be called a screen semantic context. Based on this, the voice interaction method described in this embodiment of the invention can also be called a voice interaction method based on screen-side semantic context, achieving voice interaction based on screen-side semantics.

[0040] In response to a user's voice command, the context is obtained. For example, after detecting a user's voice command, the context can be directly obtained from the existing context, reconstructed from the existing context, or incrementally updated from the context constructed in the previous frame to obtain the latest context, etc., without specific limitations.

[0041] In this step, by obtaining the context describing the semantic attributes of controls and the semantic relationships between controls, screen visual constraints are injected into subsequent speech recognition and intent understanding. This allows the system to process based on the current real UI information without relying on general priors, fundamentally reducing processing deviations caused by unknown scenarios.

[0042] S120. Based on context, perform speech recognition on user voice commands.

[0043] In this process, the context is used as a dynamic knowledge base (or constrained by the context) to perform speech recognition on the user's voice commands, so as to convert the user's voice signal into a text sequence.

[0044] For example, dynamic hot word injection involves dynamically loading all text tags and function descriptions extracted from the context as temporary hot words into the speech recognizer's decoder. For instance, when the screen displays "pasteurization," this technical term is temporarily assigned a very high weight to significantly improve its recognition accuracy.

[0045] As another example, dynamic biasing of the language model: Based on the current business function of the interface (such as "selection of food preparation mode"), the speech recognition language model is dynamically biased to make it more inclined to decode word sequences related to the business function, so as to improve the accuracy of speech recognition.

[0046] Another example is visual-speech joint encoding and alignment: cross-modal joint encoding of the semantics of UI controls represented by the context and user voice commands is performed to establish a fine-grained alignment mapping relationship at the control level, and speech recognition is performed based on this mapping relationship to improve the accuracy of speech recognition.

[0047] In this step, decoding constraints are performed using screen-specific semantic attributes and / or semantic relationships in the context, enabling the speech recognition engine to obtain vocabulary preferences in vertical scenarios and perform speech recognition based on these preferences. This significantly improves the recognition accuracy of user voice commands involving specific terms, abbreviations, and functional descriptions.

[0048] S130. Based on the context, perform intent understanding on the speech recognition results, and generate control operation instructions based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each control and the operation action to be performed on the target control.

[0049] The speech recognition result can be understood as the text sequence output after the speech recognition in the previous step. It may contain one or more difficult-to-understand contents such as referential words (such as this, that, the left and the second, etc.), vague descriptions (such as continue), and colloquial action expressions (such as turn up and turn on, etc.). This is related to the actual situation and no specific limitation is made here.

[0050] Control operation instructions can be understood as executable instructions for UI controls generated based on intent recognition results, which explicitly specify the target control to be operated on and the specific operation to be performed on the target control.

[0051] Constrained by context, the speech recognition results are subjected to intent understanding, which means parsing the recognized text sequence into the target control of the user's intent operation and the operation action to be performed on the target control (such as clicking, swiping, inputting values ​​or switching states). Specifically, this may include aligning ambiguous references with specific controls in the context, and mapping colloquial expressions of actions to standardized operation actions supported by the target device.

[0052] For example, referential resolution: when the user says "open the second one", the "second one" is resolved into a specific control ID by combining the control list and positional relationship in the context, thus obtaining the target control.

[0053] Another example is fuzzy intent alignment: when a user says "Continue", the semantic similarity between the user's statement (i.e., the speech recognition result) and all operable controls in the context is calculated, the "Next" button is found as the target control, and it is understood that "Continue" means to advance the process.

[0054] Another example is multi-turn context enhancement: multi-turn historical context and voice dialogue history are jointly modeled in a unified spatiotemporal dimension to assist cross-modal alignment in the current turn and perform intent understanding based on this.

[0055] In this step, by leveraging the semantic attributes of controls and / or the semantic relationships between controls provided by the context, intent understanding can accurately anchor vague references to a unique control and standardize colloquial action expressions into standard operation actions, breaking the limitation of traditional solutions that cannot accurately interpret user intent due to a lack of visual context awareness.

[0056] Thus, it can be seen that there is explicit alignment between user voice commands and UI controls, which can establish a complete mapping link from user voice command to target control to execution of operation, thereby achieving end-to-end alignment.

[0057] Based on this, optionally, for the target control represented by the intent understanding result, the operability of the target control is verified based on the context; in response to the operability representation that the target control is operable, control operation instructions are generated based on the intent understanding result.

[0058] The intent understanding result can be understood as the structured information output after performing intent understanding on the speech recognition result, which clearly identifies the target control of the user's intended operation.

[0059] The context records the enabled state (i.e., operable state) of each control. Therefore, the operability of the target control can be verified based on the context, that is, whether the target control has the conditions to accept and respond to interactive actions in the current interface. Optionally, these conditions can be one or more of the following: the target control is not disabled, it is in the visible area, it is not in a read-only state, and it meets the preconditions for performing the operation. These can be set according to actual needs and are not specifically limited here. Furthermore, if the target control is verified to be operable, control operation instructions can be generated based on the intent understanding result.

[0060] The above technical solution improves the reliability of voice interaction by adding a context-based state verification step between intent understanding and instruction generation. It utilizes the real-time semantic attributes of controls in the context to perform pre-verification of controls, ensuring that control operation instructions are generated only for currently usable and interactive controls.

[0061] The technical solution of this invention, in response to a user's voice command for a target device, obtains context, which describes the semantic attributes of controls and / or the semantic relationships between controls. The controls are those currently displayed on the target device's screen. Then, based on the context, the user's voice command is recognized. Based on the context, the voice recognition result is interpreted to generate control operation instructions, which instruct the target control to be operated and the operation action to be performed on the target control. This technical solution, by transforming screen controls into semantic context and injecting the context as a constraint into the voice recognition and intent understanding process, enables the system to perform voice recognition and intent understanding based on its ability to "understand" screen content. This achieves a precise "what you see is what you say, what you say is what you get" voice interaction experience, improving the accuracy of voice interaction.

[0062] Based on this, as an alternative technical solution, the above-mentioned voice interaction method may further include:

[0063] Send control operation instructions to the target device so that the target device can provide feedback after executing the control operation instructions, wherein the feedback content includes at least one of the following:

[0064] Visual feedback for the target control, voice feedback for the target control, and voice feedback for user voice commands are at least one of the following:

[0065] In this technical solution, the target device, especially its UI system, can provide feedback to the user after executing the control operation command, allowing the user to be aware of and confirm the operation result. Optionally, the feedback content may include one or more of the following three types:

[0066] 1) Visual feedback for the target control, such as applying one or more visual interface responses to the target control on the screen, such as highlighting, blinking, color change, scaling animation, and status icon update, to intuitively prompt the user that the target control has responded to the control operation command.

[0067] 2) Voice feedback for the target control, such as through voice broadcast, outputting natural language descriptions directly related to the operation result of the target control, with the content focusing on the control state change itself;

[0068] 3) Voice feedback for user voice commands, such as responding to the user's original command intent through voice broadcast (e.g., "Cooking has started"), with the content presented in a conversational confirmation form, indicating that the system has understood and completed the user's request, forming an interactive loop.

[0069] The above technical solution achieves a "speakable and visual" linkage feedback mechanism by issuing and executing control operation instructions and simultaneously providing visual linkage prompts and voice confirmation broadcasts for the target control. This allows users to instantly obtain operation results through both visual and auditory channels, significantly improving the user's voice interaction experience.

[0070] Figure 2 This is a flowchart of another voice interaction method provided by an embodiment of the present invention. This embodiment is based on and optimized from the above technical solutions. In this embodiment, optionally, based on context, voice recognition of user voice commands may include: for each control, extracting target information of the control from the context and assigning a first weight to the target information, wherein the target information includes at least one of text labels, function descriptions, function identifiers, and current states; and performing voice recognition of user voice commands based on each target information and the first weight. Alternatively, the control is displayed on the current interface, which is the interface currently displayed on the screen; and based on context, voice recognition of user voice commands includes: obtaining the business functions provided by the current interface according to the context; obtaining candidate words related to the business functions and assigning a second weight to the candidate words; and performing voice recognition of user voice commands based on the candidate words and the second weight. The above two optional solutions can be executed one or in parallel, which can be set according to actual needs and is not specifically limited here. Explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0071] See Figure 2 The method in this embodiment may specifically include the following steps:

[0072] S210. In response to a user voice command for a target device, obtain the context, wherein the context is used to describe the semantic attributes of controls and / or the semantic relationships between controls, the controls are those displayed on the current interface, and the current interface is the interface currently displayed on the screen.

[0073] S220. For each control, extract the target information of the control from the context and assign a first weight to the target information, wherein the target information includes at least one of text label, function description, function identifier and current state.

[0074] The target information can be understood as the semantic attributes of the control extracted from the context to assist speech recognition. In this embodiment of the invention, the target information may include one or more of the following: text label (i.e., visible text identifier displayed or associated on the control), function description (i.e., semantic description text of the purpose or operation result of the control), function identifier (i.e., identifier that uniquely identifies the function of the control), and current state (i.e., the mode or value of the control in the current interface, such as the "on / off" mode of the switch control, the "selected / unselected" mode of the check box, and the current value of the slider control).

[0075] The first weight can be understood as a numerical priority coefficient for the allocation of target information. It can be used to increase the tendency of these words to be selected during the speech recognition decoding process. The weight value is relatively higher than that of general words.

[0076] For example, all controls in the context are traversed, and their text labels and functional descriptions are extracted for each control to obtain multiple temporary hot words. Each temporary hot word is uniformly assigned a preset high weight value, or high weight values ​​are assigned to the temporary hot words of different controls based on one or more semantic attributes such as the size ratio of each control on the screen, the degree of centering of the position, and the interaction priority. Then, all temporary hot words and their corresponding high weight values ​​(i.e., the first weight) are dynamically loaded into the decoder of the speech recognizer in the form of key-value pairs so that the decoder performs speech recognition with these contents as constraints.

[0077] For another example, in addition to obtaining temporary hot words based on text tags and function descriptions, temporary hot words can also be obtained based on function identifiers and the current state. For instance, the function identifier itself can be used as a temporary hot word to cover precise instructions that developers or advanced users might use. For another example, for a switch control, taking a stove ignition switch control as an example, if its current state is "off," then the temporary hot word could be one or more of "turn on," "ignite," and "turn on the stove"; if its current state is "on," then the temporary hot word could be one or more of "turn off," "turn off the stove," and "extinguish the flame." This allows the voice recognizer to prioritize the recognition of safe and reasonable instructions based on whether the stove is actually burning, avoiding misidentifying "turn off" as a valid operation when the stove is already off, or missing the critical safety instruction "turn off" when the stove is already on.

[0078] In this step, one or more of the following are extracted from the context: text labels, function descriptions, function identifiers, and current states of all controls on the current screen. These are then assigned a first weight higher than that of general vocabulary. This allows the speech recognition decoder to actively favor words that actually exist on the screen when sorting candidate words, which significantly reduces the misrecognition rate of words specific to the vertical domain and improves the consistency between the speech recognition results and the screen content.

[0079] S230. Based on the target information and the first weight, perform speech recognition on the user's voice commands.

[0080] For example, in the language model decoding network of the speech recognition decoder, the loaded temporary hot words and their first weights are used as bias terms. At each decoding step, additional score bonuses are applied to the paths containing these temporary hot words, and the text sequence with the highest score is finally output as the speech recognition result.

[0081] Temporary hot words and their first weights are input into the candidate word prefix tree of the speech recognition decoder. When decoding the expansion node, paths that match these temporary hot words are expanded first, and the probability distribution of the matching paths is dynamically scaled so that the final output text sequence is semantically closer to the current screen content.

[0082] The aforementioned speech recognition scheme extracts text labels and function descriptions of all controls from the context as temporary hot words and assigns these temporary hot words a first weight higher than that of general vocabulary. This achieves real-time constraint enhancement of the speech recognition decoder, enabling the speech recognition process to actively favor the text labels and function descriptions that actually exist on the current screen, thus significantly improving the accuracy of speech recognition.

[0083] S240. Based on the context, obtain the business functions provided by the current interface.

[0084] The current interface can be understood as the user interface currently displayed on the target device's screen; it is a snapshot of the screen content at the current moment. Based on this snapshot, various controls are displayed on the current interface.

[0085] As explained above, the context can be understood as a structured on-screen semantic context, which not only includes the semantic attributes of controls and / or the semantic relationships between controls, but also implies the overall business semantics of the current interface through one or more of the following: control grouping, page identification, and functional area division.

[0086] Business functions can be understood as the operational scenarios or task scope provided to users by the current interface as a whole. In this embodiment of the invention, for example, the business function may be the selection of food preparation mode, the adjustment of cooking parameters, or the setting of cleaning program, which represents the upper-level business domain served by the current interface.

[0087] For example, the page identifier or view controller name of the current interface is extracted from the context, and the page identifier or view controller name is mapped to a predefined business function classification table to obtain the business function type to which the current interface belongs, that is, to obtain the business function provided by the current interface.

[0088] For another example, by aggregating and statistically analyzing the functional descriptions of all controls in the context, extracting high-frequency keywords, or through semantic clustering analysis, the overall semantics of the control set can be summarized into a single business function. For example, multiple temperature controls and timer controls may all point to the business function of "baking mode setting".

[0089] In this step, by parsing the business functions (or functional domains) provided by the current interface from the context, a scene-level semantic guidance is provided for the speech recognition language model, enabling the decoder to know the operational context of the current dialogue in advance, thereby adjusting the vocabulary preferences more effectively.

[0090] S250: Obtain candidate words related to business functions and assign a second weight to the candidate words.

[0091] Candidate words can be understood as words that are strongly related to the corresponding functional scenarios and are selected based on business functions. They may include one or more frequently occurring words such as function names, operation actions, and status descriptions.

[0092] The second weight can be understood as a numerical priority coefficient assigned to candidate words, which can be used to enhance the tendency during speech recognition decoding. Its setting is positively correlated with the domain matching degree of business functions.

[0093] For example, based on the obtained business functions, the corresponding extended vocabulary (such as "preparation mode selection" associated with one or more of "shredding", "slicing", and "weighing") is retrieved from the preset vocabulary library, and the candidate words in the extended vocabulary are uniformly assigned a preset second weight and loaded into the decoder.

[0094] For another example, the business function is input into the language model dynamic bias module. This module recalls candidate words and their relevance scores that are semantically similar to the business function from the cloud or local lexicon in real time, normalizes the relevance scores into a second weight, and then loads these candidate words and their corresponding second weights into the decoder.

[0095] In this step, by acquiring candidate words that are strongly related to the business functions of the current interface and assigning them a second weight, the decoding space of speech recognition is further focused on high-frequency words in the current functional scenario, supplementing the extended expressions that cannot be covered by control text labels alone, thereby improving the accuracy of speech recognition.

[0096] S260. Based on candidate words and second weights, perform speech recognition on user voice commands.

[0097] For example, candidate words and their second weights are directly incorporated into the probability distribution of the language model to amplify the probability of matching candidate word paths during decoding expansion, so that the entire decoding process always favors outputting text sequences that are consistent with the semantics of the current business function.

[0098] The aforementioned speech recognition scheme achieves dynamic functional domain bias in the speech recognition language model by extracting the overall business function of the current interface from the context and obtaining extended candidate words based on it, assigning them a second weight. This allows the speech recognition engine to not only focus on the text labels of individual controls but also understand the functional scenario of the entire interface. Therefore, even when users use colloquial expressions at the functional level, it can still accurately decode the text sequence matching the functional scenario, effectively compensating for the insufficient coverage of relying solely on text labels and further improving the adaptability and accuracy of speech recognition in multi-functional scenarios.

[0099] It should be noted that, as explained above, S220-S230 is one speech recognition solution, and S240-S260 is another speech recognition solution. These two speech recognition solutions can be executed one or in parallel, which can be set according to actual needs, and no specific limitation is made here.

[0100] S270. Based on the context, perform intent understanding on the speech recognition results, and generate control operation instructions based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each control and the operation action to be performed on the target control.

[0101] The technical solution of this invention, through the synergistic effect of the two speech recognition schemes, ensures accurate recognition of single-control terminology while also taking into account the generalized expression of functional levels in user speech. It provides complete scene constraints for speech recognition, minimizes the misrecognition rate of general speech recognition engines in vertical scenarios, and provides a high-quality, high-scene-consistency text input foundation for subsequent intent understanding.

[0102] Figure 3 This is a flowchart of another voice interaction method provided by an embodiment of the present invention. This embodiment is based on the above-mentioned technical solutions and optimized. In this embodiment, optionally, performing intent understanding on the voice recognition result based on context may include at least one of the following steps: in response to the voice recognition result being related to the position of the control, extracting the position information of each control from the context, and performing intent understanding on the voice recognition result based on the position information to obtain an intent understanding result representing the target control; and for each control, determining the semantic similarity between the control and each word in the voice recognition result based on the context, so as to perform intent understanding on the voice recognition result based on each semantic similarity to obtain an intent understanding result representing the target control. Alternatively, performing intent understanding on the voice recognition result based on context may include: for historical voice commands detected before the user's voice command, obtaining the historical recognition result and historical context corresponding to the historical voice command; and performing intent understanding on the voice recognition result based on the historical recognition result, historical context, and context. The above three optional solutions may be executed in one, two, or in parallel, which can be set according to actual needs and is not specifically limited here. Explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0103] See Figure 3 The method in this embodiment may specifically include the following steps:

[0104] S310. In response to a user voice command for the target device, obtain the context, wherein the context is used to describe the semantic attributes of the controls and / or the semantic relationships between the controls, and the controls are the controls currently displayed on the screen of the target device.

[0105] S320: Based on context, perform speech recognition on user voice commands.

[0106] S330. In response to the fact that the speech recognition result is related to the position of the control, the position information of each control is extracted from the context, and the intention understanding is performed on the speech recognition result based on the position information to obtain the intention understanding result, wherein the intention understanding result at least represents the target control to be operated in each control.

[0107] Among them, the speech recognition results are related to the position of the control, indicating that the speech recognition results contain descriptions that indicate spatial orientation, such as pronouns like "the second one," "the one in the upper left corner," and "the one on the right," which depend on the arrangement or geometric position of the control on the screen.

[0108] The semantic attributes described by the context include the spatial layout attributes of the corresponding controls. Therefore, the position information of each control can be extracted from the context. In this embodiment of the invention, the position information can optionally be represented based on one or more of the following: the arrangement number of the corresponding control on the screen, the coordinate range, the relative position (e.g., the top, bottom, left, and right relationship relative to other controls), and the layer depth. No specific limitation is made here.

[0109] Based on this, the speech recognition results can be used to understand intent based on location information, thereby mapping the control location-related descriptions in the speech recognition results to specific controls to obtain the target control.

[0110] For example, after detecting that the speech recognition result contains directional words or ordinal numbers, the coordinate range and sequence number of all controls are extracted from the context, and the controls are sorted according to the order of the screen from left to right and from top to bottom. The user's mention of "the Nth" or "top left corner" is matched with the controls in the corresponding position or coordinate area after sorting to determine the unique target control.

[0111] The above intent understanding scheme extracts the position information of controls from the context and parses the user's spatial reference accordingly, enabling the system to accurately anchor common location descriptions in everyday speech to specific screen controls. This solves the understanding problem when users rely on visual location for reference because they do not know the control names.

[0112] S340. For each control, based on the context, determine the semantic similarity between the control and each word in the speech recognition result, so as to perform intent understanding on the speech recognition result based on each semantic similarity, and obtain the intent understanding result, wherein the intent understanding result at least represents the target control.

[0113] In this context, a lexical unit can be understood as a single semantic unit obtained after word segmentation or tokenization of the speech recognition result, such as button, continue, increase, start, and confirm, without being specifically defined here.

[0114] Semantic similarity can be understood as the degree of proximity between the corresponding control and the corresponding word in the semantic space. In the embodiments of the present invention, optionally, it can be represented by one or more of the following methods: vector cosine similarity, semantic distance, or matching score of pre-trained language model, without specific limitation.

[0115] For example, the text labels and function descriptions of each control in the context are encoded into semantic vectors, and each word in the speech recognition result is also encoded into a semantic vector. The cosine similarity between each word and the text label and function description of each control is calculated to obtain the semantic similarity.

[0116] As another example, the text labels and function descriptions of all controls in the context are concatenated into a control-level semantic representation, each word is encoded into a query vector, and a cross-attention mechanism is used to calculate the matching score between the query vector and the semantic representation of each control to obtain the corresponding semantic similarity.

[0117] Furthermore, based on the semantic similarity, the speech recognition results are subjected to intent understanding, so as to map the spoken expression to the control with the closest semantics through semantic matching, and obtain the intent understanding result representing the target control.

[0118] Based on the above examples, for instance, the cosine similarity of each control with all words is summarized, and the control with the highest overall cosine similarity is selected as the target control. Another example is to sort the controls by matching score from highest to lowest, and determine the control with the highest matching score as the target control.

[0119] The aforementioned intent understanding scheme achieves fuzzy intent alignment from colloquial and imprecise expressions to specific controls by calculating the fine-grained semantic similarity between each control and each word in the user's expression. This enables the system to understand fuzzy instructions such as "Continue" and "Start" and accurately map them to the corresponding controls, breaking through the limitation of traditional schemes that must rely on precise control names.

[0120] Based on this, optionally, the above intent understanding scheme can be considered as a fine-grained alignment mapping relationship at the control level obtained by cross-modal joint encoding of control semantic features and speech acoustic features. Building upon this, a multimodal large model (a unified architecture natively supporting multimodal inputs such as text, image, and speech) is used as the backbone network. Leveraging its cross-modal alignment capabilities established during the pre-training phase, a tightly coupled vision-speech joint encoding and alignment network is constructed. This network maps control semantic features and speech acoustic features to a unified cross-modal latent space, and then establishes a fine-grained alignment mapping at the control level within this cross-modal latent space.

[0121] For example, the control tree context encoder introduces a lightweight encoder to model the context of the full-screen control sequence; the speech encoder uses the user's original speech waveform (i.e., the user's voice command speech waveform); the acoustic front end uses a pre-trained speech encoder from a large model as the acoustic front end to extract speech acoustic features; the hot word awareness bias introduces hot word priors from the visual modality at the encoder level, calculates the attention score between the speech acoustic features of each frame and all hot words, and injects high-hot word information in a residual addition manner; the semantic encoding layer overlays several context-aware speech semantic representations on the biased speech acoustic features; and then fine-grained alignment matching upgrades the traditional coarse-grained alignment of "full-screen - full sentence" to fine-grained precise alignment of "control attribute - speech segment". For each operable attribute of the control, an independent anchor vector is generated in the shared space to obtain semantic similarity based on this.

[0122] S350. For historical voice commands detected before the user's voice command, obtain the historical recognition results and historical context corresponding to the historical voice command, and perform intent understanding on the voice recognition results based on the historical recognition results, historical context and context, to obtain the intent understanding results.

[0123] Among them, historical voice commands can be understood as voice commands that the user has issued in the same interactive session before the current user voice command.

[0124] Historical recognition results can be understood as the text sequence output after speech recognition of historical voice commands.

[0125] Historical context can be understood as the context constructed on the target device screen at the moment the historical voice command was issued, which records the semantic attributes of the controls displayed on the screen at that time and the semantic relationships between the controls.

[0126] For example, when the intent understanding of the current round is initiated, the historical recognition results corresponding to the historical voice commands of the most recent N rounds or all rounds can be extracted from the session management module, and the context corresponding to these historical voice commands can be obtained from the context record as the historical context.

[0127] Furthermore, intent understanding is performed based on the context and the corresponding historical recognition results and historical context.

[0128] For example, the historical recognition results, historical context, and current context are jointly modeled in a unified spatiotemporal dimension, and the current speech recognition result is used as a query item. Attention is calculated in the joint feature space to identify whether the current speech recognition result contains a reference to the previous control or an omitted subject. If so, the relevant control information in the historical context is added to the current intent understanding result.

[0129] For another example, the action and control mention records in the historical recognition results are used as prior clues. Combined with the controls that have been operated on and their state changes in the historical context, the continued state of these controls is tracked in the current context. When the current speech recognition result shows one or more cross-round references such as "continue", "return" and "confirm it", the historical context is used to lock the mapping position of the corresponding control in the current context, thereby completing the cross-round intent completion and reference resolution.

[0130] The aforementioned intent understanding scheme, by introducing historical recognition results and historical context corresponding to historical voice commands, places the intent understanding of the current round within the temporal context of multi-round dialogue for joint modeling. This enables the perception of preceding operations and contextual changes, thereby accurately resolving omitted expressions or cross-round references in the current user's voice commands that rely on historical information (such as historical context and / or historical voice commands). This achieves a deep fusion of dialogue history and dynamic semantic context, effectively compensating for the information loss problem of single-round intent understanding in multi-round interactions. It significantly enhances the intent completion and reference resolution capabilities in continuous dialogue scenarios, providing users with a more natural and fluent continuous voice interaction experience.

[0131] It should be noted that, as explained above, S330, S340 and S350 are three intent understanding schemes, which can be executed one by one or in parallel. This can be set according to actual needs, and no specific limitation is made here.

[0132] S360. Based on the intent understanding result, generate control operation instructions, wherein the control operation instructions are used to instruct the target control and the operation action to be performed on the target control.

[0133] The technical solution of this invention, through the synergistic effect of the above three intent understanding schemes, achieves precise spatial positioning and coherent temporal understanding. It not only solves the problems of arbitrariness and ambiguity in single-turn instructions, but also overcomes the semantic breaks and omissions in multi-turn dialogues. Together, they construct an intent understanding capability with visual perception and dialogue memory, which can accurately understand various spoken expressions of users in real and complex interactive scenarios.

[0134] Figure 4 This is a flowchart of another voice interaction method provided by an embodiment of the present invention. This embodiment is based on and optimized from the above-described technical solutions. In this embodiment, optionally, the controls are displayed on the current interface, which is the interface currently displayed on the screen; the context can be constructed in the following way: parsing the control tree corresponding to the current interface to obtain the semantic attributes of each control; obtaining the interaction logic of each control, and obtaining the semantic relationship between each control based on each interaction logic and each semantic attribute; constructing the context based on each semantic attribute and / or semantic relationship. The explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0135] See Figure 4 The method in this embodiment may specifically include the following steps:

[0136] S410. Parse the control tree corresponding to the current interface to obtain the semantic attributes of each control. The current interface is the interface currently displayed on the screen of the target device, and each control is located on the current interface.

[0137] The control tree can be understood as the view hierarchy of the current interface. It is organized in a tree data structure, with the root node representing the entire interface container and the child nodes expanding layer by layer to represent nested view groups and controls at each level, fully reflecting the layout hierarchy and containment relationship of the interface.

[0138] For example, the complete control tree of the current interface can be obtained in real time through a system-level Hook mechanism or accessibility service. Then, each node in the control tree is traversed, and one or more of the control type, text label, location information, function identifier, current state and operable state of the corresponding control are extracted from the node attributes. The above information is then structured and stored as semantic attribute records for each control.

[0139] In this step, by parsing the current interface control tree in real time, the complete semantic attributes of all controls on the screen can be automatically captured, providing a data foundation for the subsequent construction of a structured context. There is no need to manually predefine the semantic template of each interface, which has good versatility and dynamic adaptation capabilities.

[0140] S420. Obtain the interaction logic of each control, and based on each interaction logic and each semantic attribute, obtain the semantic relationship between each control.

[0141] Interaction logic can be understood as the functional association or triggering dependency generated by the operation process between a single control or between controls. For example, one or more of the following represent the linkage rules of controls in the business operation chain: "clicking a button will jump to a page", "turning on a switch will enable the related parameter adjustment slider", and "activating the next step button after completing the current step".

[0142] For example, by parsing the parent-child container structure and sibling node arrangement order in the control tree, and using the control type (such as the "Next" button and the "Back" button often forming a complementary relationship before and after the process) and the text similarity of the function description, the spatial inclusion relationship, function grouping relationship and operation sequence dependency relationship between controls can be summarized, forming a semantic relationship covering both layout and logic dimensions.

[0143] In this step, by introducing the interaction logic of the controls and combining it with semantic attributes, the context can not only describe what controls are on the current interface, but also how the controls cooperate with each other. This provides richer constraints for subsequent speech recognition and intent understanding, and effectively supports cross-control referencing resolution and process awareness.

[0144] S430. Construct a context based on each semantic attribute and / or semantic relationship.

[0145] For example, each control is treated as an independent node, and its semantic attributes are serialized into a set of key-value pairs of attributes for the node. Directed edges are established between each pair of nodes with semantic relationships, with the relationship type added to the edges. Finally, all nodes and edges are output as a JSON-formatted graph, which serves as context for subsequent modules.

[0146] For another example, the semantic attributes of each control are encoded as node feature vectors, and various semantic relationships are encoded as adjacency matrices or sets of relationship triples. All nodes and relationships are integrated into a graph data structure in memory. At the same time, a fast index table from control ID to node address is maintained so that subsequent steps can quickly retrieve relevant information based on control ID or voice query. The final output is a graph-based context that can be directly queried.

[0147] In this step, by integrating semantic attributes and semantic relationships into a structured context, a dynamic knowledge base that clearly describes the screen content is formed, providing on-screen visual semantic input for speech recognition and intent understanding, fundamentally ensuring the quality of information supply in the subsequent "what you see is what you can say" interaction link.

[0148] Optionally, in response to a switch in the current interface or a change in controls, execution S410 can be returned to rebuild the context. A switch in the current interface can be understood as the target device screen transitioning from the current interface to another, causing a change in the overall set of controls displayed on the screen. A change in controls can be understood as an update of controls on the screen without switching interfaces, such as one or more localized dynamic changes, like a control's enabled state changing from disabled to enabled, text label content being refreshed, visibility changing from hidden to displayed, or numerical updates. Upon detecting a switch in the current interface or a change in controls, the context can be rebuilt, thus ensuring that the voice interaction context is always synchronized with the current screen content.

[0149] S440: In response to a user voice command for the target device, obtain the context.

[0150] S450: Based on context, perform speech recognition on user voice commands.

[0151] S460. Based on the context, perform intent understanding on the speech recognition results, and generate control operation instructions based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each control and the operation action to be performed on the target control.

[0152] The technical solution of this invention extracts the semantic attributes of each control, obtains the interaction logic, and deduces the semantic relationship between controls by parsing the control tree of the current interface in real time, and finally constructs a structured context. This enables the system to have the perception ability to "understand" the screen content, and transforms the static interface visual information into a knowledge base that can be directly called by downstream voice interaction tasks (i.e., voice recognition tasks and intent understanding tasks), thereby providing high-quality data source support for context enhancement of voice recognition and intent understanding.

[0153] Based on this, an optional technical solution is to encapsulate the above-mentioned voice interaction method into a middleware application. The middleware is integrated into the target device and provides an interface so that the middleware can obtain the control tree and various interaction logics output by the target device through the interface. The middleware is a software layer independent of the user interface system and operating system of the target device. The interface follows a preset control semantic description protocol and data exchange format, so that different target devices can output the control tree and various interaction logics to the middleware through the same interface.

[0154] In this context, middleware can be understood as a standardized software layer independent of the target device's user interface (UI) system and operating system. It encapsulates voice interaction capabilities into reusable, independent modules that run on the target device, achieving a universal, plug-and-play cross-device integration middleware architecture that can be quickly integrated across different devices. The UI system can be understood as the upper-level interface rendering and interaction response framework on the target device, responsible for one or more of the following: visual presentation of controls, layout management, and handling and feedback of user touch / click events. The operating system can be understood as the underlying hardware resource management and basic service support platform of the target device, responsible for one or more of the following: process scheduling, memory management, file system, network communication, and underlying hardware drivers.

[0155] The interface can be understood as a standardized data acquisition channel provided by the middleware to the outside world, following a unified (i.e., preset) control semantic description protocol and data exchange format, enabling different target devices to output the control tree and various interaction logics to the middleware through the same interface, so as to decouple the middleware from the specific product UI and construct a context without the need to perceive the underlying UI differences. Among them, the control semantic description protocol can be understood as a standardized specification for uniformly expressing the functional semantics of controls. In this technical solution, optionally, this standardized specification can define one or more of the control type, function identifier naming rules, status value enumeration, hierarchical relationship expression method, and operation intention mapping rules, etc., to ensure that different device manufacturers use consistent semantic tags when describing the same type of controls. The data exchange format can be understood as a standardized data organization method for structured transmission of the control tree and its related information. In this technical solution, optionally, one or more of the key names, data types, and nested structures of the control attribute fields can be specified, enabling the middleware to uniformly parse and construct a context without caring about the specific implementation differences of the underlying UI systems of each target device.

[0156] In the above technical solution, by encapsulating the aforementioned voice interaction method as a standardized middleware application and integrating it into the target device to obtain the control tree and interaction logic through a unified interface, this design completely decouples the core capabilities such as speech recognition and intention understanding from the heterogeneous UIs of different devices. Each device only needs to output the screen content following the standardized UI semantic description protocol, and the middleware can adaptively complete the context construction and voice interaction processing, thus having the plug-and-play integration ability, greatly reducing the development cost and implementation cycle of voice skill adaptation, and at the same time ensuring the consistency and reliability of the voice interaction experience between different devices.

[0157] On this basis, in order to better understand the above technical solutions as a whole, the following will take a visual and verbal interaction system based on dynamic screen perception as an example to illustrate it.

[0158] Exemplarily, see Figure 5 , this system includes a UI dynamic perception and semantic abstraction layer, a context-enhanced speech recognition layer, a multi-modal fusion intention understanding layer, and a visual and verbal linkage execution and feedback layer. Among them,

[0159] The first layer: UI dynamic perception and semantic abstraction layer, as the "eyes" of the system, is used to capture the screen content in real time.

[0160] 1) Control tree parsing: Through the system-level Hook mechanism or accessibility service, the complete control tree of the current interface is obtained and parsed in real time to obtain information such as control type, text label, position information, function identifier, current status, and operable status.

[0161] 2) Interaction logic acquisition: Obtain the expected interaction feedback of the control.

[0162] 3) Semantic Graph Construction: The above information is integrated to generate a structured JSON semantic graph (i.e., semantic context). This semantic graph can clearly describe what controls are on the current screen, what these controls are, what they can do, and what their current state is.

[0163] The second layer is a context-enhanced speech recognition layer, which uses semantic graphs to optimize speech recognition.

[0164] 1) Dynamic hot word injection: All text tags and functional descriptions extracted from the semantic graph are used as temporary hot words and dynamically loaded into the decoder of the speech recognizer. For example, when the screen displays "pasteurization", this technical term is temporarily given a very high weight to significantly improve the recognition accuracy of this technical term.

[0165] 2) Dynamic bias of language model: Based on the business functions provided by the current interface (such as "selection of food preparation mode"), dynamically bias the speech recognition language model so that it is more inclined to decode word sequences related to the business function.

[0166] The third layer: the multimodal fusion intent understanding layer, which acts as the "brain" of the system, is responsible for parsing the user's vague verbal expressions into precise operation instructions.

[0167] 1) Reference resolution: When the user says "open the second one", the "second one" is resolved into a specific control ID by combining the control list and positional relationship in the semantic graph, thus obtaining the target control.

[0168] 2) Fuzzy Intent Alignment: When the user says "Continue", calculate the semantic similarity between the user's statement and all operable controls in the semantic graph, find the "Next" button as the target control, and understand that "Continue" means to advance the process.

[0169] 3) Multi-turn context enhancement: Jointly model the multi-turn historical context and the history of voice dialogue in a unified spatiotemporal dimension to assist in cross-modal alignment of the current turn and perform intent understanding based on this.

[0170] 4) Operational feasibility verification: Before generating control operation instructions, check whether the target control is in an operable state in the current semantic graph to avoid issuing invalid instructions to grayed-out unusable controls.

[0171] In this example, optionally, it has the ability to integrate voice and touch interaction, that is, it supports users to perform supplementary operations through touch after voice operation, and also supports voice guidance during touch operation.

[0172] The fourth layer: the visual and verbal linkage execution and feedback layer, is responsible for instruction execution and closed-loop feedback.

[0173] 1) Command simulation execution: Send the parsed control operation commands to the UI system for execution.

[0174] 2) Visual Synchronized Feedback: Visual feedback such as highlighting and / or animation effects is applied to the target control on the screen and synchronized with voice broadcast, achieving "visual and spoken feedback".

[0175] 3) Voice confirmation broadcast: After execution, the text-to-speech (TTS) engine will broadcast natural language feedback, such as "Cooking has started", forming an interactive loop.

[0176] Therefore, the above system can dynamically construct a structured semantic context (i.e., semantic graph) by parsing the control tree and interaction logic of the screen UI in real time. Then, the semantic graph is injected into the speech recognition and intent understanding process in real time, enabling the system to perform speech recognition and intent understanding based on being able to "understand" the screen content, thereby achieving a precise "what you see can be spoken, what you say can be obtained" voice interaction experience.

[0177] Figure 6 This is a structural block diagram of a voice interaction device provided in an embodiment of the present invention. This device is used to execute the voice interaction method provided in any of the above embodiments. This device and the voice interaction methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the voice interaction device can be found in the embodiments of the above voice interaction methods. See also... Figure 6 The device may specifically include: a context acquisition module 510, a speech recognition module 520, and an intent understanding module 530.

[0178] The context acquisition module 510 is used to acquire the context in response to a user voice command for the target device, wherein the context is used to describe the semantic attributes of the controls and / or the semantic relationships between the controls, and the controls are the controls currently displayed on the screen of the target device;

[0179] The speech recognition module 520 is used to perform speech recognition on user voice commands based on context;

[0180] The intent understanding module 530 is used to perform intent understanding on the speech recognition results based on the context, and to generate control operation instructions based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each control and the operation action to be performed on the target control.

[0181] Optionally, the speech recognition module 520 may include:

[0182] The first weighting unit is used to extract the target information of each control from the context and assign a first weight to the target information. The target information includes at least one of text label, function description, function identifier and current state.

[0183] The first speech recognition unit is used to perform speech recognition on user speech commands based on each target information and the first weight.

[0184] Optionally, the controls are displayed on the current interface, which is the interface currently displayed on the screen. The speech recognition module 520 may include:

[0185] The business function retrieval unit is used to retrieve the business functions provided by the current interface based on the context.

[0186] The second weighting unit is used to obtain candidate words related to business functions and assign a second weight to the candidate words.

[0187] The second speech recognition unit is used to perform speech recognition on user voice commands based on candidate words and a second weight.

[0188] Optionally, the intent understanding module 530 may include at least one of the following units:

[0189] The first intent understanding unit is configured to, in response to the relationship between the speech recognition result and the control position, extract the position information of each control from the context, and perform intent understanding on the speech recognition result based on the position information to obtain an intent understanding result representing the target control; and,

[0190] The second intent understanding unit is used to determine the semantic similarity between each control and each word in the speech recognition result based on the context, so as to perform intent understanding on the speech recognition result based on each semantic similarity and obtain the intent understanding result representing the target control.

[0191] Optionally, the intent understanding module 530 may include:

[0192] The historical data acquisition unit is used to acquire the historical recognition results and historical context corresponding to the historical voice commands detected before the user's voice commands.

[0193] The third intent understanding unit is used to perform intent understanding on the speech recognition results based on historical recognition results, historical context, and context.

[0194] Optionally, the intent understanding module 530 may include:

[0195] The operability verification unit is used to verify the operability of the target control based on the context, for the target control represented by the intent understanding result.

[0196] The control operation instruction generation unit is used to generate control operation instructions in response to the operability characterization of the target control being operable, based on the intent understanding result.

[0197] Optionally, the controls are displayed on the current screen, which is the screen currently being displayed.

[0198] The context is constructed using the following modules:

[0199] The semantic attribute acquisition module is used to parse the control tree corresponding to the current interface to obtain the semantic attributes of each control;

[0200] The semantic relationship acquisition module is used to obtain the interaction logic of each control and, based on the interaction logic and semantic attributes, obtain the semantic relationship between each control.

[0201] The context building module is used to construct a context based on various semantic attributes and / or semantic relationships.

[0202] In addition, optionally, the aforementioned voice interaction device may further include:

[0203] The context reconstruction module is used to reconstruct the context in response to changes in the current interface or controls by calling the semantic property acquisition module, semantic relationship acquisition module, and context construction module again.

[0204] Alternatively, the aforementioned voice interaction is encapsulated as a middleware application. The middleware is integrated into the target device and provides an interface that allows the middleware to obtain the control tree and various interaction logics output by the target device. The middleware is a software layer independent of the user interface system and operating system of the target device. The interface follows a preset control semantic description protocol and data exchange format, enabling different target devices to output the control tree and various interaction logics to the middleware through the same interface.

[0205] Optionally, the above-mentioned voice interaction device may further include:

[0206] The control operation instruction sending module is used to send control operation instructions to the target device so that the target device can provide feedback after executing the control operation instructions. The feedback content includes at least one of the following:

[0207] Visual feedback for the target control, voice feedback for the target control, and voice feedback for user voice commands are at least one of the following:

[0208] The voice interaction device provided in this embodiment of the invention, through the cooperation of various modules, transforms the screen controls into semantic context and injects them as constraints into the voice recognition and intent understanding process. This enables the system to perform voice recognition and intent understanding based on its ability to "understand" the screen content, thereby achieving a precise "what you see is what you can say, what you say is what you get" voice interaction experience and improving the accuracy of voice interaction.

[0209] The voice interaction device provided in the embodiments of the present invention can execute the voice interaction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0210] It is worth noting that in the embodiments of the above-mentioned voice interaction device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0211] Figure 7 A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0212] like Figure 7 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0213] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0214] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as voice interaction methods.

[0215] In some embodiments, the voice interaction method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the voice interaction method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the voice interaction method by any other suitable means (e.g., by means of firmware).

[0216] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips or system-on-a-chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0217] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0218] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0219] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0220] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0221] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0222] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0223] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0224] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A voice interaction method, characterized in that, include: In response to a user voice command for a target device, a context is obtained, wherein the context is used to describe the semantic attributes of controls and / or the semantic relationships between the controls, and the controls are the controls currently displayed on the screen of the target device; Based on the context, the user's voice command is recognized. Based on the context, the speech recognition results are subjected to intent understanding, and control operation instructions are generated based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each of the controls and the operation action to be performed on the target control.

2. The method according to claim 1, characterized in that, The step of performing speech recognition on the user's voice command based on the context includes: For each control, target information of the control is extracted from the context, and a first weight is assigned to the target information, wherein the target information includes at least one of text label, function description, function identifier and current state; Based on the target information and the first weight, the user's voice command is recognized.

3. The method according to claim 1, characterized in that, The control is displayed on the current interface, which is the interface currently displayed on the screen; The step of performing speech recognition on the user's voice command based on the context includes: Based on the context, the business functions provided by the current interface are obtained; Obtain candidate words related to the business function and assign a second weight to the candidate words; Based on the candidate words and the second weight, speech recognition is performed on the user's voice command.

4. The method according to claim 1, characterized in that, The intention understanding based on the context includes at least one of the following steps: In response to the fact that the speech recognition result is related to the position of the control, the position information of each control is extracted from the context, and the speech recognition result is subjected to intent understanding based on the position information to obtain the intent understanding result representing the target control; as well as, For each control, based on the context, the semantic similarity between the control and each word in the speech recognition result is determined, so as to perform intent understanding on the speech recognition result based on each semantic similarity, and obtain the intent understanding result representing the target control.

5. The method according to claim 1, characterized in that, The process of understanding intent based on the context of speech recognition results includes: For historical voice commands detected before the user's voice command, obtain the historical recognition result and historical context corresponding to the historical voice command; Based on the historical recognition results, the historical context, and the context, the speech recognition results are used to understand intent.

6. The method according to claim 1, characterized in that, The generation of control operation instructions based on the intent understanding result includes: Based on the context, the operability of the target control is verified for the target control as represented by the intent understanding result. In response to the operability characterization that the target control is operable, control operation instructions are generated based on the intent understanding result.

7. The method according to claim 1, characterized in that, The control is displayed on the current interface, which is the interface currently displayed on the screen; The context is constructed in the following manner: The control tree corresponding to the current interface is parsed to obtain the semantic attributes of each control; Obtain the interaction logic of each control, and based on the interaction logic and semantic attributes, obtain the semantic relationship between the controls; The context is constructed based on the semantic attributes and / or semantic relationships described herein.

8. The method according to claim 7, characterized in that, Also includes: In response to the switching of the current interface or the change of the control, the process returns to the step of parsing the control tree corresponding to the current interface in order to reconstruct the context.

9. The method according to claim 7, characterized in that, The method is encapsulated as a middleware application, which is integrated into the target device. The middleware provides an interface to obtain the control tree and various interaction logics output by the target device. The middleware is a software layer independent of the user interface system and operating system of the target device. The interface follows a preset control semantic description protocol and data exchange format, enabling different target devices to output the control tree and various interaction logics to the middleware through the same interface.

10. The method according to claim 1, characterized in that, Also includes: The control operation instruction is sent to the target device so that the target device provides feedback after executing the control operation instruction, wherein the feedback content includes at least one of the following: At least one of the following: visual feedback to the target control, voice feedback to the target control, and voice feedback to the user's voice command.

11. A voice interaction device, characterized in that, include: The context acquisition module is used to acquire context in response to a user voice command for a target device, wherein the context is used to describe the semantic attributes of controls and / or the semantic relationships between the controls, and the controls are the controls currently displayed on the screen of the target device; A speech recognition module is used to perform speech recognition on the user's voice commands based on the context; The intent understanding module is used to perform intent understanding on the speech recognition results based on the context, and to generate control operation instructions based on the intent understanding results. The control operation instructions are used to instruct the target control to be operated in each of the controls and the operation action to be performed on the target control.

12. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to cause the at least one processor to perform the voice interaction method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the voice interaction method as described in any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the voice interaction method as described in any one of claims 1-10.

Images