An eye movement-assisted method for recognizing speech interaction intentions

Abstract

An eye movement-assisted method for recognizing speech interaction intentions, comprising: performing intention recognition and slot filling on a speech interaction instruction to obtain an interaction intention and slot information; performing eye movement tracking on a desktop interaction personnel to recognize an intention manipulation area of the interaction personnel on an interaction software UI; parsing the functions of the intention manipulation area of the interaction software and extracting a knowledge graph subgraph associated with the intention manipulation area from the knowledge graph of the interaction software; analyzing the slot information associated with the interaction intention, and extracting and improving the slot information with missing sentence components, unclear pronoun references, and insufficient noun constraints in the speech instruction from the knowledge graph subgraph; based on the interaction intention and the improved slot information, indexing and matching a response method for the speech instruction from the knowledge graph and sending it to the interaction software for response; The present invention is oriented to desktop human-computer interaction. Aiming at problems such as missing sentence components, unclear pronoun references, and insufficient noun constraints in speech instructions during natural speech interaction, it integrates eye movement and speech interaction, and realizes the supplement of slot information of interaction intentions through the functions of the intention manipulation area pointed by eye movement, reduces the memory load of interaction personnel for speech interaction instructions, and realizes natural speech interaction.

Description

Technical Field

[0001] The present invention relates to the field of multimodal human-computer interaction, and particularly to a method for identifying speech interaction intentions assisted by eye movement. Background Art

[0002] With the development of technologies such as speech recognition, eye movement tracking, and behavior recognition, natural interaction methods such as speech, eye movement, and gesture have gradually become new interaction modalities in addition to traditional interaction methods such as mouse, keyboard, and touch, bringing great convenience and efficiency improvement to the existing human-computer interaction.

[0003] In the prior art, it is inclined to use the form of multi-round dialogue to solve problems such as unclear interaction intentions or incomplete information in the natural speech interaction process. However, people can automatically associate and complete information such as default sentence components, pronouns with unclear references, and nouns with insufficient constraints according to the interaction context during the natural speech interaction process, and accurately understand the interaction intentions of the other party. Although the above information can also be associated and completed through multi-round dialogue, the intelligence and naturalness of the speech interaction algorithm always seem insufficient, and multi-round dialogue also delays the confirmation time of interaction intentions and reduces the efficiency of speech interaction, which runs counter to the original intention of improving interaction efficiency through natural interaction methods such as speech and eye movement.

[0004] Patent application CN112114672A discloses an eye movement combined with speech assisted interaction device and method, including a speech unit, a positioning unit, and a main control unit; the speech unit is used to obtain the voice signal of the user and transmit the voice signal to the main control unit; the positioning unit is used to obtain the fixation point signal, identify the position information corresponding to the fixation point signal, and transmit the fixation point signal including the position information to the main control unit; the main control unit is used to obtain the fixation point signal and convert the fixation point signal including the position information into screen coordinates; it is also used to obtain the voice signal, identify the control instruction in the voice signal, and execute the control instruction at the corresponding screen coordinates. It realizes determining the position information on the screen through the positioning unit (eye movement), and then combining with the speech unit to perform corresponding operations at this position. However, this patent application only realizes mapping the speech instruction to the screen area being stared at for execution, and does not use the fixation information to improve the recognition of the speech instruction, and does not solve the problem of incomplete slot information caused by situations such as default sentence components, unclear pronoun references, and insufficient noun constraints in natural interaction speech instructions. Summary of the Invention

[0005] To overcome the defects of the above-mentioned existing technologies, the purpose of the present invention is to propose an eye movement-assisted speech interaction intention recognition method. Aiming at problems such as the lack of sentence components in speech commands, unclear pronoun references, and insufficient name constraints, the eye movement tracking technology is used to identify and analyze the function of the intention manipulation area of the interaction personnel, and then combined with the software knowledge graph to improve the slot information of the speech command; combined with eye movement assistance, the present invention can avoid multi-round conversations to a certain extent and improve the efficiency of speech interaction.

[0006] To achieve the above purpose, the present invention provides the following technical solutions:

[0007] An eye movement-assisted speech interaction intention recognition method specifically includes the following steps:

[0008] 1) Perform intention recognition and slot filling on the speech interaction command to obtain the interaction intention and slot information;

[0009] 2) Identify the intention manipulation area of the interaction personnel on the interaction software UI;

[0010] 3) Extract the knowledge graph subgraph associated with the intention manipulation area;

[0011] 4) Analyze the slot information associated with the interaction intention recognized in step 1), and extract and improve the slot information in step 1) from the knowledge graph subgraph obtained in step 3);

[0012] 5) Based on the interaction intention in step 1) and the improved slot information in step 4), index and match the response method of the speech command from the knowledge graph, and send it to the interaction software for response.

[0013] In the above step 1), the interaction intention and slot information are obtained through known joint intention recognition and slot filling algorithms.

[0014] In the above step 2), the real-time detection of the gaze points of the desktop interaction personnel on the UI is performed by using an eye tracker or a known eye movement tracking algorithm, and the area where the interaction personnel is currently gazing is the intention manipulation area.

[0015] In the above step 3), the function of the intention manipulation area, that is, the operable item on the UI, is obtained by analyzing the mapping relationship between the layout and function of the interaction software UI. Then, entity linking is performed from the software knowledge graph according to the entity associated with the function of the intention manipulation area, and the knowledge graph subgraph is extracted.

[0016] In step 4), according to the entity information included in the knowledge graph sub-graph, the slot information with component defaults, unclear references, and insufficient constraints in the voice command in step 1) is supplemented and confirmed uniquely to improve the slot information; this process validates the legality of the existing slot information according to the knowledge graph sub-graph while improving the missing or unclear slot information.

[0017] In step 5), according to the message format and message interaction method supported by the interaction software, the interaction intention obtained in step 1) and the slot information improved and confirmed according to step 4) are sent to the interaction software in the response method of the corresponding interaction intention that indexes and matches relevant entities from the knowledge graph, and the interaction software then makes a response to achieve natural voice interaction for desktop software.

[0018] Compared with the prior art, the beneficial effects of the present invention are:

[0019] The present invention combines eye movement assistance, integrates the interaction intention shown by eye movement with the voice interaction intention, effectively addresses problems such as component defaults in voice commands, unclear pronoun references, and insufficient noun constraints caused by the naturalness of voice commands during the voice interaction process, and to a certain extent avoids simply using voice interaction for multi-round conversations to address these problems, which is more conducive to improving the interaction efficiency while ensuring the naturalness of voice interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] To illustrate the present invention more clearly, one or more embodiments or technical solutions in the related art will be disclosed below. The accompanying drawings required for describing the embodiments or related art will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments described in one or more embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objectives, and advantages of the present disclosure will become more apparent.

[0021] Figure 1 It is a flowchart of the operation of the present invention.

[0022] Figure 2 It is an example of an interaction software applicable to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0024] In the description of the embodiments of the present invention, unless otherwise specified, the meaning of "a plurality" is two or more.

[0025] A method for identifying speech interaction intentions assisted by eye movement specifically includes the following steps:

[0026] 1) Identify the intention and fill the slots of the speech interaction instruction to obtain the interaction intention and slot information;

[0027] 2) Track the eye movement of the desktop interaction personnel to identify the intention manipulation area of the interaction personnel on the interaction software UI;

[0028] 3) Analyze the functions of the interaction software intention manipulation area identified in step 2), and extract the knowledge graph subgraph associated with the intention manipulation area from the interaction software knowledge graph;

[0029] 4) Analyze the slot information associated with the interaction intention identified in step 1), and extract and improve the slot information with missing components, unclear pronoun references, and insufficient constraints in the speech instruction from the knowledge graph subgraph obtained in step 3);

[0030] 5) Based on the interaction intention in step 1) and the improved slot information in step 4), index and match the response method of the speech instruction from the knowledge graph, and send it to the interaction software for response.

[0031] In step 1), the interaction intention and slot information are obtained by using known joint intention recognition and slot filling algorithms. However, in the process of natural speech interaction, speech instructions often have situations such as missing sentence components, unclear pronoun references, and insufficient noun constraints, resulting in incomplete recognition of slot information related to the interaction intention, which requires subsequent steps to process.

[0032] In step 2), the intention manipulation area of the interaction personnel on the interaction software UI is realized by using an existing eye tracker or a known eye movement tracking algorithm. The area that the interaction personnel are currently looking at is the intention manipulation area.

[0033] In step 2), the intention manipulation area refers to a functional area on the software UI, usually composed of a series of controls, and has regionality in the UI layout.

[0034] In step 3), the functions of the intention manipulation area can be obtained by analyzing the mapping relationship between the UI layout and functions of the interaction software, and then entity linking is performed from the software knowledge graph according to the entities associated with the functions of the intention manipulation area to extract the knowledge graph subgraph.

[0035] In step 4), according to the entity information included in the knowledge graph subgraph, the slot information with missing components, unclear references, or insufficient constraints in the voice command is filled and confirmed uniquely, which can not only fill in the missing information but also judge the legality of the existing slot information.

[0036] In step 5), according to the slot information refined and confirmed in step 4), the response methods corresponding to the interaction intents of relevant entities are indexed and matched from the knowledge graph, and the response methods are sent to the interaction software according to the message format and message interaction method supported by the interaction software, and then the interaction software makes a response to achieve natural voice interaction for desktop software.

[0037] Embodiment 1

[0038] Figure 2 An example of an interaction software to which the present invention can be applied is shown. The four-pane video display area can display the real-time images of four drones.

[0039] In Figure 2 In the shown interaction software, it is possible to control which drones' real-time images are displayed in the four panes through voice commands. An example of a voice interaction command with complete sentence components and clear semantics is "display the real-time image of drone 1 on pane 1". However, during the natural voice interaction process, the commander (software interaction personnel) may directly and simply issue the voice command "display the image of drone 1". According to step 1) of the present invention, it can be recognized that the intent of this voice command is "display the real-time image of the drone", and the relevant slot information includes "display", "drone 1", and "image". Simply analyzing this voice command cannot parse out the command executable by the interaction software because the interaction software has 4 display panes, and currently the voice command only gives which pane to display the image of drone 1 on.

[0040] In response to the above scenario, when using traditional multi-round conversations to fill in the slots, the software needs to reply "which pane do you want to display the image of drone 1 on" to prompt the commander to say the complete command or compensate for the slot information. However, in the application of the present invention, considering that the commander is observing Figure 2The upper right pane on the displayed UI, according to step 2) of the present invention, obtains the four-pane display area where the commander is observing the UAV screen; according to step 3) of the present invention, combining the knowledge graph of the software and the eye fixation position, the knowledge graph sub-graph of the four-pane display area is extracted, and it can be obtained that there are 4 display panes in the four-pane display area, as well as the area position and size of each pane. Each pane can separately display the real-time images of certain UAVs; further combining step 4) of the present invention, combining the eye movement that is fixating on the upper right pane, it can be inferred that "display the image of UAV 1 on pane 2", and the slot information "pane 2" is completed; further according to step 5), based on the voice interaction intention "display the real-time image of the UAV" and the slot information "display", "pane 2", "UAV 1", "image", the message transmission format supported by the interactive software is combined, and the message is sent to the interactive software through the "signal-slot" mechanism. The interactive software displays the real-time image of UAV 1 on pane 2 according to the received message, completing the entire voice interaction process.

[0041] For this example, although the voice command "display the image of UAV 1" clearly expresses the interaction intention, the slot information is incomplete, resulting in the software being unable to directly execute this command. However, the method of the present invention combines the eye movement information during the commander's issuing of the command, infers that the image of UAV 1 should be displayed on pane 2, and can complete the confirmation of the interaction intention and slot information without multiple rounds of dialogue. Further, if the voice command is "display UAV No. 1", then combining the eye movement and the knowledge graph, it is parsed that the commander is fixating on pane 2, and the function of pane 2 is to display the real-time image of the UAV. Therefore, it can also be inferred that the real intention is "display the real-time image of UAV 1 on pane 2". Although the two important slot information of "where to display" and "what to display" are missing in this case, the intention recognition of the voice interaction command is still completed by combining the eye movement and the knowledge graph.

[0042] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.

[0043] In addition, it should be understood that although this specification is described in terms of embodiments, not every embodiment contains only one independent technical solution. This narrative manner of the specification is only for clarity. Those skilled in the art should regard the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. An eye movement-assisted speech interaction intention recognition method, characterized in that Specifically, it includes the following steps: 1) Conduct intent recognition and slot filling on the voice interaction instruction to obtain the interaction intent and slot information; 2) Identify the intent manipulation area of the interaction personnel on the interaction software UI; Utilize an eye tracker or a known eye movement tracking algorithm to perform real-time detection of the gaze points of the desktop interaction personnel on the UI. The area where the interaction personnel are currently gazing is the intent manipulation area; 3) Extract the sub-graph of the knowledge graph associated with the intent manipulation area; obtain the function of the intent manipulation area, i.e., the operable items on the UI, by parsing the mapping relationship between the UI layout and functions of the interaction software, and then perform entity linking from the software knowledge graph according to the entities associated with the function of the intent manipulation area to extract the sub-graph of the knowledge graph; 4) Analyze the slot information associated with the interaction intent identified in step 1), and extract and improve the slot information in step 1) from the sub-graph of the knowledge graph obtained in step 3); 5) Based on the interaction intent in step 1) and the improved slot information in step 4), index and match the response method of the voice instruction from the knowledge graph and send it to the interaction software for response; according to the message format and message interaction method supported by the interaction software, send the interaction intent obtained in step 1) and the slot information improved and confirmed according to step 4) to the interaction software by indexing and matching the response method of the corresponding interaction intent of the relevant entities from the knowledge graph, and the interaction software then makes a response to achieve natural voice interaction for desktop software.

2. The eye movement-assisted speech interaction intention recognition method according to claim 1, wherein In step 1), the interaction intent and slot information are obtained through a known joint intent recognition and slot filling algorithm.

3. The eye movement-assisted voice interaction intention recognition method according to claim 1 or 2, characterized in that In step 4), according to the entity information contained in the sub-graph of the knowledge graph, the slot information in the voice instruction in step 1) that includes component defaults, unclear references, and insufficient constraints is uniquely supplemented and confirmed to improve the slot information; this process validates the legality of the existing slot information according to the sub-graph of the knowledge graph while improving the missing or unclear slot information.

Images