Interaction instruction identification method and device and electronic whiteboard cooperation system

Abstract

The invention relates to an interaction instruction identification method and device and an electronic whiteboard cooperation system, and the method comprises the steps: obtaining the interaction data of each mode in a target scene; based on a multi-modal cross attention mechanism, carrying out data fusion on the interaction data of each modal to obtain fused modal data, and combining the fused modal data and the interaction data of each modal into target modal data; judging whether the user intentions corresponding to the target modal data have intention conflicts or not, and if yes, determining a to-be-executed intention from different user intentions based on a preset conflict resolution rule in the target scene; and generating an execution operation instruction according to the to-be-executed intention. Based on a multi-modal cross attention mechanism, accurate fusion of interaction data of different modals can be completed, the data isomerism of different modals and the expression form difference of the same intention are solved, and the accuracy of interaction instruction recognition is finally improved in combination with a conflict resolution rule.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction, and in particular to interactive command recognition methods, devices, and electronic whiteboard collaborative systems. Background Technology

[0002] Human-computer interaction (HCI) technology, as a crucial component of information technology, is widely applied in education, office work, smart homes, and industrial control. As users increasingly demand natural and efficient interaction, traditional single-modal interaction methods are struggling to meet the needs of complex scenarios. (For example, in interactive whiteboards, pure touch interaction suffers from drawbacks such as long-distance operation failures, parallel operation conflicts, and difficulties in precise operation due to reliance on physical contact with fingers or touchpads, leading to decreased interaction efficiency and increased complexity.) Therefore, multimodal interaction technology has become a research hotspot, aiming to improve the flexibility and efficiency of interaction by integrating multiple input channels (such as touch, voice, vision, and gestures). However, current technologies still have limitations in the depth and coordination of multimodal fusion.

[0003] While some related technologies support multiple input methods such as voice, gestures, touch, and eye tracking, these modalities are often used independently. For example, eye tracking cannot be used when using gesture recognition, leading to redundant and fragmented interaction functions, unnatural interactions, high learning costs, and poor user experience. Other related technologies employ a simple fusion of multimodal interaction methods, allowing voice, gestures, touch, and eye tracking to work simultaneously and then providing feedback to the user after simple fusion. These technologies often only perform simple priority processing, which may lead to priority confusion and modal conflicts during parallel input, further impacting user experience. Therefore, they cannot achieve accurate recognition of interactive commands in multimodal input scenarios.

[0004] There is currently no effective solution to the problem of low accuracy in recognizing interactive commands under multimodal input in related technologies. Summary of the Invention

[0005] This embodiment provides an interactive command recognition method, device, and electronic whiteboard collaborative system to solve the problem of low accuracy in interactive command recognition under multimodal input in related technologies.

[0006] Firstly, this embodiment provides an interactive command recognition method for multimodal human-computer interaction; the method includes:

[0007] Acquire interaction data for each modality in the target scenario;

[0008] Based on the multimodal cross-attention mechanism, the interaction data of each modality is fused to obtain fused modal data, and the fused modal data and the interaction data of each modality are combined to form the target modal data;

[0009] Determine whether there is an intent conflict between the user intent corresponding to the target modal data. If so, determine the intent to be executed from the different user intents based on the preset conflict resolution rules in the target scenario.

[0010] Based on the stated intent to be executed, an execution operation instruction is generated.

[0011] In some embodiments, acquiring modal interaction data in the target scene includes:

[0012] The input data generated by different signal processing modules in the target scene is obtained; wherein, different signal processing modules receive and process input signals of different modes to obtain the input data.

[0013] The input data are processed for time alignment and spatial alignment to obtain the temporally and spatially aligned modal interaction data.

[0014] In some embodiments, the input data are subjected to time alignment and spatial alignment to obtain spatiotemporally aligned modal interaction data, including:

[0015] Based on the signal timestamps corresponding to each of the input data, a reference time point for time alignment is determined.

[0016] Align each of the input data to the reference time point;

[0017] The time-aligned input data is transformed into a preset reference coordinate system to obtain spatially aligned modal interaction data.

[0018] In some embodiments, based on a multimodal cross-attention mechanism, the modal interaction data is fused to obtain fused modal data, and the fused modal data and the modal interaction data are combined to form target modal data, including:

[0019] Based on the constructed cross-modal embedding space, the modal interaction data are mapped to a unified semantic dimension to obtain the modal interaction data after the semantic dimension is unified.

[0020] Based on the multimodal cross-attention mechanism, the modal contribution of each modal interaction data after semantic dimension unification is determined;

[0021] Based on the modal contribution, modal fusion is performed on the semantically unified interaction data of each modality to obtain fused modal data.

[0022] In some embodiments, determining whether there is an intent conflict in the user intent corresponding to the target modality data includes:

[0023] Semantic structural separation and identification are performed on the semantic features associated with different modal data in the target modal data to obtain different modal semantic features corresponding to different semantic structures;

[0024] Semantic reorganization of different modal semantic features is performed to determine the user intent corresponding to the target modal data;

[0025] In cases where the target modal data contains multiple different user intentions, determine whether there is an intention conflict between the different user intentions.

[0026] In some embodiments, the target modal data includes speech recognition data, gesture recognition data, and gaze recognition data; semantic structure separation and recognition are performed on the semantic features associated with different modal data in the target modal data to obtain different modal semantic features corresponding to different semantic structures, including:

[0027] The speech recognition data is subjected to action semantic extraction to obtain the action semantic features in the user's intent;

[0028] The gesture recognition data is subjected to semantic extraction of operation parameters to obtain the operation parameter features in the user's intent;

[0029] The gaze recognition data is subjected to target localization semantic extraction to obtain the target localization features in the user's intent.

[0030] In some embodiments, based on preset conflict resolution rules for the target scenario, the intent to be executed is determined from different user intents, including:

[0031] Based on the pre-configured initial weights of each modal data in the target modal data, the scene coefficients of the target scene, and the confidence levels of each modal data in the target modal data, the modal weights of each modal data in the target modal data are determined.

[0032] When the conflict resolution rule is a priority-based coverage rule, it is determined whether the confidence level of the modality data with the highest modality weight in the target modality data is greater than a preset confidence threshold; if so, the user intent corresponding to the modality data with the highest weight in the target modality data is determined as the intent to be executed.

[0033] When the conflict resolution rule is a semantic negotiation rule, it is determined whether each user intent satisfies the semantic merging condition; if so, a composite instruction is generated based on each user intent; otherwise, the user intent corresponding to the modality data with the highest weight in the target modality data is determined as the intent to be executed.

[0034] When the conflict resolution rule is a two-confirmation rule, according to each user intent, an intent to be confirmed is output to the user, and in response to the user's selection operation for the intent to be confirmed, the intent selected by the user is taken as the intent to be executed.

[0035] In some embodiments, the target scenario is an interactive whiteboard scenario; based on the intent to be executed, an execution operation instruction is generated, including:

[0036] Based on the intended action, generate operation instructions for the target operation on the electronic whiteboard.

[0037] Secondly, this embodiment provides an interactive command recognition device for multimodal human-computer interaction; the interactive command recognition device includes: an acquisition module, a data fusion module, a conflict resolution module, and a command generation module; wherein:

[0038] The acquisition module is used to acquire the interaction data of each modality in the target scene;

[0039] The data fusion module is used to fuse the interaction data of each modality based on a multimodal cross-attention mechanism to obtain fused modal data, and to combine the fused modal data and the interaction data of each modality into target modal data.

[0040] The conflict resolution module is used to determine whether there is an intent conflict between the user intent corresponding to the target modal data. If so, it determines the intent to be executed from the different user intents based on the preset conflict resolution rules in the target scenario.

[0041] The instruction generation module is used to generate execution operation instructions based on the intended action.

[0042] Thirdly, this embodiment provides an electronic whiteboard collaborative system, including: an electronic whiteboard, a signal processing module, and a server; wherein the electronic whiteboard and the signal processing module are both connected to the server;

[0043] The signal processing module is used to receive input signals from users in different modalities, process the input signals, and send the processing results to the server.

[0044] The server is used to execute the interactive instruction recognition method described in the first aspect above;

[0045] The electronic whiteboard is used to respond to execution operation instructions generated by the server.

[0046] Compared with related technologies, this embodiment provides an interactive command recognition method, device, and electronic whiteboard collaborative system. The interactive command recognition method acquires multimodal interaction data in a target scenario; based on a multimodal cross-attention mechanism, it fuses the multimodal interaction data to obtain fused modal data, and combines the fused modal data with the multimodal interaction data to form target modal data; it determines whether there is an intent conflict corresponding to the target modal data; if so, it determines the intent to be executed from different user intents based on preset conflict resolution rules in the target scenario; and it generates an execution operation command based on the intent to be executed. This method can accurately fuse different modal interaction data based on a multimodal cross-attention mechanism, solving the problems of data heterogeneity across different modalities and differences in the expression of the same intent, and ultimately improving the accuracy of interactive command recognition by combining conflict resolution rules.

[0047] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0048] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0049] Figure 1 This is a hardware structure block diagram of the terminal of the interactive command recognition method in this embodiment;

[0050] Figure 2 This is a flowchart of the interactive command recognition method in this embodiment;

[0051] Figure 3 These are schematic diagrams illustrating the business processing procedures of some embodiments;

[0052] Figure 4 This is a flowchart of an interactive instruction recognition method for some embodiments;

[0053] Figure 5 This is a flowchart illustrating the processing of semantic negotiation rules in some embodiments;

[0054] Figure 6 These are flowcharts illustrating the conflict resolution process in some of the embodiments;

[0055] Figure 7 These are semantic decoupling and semantic reorganization architecture diagrams of some embodiments;

[0056] Figure 8 This is a structural block diagram of the interactive command recognition device in this embodiment;

[0057] Figure 9 This is a schematic diagram of the electronic whiteboard system in this embodiment. Detailed Implementation

[0058] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0059] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these” used in this application do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to these processes, methods, products, or devices. Words such as “connected,” “linked,” and “coupled” used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. Normally, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," "third," etc., used in this application are merely to distinguish similar objects and do not represent a specific order of objects.

[0060] The method embodiments provided in this example can be executed on a terminal, computer, or similar computing device. For example, it can run on a terminal. Figure 1 This is a hardware structure block diagram of the terminal for the interactive command recognition method in this embodiment. For example... Figure 1 As shown, a terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The terminal may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the terminal described above. For example, the terminal may also include components that are larger than... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown are illustrated.

[0061] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the interactive instruction recognition method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0062] The transmission device 106 is used to receive or send data via a network. This network includes a wireless network provided by the terminal's communication provider. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 can be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0063] This embodiment provides an interactive command recognition method. Figure 2 This is a flowchart of the interactive command recognition method in this embodiment, such as... Figure 2 As shown, the process includes the following steps:

[0064] Step S210: Obtain the interaction data of each modality in the target scene.

[0065] The target scenario can be any scenario suitable for multimodal intent recognition for multimodal human-computer interaction, including interactive scenarios such as electronic whiteboards, smart home control, and augmented reality games. The interaction data for each modality in the target scenario can be generated by signal processing of the modal input signals generated by the user interaction layer by the modal signal acquisition modules set up in the target scenario. For example, in the target scenario, the user interaction layer generates one or more modal input signals such as voice, gesture, touch, and gaze. The voice recognition module, gesture recognition module, touch detection module, and eye tracker signal acquisition modules acquire the corresponding modal input signals, process and recognize them, and output the corresponding modal recognition data (e.g., voice recognition results, gesture recognition results, gaze tracking results, touch detection results). In some embodiments, the data output by the signal acquisition modules can be further processed (e.g., spatiotemporal alignment) to obtain the aforementioned modal interaction data.

[0066] Step S220: Based on the multimodal cross-attention mechanism, data fusion is performed on the interaction data of each modality to obtain fused modal data, and the fused modal data and the interaction data of each modality are combined to form the target modal data.

[0067] This step requires cross-modal data fusion of interaction data from different modalities. Specifically, a multimodal cross-attention (MCA) mechanism is introduced. MCA is a key mechanism that enables deep and refined interaction between different modalities (such as text, images, and speech). Its core idea is to allow one modality (as a query) to actively inquire about or focus on the most relevant and important parts of another modality (providing information). This step aims to eliminate the inconsistencies in data dimensions such as speech (temporal signals), gestures (spatial actions), and gaze (coordinate points) through cross-modal data fusion, thus solving the problem of modal heterogeneity. Furthermore, considering the differences in how different modalities express the same intention (for example, "amplify" in a speech signal corresponds to "pinch" in a gesture signal), semantic dimension unification of the interaction data from different modalities is necessary. Additionally, the real-time performance of multimodal parsing, fusion, and response also needs to be considered. Therefore, semantic parsing (analyzing the semantics of voice commands based on natural language processing and speech recognition) and semantic binding (dynamically associating the semantics of different modalities, such as associating the semantics of "magnify" expressed by gestures with the semantics of "this" expressed by speech to focus on the target, and using the semantics expressed by gaze, such as looking at a certain area, as an auxiliary, prioritizing the processing of interactive commands that achieve the gazed area) can be achieved. Thus, by combining a multimodal cross-attention mechanism, the interactive data of each modality can be unified under the same semantic dimension. By introducing a multimodal cross-attention mechanism, the spatiotemporal relationships between different modal interactive data can be explicitly captured (e.g., capturing the spatiotemporal relationship between gesture trajectories and gaze movements), thereby eliminating the modal heterogeneity problem, achieving fused modal data formed by cross-modal fusion, and ultimately obtaining the target modal data.

[0068] For example, the aforementioned fused modal data may include at least one or more of the following: fused modal data resulting from the fusion of gesture interaction data and voice interaction data; fused modal data resulting from the fusion of gaze interaction data and voice interaction data; and fused modal data resulting from the fusion of gaze interaction data, voice interaction data, and gesture interaction data. It is understood that those skilled in the art can also fuse interaction data from other different modalities according to the needs of actual application scenarios. Therefore, based on the modal interaction data obtained in step S210, one or more fused modal data can be further added to expand the modal richness.

[0069] Step S230: Determine whether there is an intent conflict between the user intent corresponding to the target modal data. If so, determine the intent to be executed from different user intents based on the preset conflict resolution rules in the target scenario.

[0070] After completing the cross-modal data fusion described above, it can be determined whether there are intent conflicts among the various user intents expressed in the target modality data. An intent conflict refers to the simultaneous existence of two or more intents that contradict each other and cannot be satisfied simultaneously. These intents exhibit logical or executional mutual exclusion, making it impossible for the execution vehicle in the target scenario to determine which one to prioritize, or to execute them simultaneously without violating logic. For example, the intent corresponding to voice interaction data might be "delete," while the intent corresponding to touch operation might be "save." Therefore, by identifying the various user intents contained in the target modality data, it is possible to determine whether intent conflicts exist.

[0071] When a conflict of intents is identified, a pre-defined conflict resolution rule for the target scenario needs to be invoked. Based on this rule, the final intent to be executed is determined from multiple conflicting user intents. This conflict resolution rule can be a pre-set logic tailored to the target scenario, adapted to the accuracy and rigor required for intent recognition in that scenario, and capable of dynamic context awareness. It can be implemented using computer programming languages ​​or configuration files. For example, when the target scenario is a high-risk scenario such as medical settings where strict avoidance of misoperation is required, the corresponding conflict resolution rule is to force the intent corresponding to the higher-priority modality to override the intent of the lower-priority modality to prevent security issues caused by user misoperation. When the target scenario is a scenario with a certain degree of fault tolerance, such as a meeting, the corresponding conflict resolution rule can be a semantic negotiation rule to determine whether semantics can be merged. When the target scenario is a scenario with high fault tolerance, such as education, the corresponding conflict resolution rule can require the user to confirm again.

[0072] Therefore, by using conflict resolution rules, we can minimize the error rate while ensuring the efficiency of intent recognition. This upgrades the static rules of traditional conflict handling to dynamic context awareness that are adapted to the target scenario, enabling machines to accurately identify the user's actual intent in a human-computer interaction.

[0073] Step S240: Generate execution operation instructions based on the intent to be executed. Based on the final determined intent to be executed, generate the final execution operation instructions and provide feedback on the operation results through visual (e.g., highlighting), auditory (e.g., voice confirmation), and tactile (e.g., vibration).

[0074] In related technologies, a single-modal interaction mode is often used, or even if multimodal interaction is used, it is done in an independent manner. For example, in interactive whiteboards, some technologies use a pure touch mode, which requires users to walk to the whiteboard to operate it, making air-based operation impossible, thus reducing efficiency and failing to meet the needs of long-distance operation. Furthermore, when multiple users touch simultaneously, accidental touches are prone to occur, such as misrecognition of palms. Additionally, this single-modal interaction also presents difficulties in fine-tuning; for example, selecting a 5-pixel (px) wide line on a 1:100 scale design drawing using touch is prone to errors due to insufficient finger precision. In some existing technologies, interactive whiteboards add gesture recognition to the touch mode, recognizing gestures within 1 meter (such as gestures to indicate "previous page" or "next page"). However, in this approach, the various modalities are independent and cannot be combined; different modalities can only serve as alternative input methods rather than complementary enhancements, failing to utilize the combined advantages of multimodal interaction. Furthermore, this type of approach suffers from unnatural interaction and a lack of adaptive modal switching. For example, when using touch drawing, switching to voice requires manually activating voice, making the operation complex and cumbersome, resulting in a poor user experience. Additionally, users need to remember which modalities are used in which scenarios, such as which scenarios require only voice and which require only gestures, which also poses a significant challenge to their learning abilities.

[0075] Furthermore, some existing technologies involve a simple fusion of multimodal interaction modes. These methods set fixed priorities and determine user intentions based on these priorities. However, this approach fails to consider conflicts between different intentions and lacks a dynamic conflict arbitration mechanism. When priorities become confused, the true intention cannot be confirmed, reducing the efficiency of human-computer interaction.

[0076] This embodiment, through steps S210 to S240, fuses multimodal interaction methods to dynamically associate user input from different modalities, such as voice commands, gestures, and gaze focus. Furthermore, considering the potential for competition during modal input, this embodiment introduces conflict resolution rules to dynamically and specifically understand the user's true intent based on the target scenario. Therefore, it solves the problems of traditional electronic whiteboards, such as single-mode interaction, modal fragmentation, and inefficient operation. By integrating multiple input methods such as voice, gestures, touch, and gaze, combined with dynamic intent understanding and conflict arbitration, a natural, efficient, and accurate human-computer interaction experience is achieved. Specifically, based on a dynamic attention fusion mechanism, different modalities are mapped to a unified semantic space through comparative learning, thereby achieving multimodal collaborative effects. This enables seamless coordination of multimodal data such as voice, gestures, and gaze, allowing users to automatically coordinate multimodal features without consciously distinguishing interaction methods, thus improving recognition accuracy. The conflict resolution rules improve the accuracy of interactive command recognition, enhancing the accuracy and real-time nature of user intent, overcoming the shortcomings of single-modal recognition errors and inaccurate multimodal intent understanding in existing technologies. By adopting a multimodal fusion approach, any modality can be used as an input to issue commands and interact with the system. This allows users to interact without relying on a single modality, improving fault tolerance and avoiding the problem of single-modal failures causing the interactive system to crash, which is common in related technologies.

[0077] Therefore, through the aforementioned steps S210 to S240, interaction data of each modality in the target scenario is acquired; based on the multimodal cross-attention mechanism, the interaction data of each modality is fused to obtain fused modal data, and the fused modal data and the interaction data of each modality are combined to form target modal data; it is determined whether there is an intent conflict corresponding to the target modal data; if so, based on the preset conflict resolution rules in the target scenario, the intent to be executed is determined from different user intents; and an execution operation instruction is generated according to the intent to be executed. This mechanism can accurately fuse interaction data of different modalities based on the multimodal cross-attention mechanism, solving the problems of data heterogeneity of different modalities and differences in the expression of the same intent, and ultimately improving the accuracy of interaction instruction recognition by combining conflict resolution rules.

[0078] In one embodiment, based on the above step S210, obtaining the modal interaction data in the target scene may specifically include:

[0079] The system acquires input data generated by different signal processing modules in the target scene. The different signal processing modules receive and process input signals of different modes to obtain input data. The system performs time alignment and spatial alignment processing on each input data to obtain spatiotemporally aligned interaction data of each mode.

[0080] For example, the aforementioned different signal processing modules may include a speech recognition module, a gesture recognition module, a touch detection module, and an eye tracker or vision algorithm module. The speech recognition module collects the user's input speech signal through a microphone array and uses noise reduction and Automatic Speech Recognition (ASR) technology to convert the speech signal into recognized text data, which contains semantics expressing the user's intent. The gesture recognition module can use devices such as depth cameras (e.g., RGB-D) or infrared optical sensors to capture user gesture images and identify the type of gesture (e.g., swipe gestures, zoom gestures, rotation gestures, etc.). The touch detection module can use capacitive touch detection or electromagnetic touch detection to detect the user's direct operations (e.g., writing, clicking, double-clicking, etc.). Furthermore, for eye tracking, an eye tracker or vision algorithm can be used to identify the user's eye gaze point to assist in target locking. In this way, different modalities of input data can be constructed.

[0081] After obtaining the aforementioned input data, timestamp alignment techniques can be used to align the input data of each modality to a unified time point. Specifically, a reference time point for time alignment can be determined based on the signal timestamps of the input signals corresponding to each input data point when they were acquired. Then, all input data whose signal timestamps satisfy a certain time window are aligned to this reference time point, thus completing the time alignment. Furthermore, a unified coordinate system can be established, and different input data can be transformed into this coordinate system to achieve spatial alignment.

[0082] This embodiment considers that time delays or spatial positioning differences between input data from different modalities can further lead to command fragmentation. For example, when a user expresses the intention to "move this block" via voice, the corresponding gesture operation may be delayed by 0.5 seconds, causing the interactive system to mistakenly apply the gesture to other objects. To address this, this embodiment, after initially processing the user's input signal to obtain input data for each modality, introduces a spatiotemporal alignment algorithm to calibrate the spatiotemporal relationship of each modal input data (such as voice recognition data, gesture recognition data, and gaze recognition data) in real time. This eliminates command misalignment caused by time delays and spatial positioning differences in input data caused by hardware or algorithms (e.g., voice recognition requires 200 milliseconds, while gesture detection requires 50 milliseconds), solving the modal fragmentation problem and preventing the gesture from prematurely triggering other operations (such as page turning) when the user expresses "zoom in on this," thereby improving the accuracy of interactive command recognition and enhancing the coordination between different modalities.

[0083] In one embodiment, the input data is subjected to time alignment and spatial alignment to obtain spatiotemporally aligned modal interaction data, which may specifically include:

[0084] Based on the signal timestamps corresponding to each input data, a reference time point for time alignment is determined; each input data is time-aligned to the reference time point; the time-aligned input data is then transformed into a preset reference coordinate system to obtain spatially aligned modal interaction data.

[0085] Specifically, there are hardware latency differences between input data of different modalities. For example, speech recognition requires 200 milliseconds, while gesture detection requires 50 milliseconds, which can lead to instruction misalignment. To address this, in this embodiment, a unified clock source is allocated to all sensors used to collect user input signals, such as microphones, cameras, and touchscreens, thereby controlling the latency error within ±10 milliseconds. This can be achieved by taking the earliest timestamp (base_time) from the signal timestamps corresponding to each input data, for example, from the signal timestamps at the output time of each input data. For instance, the earliest timestamp can be taken from the signal timestamps corresponding to speech recognition data, gesture recognition data, and gaze detection data. Then, the longest latency (max_delay) among the input data of these multiple modalities is calculated. The latency corresponding to the input data of different modalities is the time required for the input signal to be recognized and processed by the corresponding signal processing module and converted into input data. For example, the longest latency (max_delay) can be the latency corresponding to the speech modality (the speech modality has higher latency due to ASR processing and network transmission). Next, based on the earliest timestamp `base_time` and the longest delay, the base time point is determined as `base_time + max_delay`. Then, input data whose signal timestamps fall between `base_time` and `base_time + max_delay` are considered collaborative data. The input data of different collaborative modalities are aligned to this base time point in the time dimension (e.g., all modal input data are associated with and stored at this base time point).

[0086] Furthermore, after aligning in the time dimension, a unified coordinate system can be established. For example, in a whiteboard collaborative interaction scenario, the coordinate system where the whiteboard screen is located can be used as a unified reference coordinate system. The position coordinates of input data from modalities such as gestures, gaze, and touch can be mapped to this reference coordinate system to achieve spatial alignment.

[0087] This embodiment solves the problem of modal fragmentation and avoids instruction misalignment by performing spatiotemporal alignment processing on the input data of each modality.

[0088] Furthermore, in one embodiment, based on the above step S220, and based on the multimodal cross-attention mechanism, data fusion is performed on the interaction data of each modality to obtain fused modal data, and the fused modal data and the interaction data of each modality are combined to form target modal data, which may specifically include:

[0089] Based on the constructed cross-modal embedding space, the modal interaction data are mapped to a unified semantic dimension to obtain the modal interaction data after semantic dimension unification. Based on the multimodal cross-attention mechanism, the modal contribution degree corresponding to each modal interaction data after semantic dimension unification is determined. Based on the modal contribution degree, modal fusion is performed on the modal interaction data after semantic dimension unification to obtain fused modal data.

[0090] This involves constructing a cross-modal embedding space to map intermodal interaction data to a unified semantic dimension. Specifically, the cross-modal embedding space is a shared, high-dimensional vector space (typically hundreds to thousands of dimensions). Within this space, interaction data from different modalities (such as speech, gesture, and gaze modalities) will have similar vector representations (embeddings) if they are semantically similar, while semantically dissimilar data will have significantly different vector representations. Therefore, vector encoding can be performed on the intermodal interaction data within the cross-modal embedding space to map them to a unified semantic dimension. The pseudocode for mapping intermodal interaction data to a unified semantic dimension is as follows:

[0091] # Pseudocode: Multimodal Feature Encoding

[0092] class ModalityEncoder:

[0093] def encode_voice(self, speech_text): # Speech text encoding

[0094] return bert_model(speech_text)[CLS_token] # 768-dimensional vector

[0095] def encode_gesture(self, gesture_points): # Gesture trajectory encoding

[0096] return resnet3d(gesture_points) # 512-dimensional vector

[0097] def encode_gaze(self, gaze_coord): # Gaze coordinate encoding

[0098] return mlp([x,y,Δt]) # 256-dimensional vector;

[0099] Specifically, by using speech-text encoding, gesture trajectory encoding, and gaze coordinate encoding, speech recognition data, gesture recognition data, and gaze detection data are all mapped to a unified semantic dimension to obtain vector representations of each modality.

[0100] Subsequently, based on the multimodal cross-attention mechanism, the modal contribution of each modality can be automatically adjusted according to the input data, explicitly capturing the spatiotemporal relationship between data from different modalities. The pseudocode for the modality fusion process is as follows:

[0101] "class MultimodalFusion(nn.Module):

[0102] def forward(self, voice_emb, gesture_emb, gaze_emb):

[0103] # Generate query vector Q (based on speech)

[0104] Q = self.Wq(voice_emb)

[0105] # Gestures / eye contact as key-value pairs

[0106] K = self.Wk(torch.cat([gesture_emb, gaze_emb]))

[0107] V = self.Wv(torch.cat([gesture_emb, gaze_emb]))

[0108] # Calculate cross-modal attention weights

[0109] attn_weights = torch.softmax(Q @ K.Ts / sqrt(dim), dim=-1)

[0110] # Weighted fusion

[0111] fused_emb = attn_weights @ V

[0112] return fused_emb”;

[0113] The process involves generating a query vector Q based on the vector representation of the interaction data from the speech modality, and generating key-value pairs K and V using the vectors corresponding to the interaction data from the gesture or gaze modality. Cross-modal attention weights are calculated, and then the vector representations from different modalities are fused based on these weights. During the fusion process, the modal contribution is an internal dynamic trade-off mechanism used to generate a more accurate, robust, and context-relevant fused feature representation. Specifically, the multimodal cross-attention mechanism does not directly concatenate vector representations from different modalities. Instead, based on the current specific input, one modality queries another modality and extracts the most relevant information to enhance itself. Modal contribution determines how much information can be borrowed from other modalities, and which specific parts of the information can be borrowed to supplement or correct the representation of the current modality. This allows vector representations with high modal contributions to be fused. Furthermore, when the information in a certain modality is ambiguous or noisy, the multimodal cross-attention mechanism also allows the model to rely on another more deterministic modality to make decisions, automatically reducing the modal contribution of uncertain modalities.

[0114] Therefore, modal contribution can serve as a weight to amplify signals from reliable modalities and suppress noise from unreliable or irrelevant modalities. If a modality is missing (e.g., only speech without gestures), then the modal contribution of that modality as a key-value pair will naturally drop to zero. In this case, modal fusion relies entirely on the existing modalities for inference to ensure robustness.

[0115] Therefore, this embodiment achieves natural and accurate fusion and interaction of different modal interaction data through cross-modal semantic fusion, thereby improving the accuracy of interaction command recognition under multimodal input.

[0116] In one embodiment, based on the above step S230, determining whether there is an intent conflict with the user intent corresponding to the target modality data may specifically include:

[0117] Semantic structural separation and identification are performed on the semantic features associated with different modal data in the target modal data to obtain different modal semantic features corresponding to different semantic structures; semantic recombination of different modal semantic features is performed to determine the user intents corresponding to the target modal data; when the target modal data contains multiple different user intents, it is determined whether there are intent conflicts between different user intents.

[0118] Semantic structural separation recognition allows the semantic features associated with different modalities to correspond to different parts of the complete semantics. For example, the semantic features corresponding to the speech modality express "what to do," the semantic features corresponding to the gesture modality express "how to do it" (e.g., rotating 30 degrees counterclockwise), and the semantic features corresponding to the gaze modality express "to whom to do it" (e.g., looking at a chart on one side of the screen). This achieves semantic decoupling through disentangled representation learning. After semantic decoupling, the semantic features corresponding to different modalities are semantically recombined to determine the complete user intent. For example, if the semantic feature corresponding to the gaze modality expresses "progress bar," then the target object to be operated on is the progress bar. Based on the semantic features corresponding to the gesture modality, the complete user intent is to slide the progress bar. If the semantic feature corresponding to the gaze modality expresses an image or text, and the semantic feature corresponding to the speech modality is "delete," then the complete user intent is to delete the corresponding image or text.

[0119] Therefore, when performing interactive command recognition, semantic structure separation recognition is used to complete semantic decoupling, and then the semantic features are semantically recombined to determine the complete user intent, thereby improving the accuracy of interactive command recognition.

[0120] In one embodiment, the target modal data includes speech recognition data, gesture recognition data, and gaze recognition data; semantic structure separation and recognition are performed on the semantic features associated with different modal data in the target modal data to obtain different modal semantic features corresponding to different semantic structures, including:

[0121] Action semantic extraction is performed on speech recognition data to obtain action semantic features in user intent; operation parameter semantic extraction is performed on gesture recognition data to obtain operation parameter features in user intent; target localization semantic extraction is performed on gaze recognition data to obtain target localization features in user intent.

[0122] Action semantic features express what actions the user needs to perform, such as deleting, saving, moving, or modifying. Operation parameter features express how the user needs to do something, such as rotating 30 degrees or moving 10 pixels to the left. Target localization features indicate what the object of the action is, such as a piece of text on the screen, a button, or an icon. Therefore, by decoupling and recombining the semantics corresponding to the speech modality, gesture modality, and gaze modality respectively, the accuracy of data fusion of these modalities can be improved.

[0123] In addition, in the case of multimodal input, if the user needs to manually switch the input mode, it goes against the human instinctive multimodal collaboration habit. In this regard, this embodiment can also realize seamless modal switching technology and automatically recognize the user's interaction commands (such as enhancing voice interaction when both hands are detected to be occupied).

[0124] Additionally, in one embodiment, determining the intent to be executed from different user intents based on preset conflict resolution rules for the target scenario may include:

[0125] Based on the pre-configured initial weights of each modality in the target modal data, the scene coefficients of the target scene, and the confidence levels of each modality in the target modal data, the modal weights of each modality in the target modal data are determined. When the conflict resolution rule is a priority-based coverage rule, it is determined whether the confidence level corresponding to the modality with the highest modal weight in the target modal data is greater than a pre-set confidence threshold. If so, the user intent corresponding to the modality with the highest weight in the target modal data is determined as the intent to be executed. When the conflict resolution rule is a semantic negotiation rule, it is determined whether each user intent meets the semantic merging condition. If so, a composite instruction is generated based on each user intent. Otherwise, the user intent corresponding to the modality with the highest weight in the target modal data is determined as the intent to be executed. When the conflict resolution rule is a secondary confirmation rule, the intent to be confirmed is output to the user based on each user intent, and in response to the user's selection operation for the intent to be confirmed, the intent selected by the user is taken as the intent to be executed.

[0126] Specifically, in this embodiment, a weighted voting mechanism can be used to dynamically allocate the modal weights of each modal data based on the context. Specifically, the modal weights can be:

[0127] Modal weights = initial weights × scene coefficients × confidence level;

[0128] The initial weights can be pre-configured weights for each modality in the target scenario. For example, in a meeting scenario, the voice modality is considered more important, so its initial weight is 0.7. The scenario coefficients can be pre-determined coefficients for each modality in the target scenario. For example, the scenario coefficient for the voice modality in a meeting scenario is 1.2. The confidence level can be the confidence level of the signal recognition algorithm for each modality. For example, the confidence level for the voice modality in a meeting scenario is 90%. Thus, the modality weight for the voice modality in a meeting scenario is 0.7 × 1.2 × 0.9 = 0.756. Similarly, for a fusion modality combining gaze, voice, and gestures in an education scenario, with an initial weight of 0.9, a confidence level of 95%, and a scenario coefficient of 1.2, the corresponding modality weight is 0.9 × 1.2 × 0.95.

[0129] Among them, conflict resolution rules are a supplementary mechanism to modality priority allocation. When semantic contradictions exist (e.g., deletion and preservation semantics coexist), spatiotemporal separation exists (e.g., the required operation goals are inconsistent), and low-confidence competition exists (the dominant modality cannot be clearly identified), intent conflicts can be resolved through conflict resolution rules.

[0130] Specifically, after obtaining the modal weights of each modal data in the target modal data, it is necessary to determine whether there are any conflicts between the user intentions corresponding to each modal data in the target modal data. If there are no conflicts, the instruction can be executed directly based on the user intention; if there are conflicts, it is necessary to resolve the intention conflict based on conflict resolution rules. The specific logic of the conflict resolution rules varies depending on the target scenario. For example, in high-risk areas such as healthcare where operational safety is paramount, priority-based rules are needed to resolve conflicts and avoid accidental operations. In conventional interaction scenarios such as meetings, semantic consistency negotiation rules are used to combine different user intentions as much as possible to generate composite instructions to meet user expectations. In scenarios with higher fault tolerance, such as education, conflict resolution rules requiring secondary user confirmation can be adopted, allowing the user to choose the final solution.

[0131] For example, the priority-based mandatory coverage rule can be: when the confidence of the highest priority modality is greater than the preset confidence threshold, the intent corresponding to the high priority modality is taken as the intent to be executed, the instruction corresponding to the intent to be executed is generated and executed, and the low priority input is discarded; otherwise, a negotiation strategy is initiated to determine the final intent to be executed.

[0132] The semantic negotiation rule can be as follows: when the semantic corresponding to the speech modality is rotation and the semantic corresponding to the gesture modality is 30 degrees, the two semantics can be combined into a rotation of 30 degrees; when the semantic corresponding to the speech modality is deletion and the semantic corresponding to the gesture modality is save, deletion is forced.

[0133] The secondary confirmation rule can be: directly output the conflict intent to the user in the form of a prompt box, such as: "Conflict detected: delete vs save, please confirm the operation?", and let the user select the final intent to be executed based on the prompt box.

[0134] Therefore, this embodiment addresses the situation where competition and conflict are likely to occur when different modal inputs are used. By introducing dynamic allocation of modal priorities and conflict resolution rules, on the one hand, the primary modality can be automatically selected based on the context (such as user role and task stage). On the other hand, conflict resolution can identify the user's final intention to execute, thereby improving the accuracy of interactive command recognition.

[0135] In one embodiment, the target scenario is an interactive scenario using an electronic whiteboard; generating execution instructions based on the intent to be executed may include: generating operation instructions for the operation target on the electronic whiteboard based on the intent to be executed.

[0136] For example, a user looks at a graphic M on an electronic whiteboard, says "move," and gestures to move graphic M from its current position A to position B. Based on the cross-modal data fusion and conflict recognition described above, the final intention to be executed is determined to be to move graphic M from position A to position B, and therefore a movement command for graphic M is generated.

[0137] Figure 3 These are schematic diagrams illustrating the business processing flow of some embodiments, applicable to collaborative interaction scenarios using electronic whiteboards. For example... Figure 3 As shown, the user interaction layer first outputs input signals in different modalities, such as voice, gestures, touch, and gaze. The multimodal signal acquisition module performs corresponding voice recognition, gesture recognition, and eye-tracking processing, then outputs input data for each modality. The spatiotemporal alignment and fusion engine performs spatiotemporal alignment processing on the input data for different modalities, including timestamp synchronization and coordinate mapping, and outputs spatiotemporally aligned interaction data for each modality. After data fusion of the interaction data for each modality, the dynamic conflict arbitration module performs modal priority allocation and semantic verification to resolve intent conflicts. Finally, the intent to be executed is determined, and the electronic whiteboard execution system executes the instructions corresponding to the intent, such as rendering, instruction response, and feedback.

[0138] Figure 4 These are flowcharts of some embodiments of the interactive command recognition method, such as... Figure 4 As shown, the interactive command recognition method includes the following steps:

[0139] Step S401: Collect the voice signal emitted by the user; for example, collect the voice signal of the user saying "move this".

[0140] Step S402: It is detected that the user is dragging a target.

[0141] Step S403: It is detected that the user's gaze is fixed on the target area.

[0142] Step S404: A slight press of the stylus on a target object is detected. This touch operation can serve as an auxiliary confirmation. The execution order of steps S401 to S404 is not fixed; they can be performed simultaneously or sequentially (any one step can be performed first).

[0143] In step S405, the speech recognition module converts the speech signal from step S401 into text data.

[0144] In step S406, the gesture recognition module detects the dragging action in step S402 and calculates the displacement vector of the drag.

[0145] In step S407, the gaze tracking module determines the user's gaze point from step S403 as the target object.

[0146] In step S408, the touch detection module records the touch operation in step S404 as the press time to enhance the confidence of the operation.

[0147] Step S409: Based on the input data generated in steps S405 to S408, perform spatiotemporal alignment to obtain the interaction data of each modality.

[0148] Step S410: Perform data fusion on the interaction data of each modality to obtain the target fused data.

[0149] Step S411: Perform conflict arbitration on the target fused data to determine the intended action. For details, please refer to the conflict resolution process in the above embodiment; it will not be repeated here.

[0150] Step S412: Execute and provide feedback on the intent to be executed.

[0151] Steps S401 to S412 above achieve dynamic multimodal fusion. Unlike existing technologies that process modalities independently and use simple rule concatenation, this embodiment can achieve natural hybrid interaction based on a weighting mechanism and multimodal cross-attention, allowing users to interact without consciously distinguishing input methods. Through semantic decoupling and conflict resolution, it can overcome the mechanical intent selection inherent in existing technologies, achieving a more human-centered compromise decision. By introducing spatiotemporal alignment, it can achieve asynchronous spatial compensation, avoiding the manipulation of commands. Therefore, through semantic feature extraction, semantic feature fusion, dynamic weight allocation, and cross-modal semantic complementarity, the same interactive system can meet the differentiated needs of various scenarios emphasizing different modalities, improving the accuracy of interactive command recognition in various scenarios.

[0152] Figure 5 Here is a flowchart of the semantic negotiation rules processing for some of the embodiments, such as Figure 5 As shown, the processing flow of this semantic negotiation rule may include: first, if an intent conflict is detected, it is determined whether the semantics corresponding to the conflicting intents can be merged; if so, a composite instruction is generated based on the merged intent; otherwise, the intent with higher modality priority is determined as the intent to be executed, and the instruction corresponding to the intent to be executed is executed.

[0153] Figure 6 Here are some conflict resolution flowcharts for some embodiments, such as Figure 6As shown, the input first includes target modal data containing different modalities. Based on a weighted voting mechanism, modal weights are dynamically assigned to each type of modal data. Then, it checks whether there are intent conflicts among the various user intents contained in the target modal data. If so, the intent conflicts are resolved according to preset conflict resolution rules for different scenarios; otherwise, the user intent is executed directly.

[0154] Figure 7 Here are some examples of semantic decoupling and semantic reorganization architecture diagrams, such as Figure 7 As shown, action semantics are extracted from speech features (speech recognition data), operation parameters are extracted from gesture features (gesture recognition data), and target localization is extracted from gaze features (gaze detection data). The above action semantic extraction results, operation parameter extraction results, and target localization extraction results are recombined using an instruction reassembly engine to obtain the intention to be executed and determine the corresponding operation instruction.

[0155] This embodiment also provides an interactive command recognition device, which is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. The terms "module," "unit," "subunit," etc., used below can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0156] Figure 8 This is a structural block diagram of the interactive command recognition device 80 in this embodiment, which is used for multimodal human-computer interaction; as Figure 8 As shown, the interactive instruction recognition device 80 includes: an acquisition module 82, a data fusion module 84, a conflict resolution module 86, and an instruction generation module 88; wherein: the acquisition module 82 is used to acquire the interaction data of each modality in the target scene; the data fusion module 84 is used to fuse the interaction data of each modality based on a multimodal cross-attention mechanism to obtain fused modal data, and to combine the fused modal data and the interaction data of each modality into target modal data; the conflict resolution module 86 is used to determine whether there is an intent conflict corresponding to the user intent of the target modal data, and if so, to determine the intent to be executed from different user intents based on the preset conflict resolution rules in the target scene; the instruction generation module 88 is used to generate an execution operation instruction according to the intent to be executed.

[0157] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0158] This embodiment provides an electronic whiteboard collaborative system. Figure 9 This is a schematic diagram of the electronic whiteboard system 90 in this embodiment, as shown below. Figure 9 As shown, the electronic whiteboard system 90 includes an electronic whiteboard 92, a signal processing module 94, and a server 96; wherein, the electronic whiteboard 92 and the signal processing module 94 are both connected to the server 96; the signal processing module 94 is used to receive input signals from users in different modalities, process the input signals, and send the processing results to the server 96; the server 96 is used to execute the interactive instruction recognition method provided in any of the above embodiments; the electronic whiteboard 92 is used to respond to the execution operation instructions generated by the server.

[0159] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0160] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0161] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0162] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0163] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0164] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. An interactive instruction recognition method, characterized by, The method comprises: acquiring each modality interaction data under a target scene; based on a multi-modal cross attention mechanism, data fusion is performed on the each modality interaction data to obtain fusion modality data, and the fusion modality data and the each modality interaction data are combined into target modality data; determining whether the user intent corresponding to the target modality data has intent conflict, if yes, determining the to-be-executed intent from different user intents based on a preset conflict resolution rule under the target scene; generating an execution operation instruction according to the to-be-executed intent.

2. The interactive instruction recognition method of claim 1, wherein, Acquiring each modality interaction data under a target scene comprises: acquiring input data generated by different signal processing modules under a target scene; wherein different signal processing modules receive and process input signals of different modalities to obtain the input data; performing time alignment processing and space alignment processing on each input data to obtain each modality interaction data after time and space alignment.

3. The interactive instruction recognition method of claim 2, wherein, Performing time alignment processing and space alignment processing on each input data to obtain each modality interaction data after time and space alignment comprises: determining a reference time point for time alignment according to the signal time stamp corresponding to each input data; time aligning each input data to the reference time point; converting the input data after time alignment to a preset reference coordinate system to obtain each modality interaction data after space alignment.

4. The interactive instruction recognition method of claim 1, wherein, Based on a multi-modal cross attention mechanism, data fusion is performed on the each modality interaction data to obtain fusion modality data, and the fusion modality data and the each modality interaction data are combined into target modality data, comprising: mapping each modality interaction data to a unified semantic dimension to obtain each modality interaction data after semantic dimension unification according to the constructed cross-modal embedding space; determining the modality contribution degree corresponding to each modality interaction data after semantic dimension unification based on a multi-modal cross attention mechanism; performing modality fusion on each modality interaction data after semantic dimension unification according to the modality contribution degree to obtain fusion modality data.

5. The interactive instruction recognition method of claim 1, wherein, Determining whether the user intent corresponding to the target modality data has intent conflict comprises: performing semantic structure separation identification on the semantic features associated with different modality data in the target modality data to obtain different modality semantic features corresponding to different semantic structures; determining each user intent corresponding to the target modality data by performing semantic reorganization on different modality semantic features; in the case that the target modality data contains multiple different user intents, determining whether there is intent conflict between different user intents.

6. The interactive instruction recognition method of claim 5, wherein, The target modality data comprises voice recognition data, gesture recognition data and gaze recognition data; performing semantic structure separation identification on the semantic features associated with different modality data in the target modality data to obtain different modality semantic features corresponding to different semantic structures comprises: performing action semantic extraction on the voice recognition data to obtain action semantic features in the user intent; performing operation parameter semantic extraction on the gesture recognition data to obtain operation parameter features in the user intent; The line-of-sight recognition data is subjected to target positioning semantic extraction to obtain a target positioning feature in the user intention.

7. The interactive instruction recognition method of claim 1, wherein, Based on a preset conflict resolution rule in the target scene, a to-be-executed intention is determined from different user intentions, including: According to the initial weight of each modality data in the target modality data, the scene coefficient of the target scene, and the confidence of each modality data in the target modality data, the modality weight of each modality data in the target modality data is determined; When the conflict resolution rule is a priority-based coverage rule, it is determined whether the confidence corresponding to the modality data with the highest modality weight in the target modality data is greater than a preset confidence threshold; if yes, the user intention corresponding to the modality data with the highest weight in the target modality data is determined as the to-be-executed intention; In the case where the conflict resolution rule is a semantic negotiation rule, it is determined whether each user intention satisfies a semantic merging condition; if yes, a composite instruction is generated according to each user intention; otherwise, the user intention corresponding to the modality data with the highest weight in the target modality data is determined as the to-be-executed intention; In the case where the conflict resolution rule is a secondary confirmation rule, a to-be-confirmed intention is output to the user according to each user intention, and the user-selected intention is taken as the to-be-executed intention in response to the user's selection operation on the to-be-confirmed intention. 8.The interactive instruction recognition method of claim 1, wherein, The target scene is an electronic whiteboard collaborative interaction scene; According to the to-be-executed intention, an execution operation instruction is generated, including: According to the to-be-executed intention, an operation instruction for an operation target on the electronic whiteboard is generated.

9. An interactive instruction recognition apparatus characterized by comprising: The interactive instruction recognition device is used for multi-modal human-computer interaction; the device includes an acquisition module, a data fusion module, a conflict resolution module, and an instruction generation module; wherein: The acquisition module is used to acquire each modality interaction data in a target scene. The data fusion module is used to perform data fusion on the each modality interaction data based on a multi-modal cross-attention mechanism to obtain fusion modality data, and the fusion modality data and the each modality interaction data are combined into target modality data. The conflict resolution module is used to determine whether there is an intention conflict in the user intention corresponding to the target modality data; if yes, a to-be-executed intention is determined from different user intentions based on a preset conflict resolution rule in the target scene. The instruction generation module is used to generate an execution operation instruction according to the to-be-executed intention.

10. An electronic whiteboard collaboration system, characterized by, It includes: An electronic whiteboard, a signal processing module, and a server; wherein the electronic whiteboard and the signal processing module are connected with the server; The signal processing module is used to receive user input signals of different modalities, process the input signals, and send the processing results to the server; The server is used to execute the interactive instruction recognition method of any one of claims 1 to 8; The electronic whiteboard is used to respond to the execution operation instruction generated by the server.

Images