A data processing method and device, electronic equipment and storage medium

Abstract

The application provides a data processing method and device, electronic equipment and storage medium; the method comprises: collecting multi-modal data corresponding to a target interaction space; the multi-modal data at least comprises image data, voice data and gesture data; the target interaction space represents a physical interaction space containing objects; based on the association relationship between the image data, the voice data and the gesture data, a pronoun in the voice data is bound to a target object in the target interaction space, a target operation intention for the target object is generated; the association relationship at least comprises a time association relationship and a space association relationship; based on the target operation intention and attribute information of the target object, a corresponding interaction strategy is selected from a preset strategy set, a target task is generated; the target task is executed, and the task execution result of the target task is output to a multi-modal output layer corresponding to the target interaction space through the electronic equipment, so as to respond to the target operation for the target object.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] Current multimodal large-scale models or graphical user interface interactions primarily involve input and output processing in a purely digital space, lacking the ability to perceive the physical interaction space. They cannot establish a mapping relationship between the direction of gestures in the physical interaction space and objects. Furthermore, multimodal data such as images, voice, and gestures typically undergo only simple data processing, resulting in a low level of intelligence. Therefore, current data processing methods suffer from poor physical space perception, insufficient multimodal fusion depth, and low scene adaptability. Summary of the Invention

[0003] This application provides a data processing method, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of this application, a data processing method is provided, the method comprising: collecting multimodal data corresponding to a target interaction space; the multimodal data including at least image data, voice data, and gesture data; the target interaction space representing a physical interaction space containing objects; based on the correlation between image data, voice data, and gesture data, binding pronouns in the voice data to target objects in the target interaction space to generate a target operation intent for the target objects; the correlation including at least temporal correlation and spatial correlation; based on the target operation intent and attribute information of the target objects, selecting a corresponding interaction strategy from a preset strategy set to generate a target task; executing the target task, and outputting the task execution result of the target task to a multimodal output layer corresponding to the target interaction space via an electronic device to respond to the target operation for the target objects.

[0005] According to one embodiment of this application, the step of binding pronouns in the voice data to target objects in the target interaction space based on the correlation between image data, voice data, and gesture data, and generating a target operation intention for the target object, includes: determining the position information of each object in the target interaction space based on the image data; determining the target object in the target interaction space based on the spatial correlation between the spatial coordinates corresponding to the gesture data and the position information; and determining the target operation intention for the target object based on the temporal correlation between the voice data and the gesture data.

[0006] According to one embodiment of this application, determining the target operation intention for the target object based on the temporal correlation between the voice data and the gesture data includes: obtaining a gesture completion timestamp corresponding to each gesture data, an operation intention corresponding to each voice data, and a voice reception timestamp; determining the temporal correlation based on the voice reception timestamp and the gesture completion timestamp; in response to the temporal correlation satisfying a set correlation condition, determining the pronouns included in the voice data, and binding the pronouns to the target object to obtain a binding relationship; verifying the semantic matching between the operation intention and the target object based on the attribute information of the target object; and in response to the semantic matching satisfying a set matching condition, replacing the corresponding pronoun with the unique identifier of the target object to generate the target operation intention for the target object.

[0007] According to one embodiment of this application, determining the time correlation based on the voice reception timestamp and the gesture completion timestamp includes: determining the time difference between the voice reception timestamp and the gesture completion timestamp; determining the spatial position information of the hand within a first region of the target object after the gesture is completed; in response to the spatial position information indicating that the hand is continuously located within the first region, determining an effective time window as a first duration, otherwise determining the effective time window as a second duration, wherein the first duration is greater than the second duration; in response to the time difference being less than or equal to the effective time window, determining that the time correlation satisfies the set correlation condition.

[0008] According to one embodiment of this application, determining the target object within the target interaction space based on the spatial correlation between the spatial coordinates corresponding to the gesture data and the location information includes: determining a sequence of hand keypoint coordinates based on the spatial coordinates corresponding to the gesture data; determining a gesture movement trajectory based on the hand keypoint coordinate sequence; determining the bounding box coordinates of each object within the target interaction space as the location information; determining objects with a spatial correlation between the gesture movement trajectory and the location information as candidate objects; determining the priority of each candidate object, and determining the candidate object with the highest priority as the target object; the priority is associated with the attribute information of the candidate object and / or the voice data.

[0009] According to one embodiment of this application, the step of selecting a corresponding interaction strategy from a preset strategy set based on the target operation intention and the attribute information of the target object to generate a target task includes: determining task content based on the action type corresponding to the gesture data, the target operation intention, and the attribute information of the target object; determining a task execution scenario based on the task content and the type of the target object; selecting a corresponding interaction strategy from a set of strategies based on the task execution scenario; the interaction strategy includes at least one of information retrieval processing, text processing, image processing, and content generation processing; and generating the target task based on the interaction strategy and the task content.

[0010] According to one embodiment of this application, the step of outputting the task execution result of the target task to the multimodal output layer corresponding to the target interaction space via an electronic device includes: receiving the task execution result corresponding to the target task; determining at least one output method based on the task execution result; the output method includes at least one of displaying the target task in a window form on the display interface of the electronic device, outputting text on the display interface, and outputting audio via the audio device of the electronic device; and outputting the task execution result to the multimodal output layer via the electronic device based on the determined output method; the multimodal output layer includes at least one of outputting an image on the display interface and outputting sound via the audio device.

[0011] According to a second aspect of this application, a data processing apparatus is provided, the apparatus comprising: a data acquisition module for acquiring multimodal data corresponding to a target interaction space; the multimodal data including at least image data, voice data, and gesture data; the target interaction space representing a physical interaction space containing objects; a determination module for binding pronouns in the voice data to target objects in the target interaction space based on the correlation between the image data, voice data, and gesture data, thereby generating a target operation intent for the target object; the correlation including at least a temporal correlation and a spatial correlation; a generation module for selecting a corresponding interaction strategy from a preset strategy set based on the target operation intent and the attribute information of the target object, thereby generating a target task; and an execution module for executing the target task and outputting the task execution result of the target task to a multimodal output layer corresponding to the target interaction space via an electronic device to respond to the target operation for the target object.

[0012] According to a third aspect of this application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: acquiring multimodal data corresponding to a target interaction space; the multimodal data including at least image data, voice data, and gesture data; the target interaction space representing a physical interaction space containing objects; based on the correlation between image data, voice data, and gesture data, binding pronouns in the voice data to target objects in the target interaction space to generate a target operation intent for the target objects; the correlation including at least temporal correlation and spatial correlation; based on the target operation intent and attribute information of the target objects, selecting a corresponding interaction strategy from a preset strategy set to generate a target task; executing the target task, and outputting the task execution result of the target task to a multimodal output layer corresponding to the target interaction space through the electronic device to respond to the target operation for the target objects.

[0013] According to a fourth aspect of this application, a non-transitory computer-readable storage medium storing computer instructions is provided, the computer instructions being used to cause a computer to perform: acquiring multimodal data corresponding to a target interaction space; the multimodal data including at least image data, voice data, and gesture data; the target interaction space representing a physical interaction space containing objects; based on the correlation between image data, voice data, and gesture data, binding pronouns in the voice data to target objects in the target interaction space to generate a target operation intent for the target object; the correlation includes at least a temporal correlation and a spatial correlation; based on the target operation intent and the attribute information of the target object, selecting a corresponding interaction strategy from a preset strategy set to generate a target task; executing the target task, and outputting the task execution result of the target task to the multimodal output layer corresponding to the target interaction space through an electronic device to respond to the target operation for the target object. Attached Figure Description

[0014] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0015] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0016] Figure 1 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 1; Figure 2 This application illustrates a schematic diagram of the data processing method provided in an embodiment. Figure 2 ; Figure 3 This illustration shows an application scenario diagram of the data processing method provided in an embodiment of this application; Figure 4 This illustration shows another application scenario of the data processing method provided in the embodiments of this application; Figure 5 An optional schematic diagram of the data processing apparatus provided in an embodiment of this application is shown; Figure 6 A schematic diagram of the composition structure of the electronic device provided in the embodiments of this application is shown. Detailed Implementation

[0017] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0018] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0019] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0022] The processing flow of the data processing method provided in the embodiments of this application will be described. See [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the data processing method provided in the embodiments of this application. Figure 1 , will combine Figure 1 Steps S101-S104 are explained below.

[0023] Step S101: Collect multimodal data corresponding to the target interaction space; the multimodal data includes at least image data, voice data and gesture data; the target interaction space represents the physical interaction space containing objects.

[0024] In some embodiments, the target interaction space may include a physical interaction space containing physical objects. Multimodal data such as gestures, voice, and images can be collected through the target interaction space. Multimodal data may include: image data acquired in real-time by an RGB (Red, Green, Blue) camera, voice data acquired by a microphone, and gesture data extracted by visual algorithms. Image data may include: continuous video frame sequences, single static captured images, and text content extracted by OCR (Optical Character Recognition) algorithms. This application embodiment does not limit the specific image data. Voice data may include: Natural Language Processing (NLP) audio emitted by the user. This application embodiment does not limit the specific voice data. Gesture data may include: hand keypoint coordinate sequences, extended line vectors of finger pointing, hand movement trajectory data, and gesture completion timestamps. This application embodiment does not limit the specific gesture data. The physical interaction space may be a desktop area where objects are placed.

[0025] Step S102: Based on the relationship between image data, voice data and gesture data, bind the pronouns in the voice data to the target objects in the target interaction space to generate the target operation intention for the target objects; the relationship includes at least temporal relationship and spatial relationship.

[0026] In some embodiments, the association relationships may include temporal and spatial association relationships. Temporal association relationships may include: the time difference between the gesture completion timestamp and the voice reception timestamp, a preset effective time window (e.g., 2-3 seconds), and the duration of hand lingering near the target object. Spatial association relationships may include: the mapping relationship between the spatial coordinates of the gesture data and the object bounding box coordinates, the intersection relationship between the extended line of the finger pointing and the object bounding box, and the intersection relationship between the gesture trajectory and the object contour. Referential terms may include words such as "this," "that," "it," and "this passage." This application embodiment does not limit specific referential terms. The target object may include: an object in the target interaction space determined by the user pointing to or selecting it with a gesture. The target operation intent can be used for the generation and execution of the target task; the target operation intent represents the operation that the system needs to perform on the target object. Examples include specific operations such as translation, search, and summarization.

[0027] Step S103: Based on the target operation intention and the attribute information of the target object, select the corresponding interaction strategy from the preset strategy set to generate the target task.

[0028] In some embodiments, attribute information may include: object category (e.g., books, mobile phones, paper), text features (e.g., text content, language type, character density), image features (e.g., color, texture, size), or spatial location. This application embodiment does not limit specific attribute information. The strategy set may include: information search strategies, text translation strategies, image enhancement strategies, content generation strategies, and interactive input strategies. This application embodiment does not limit specific interaction strategies. Interaction strategies can transform abstract target operation intentions into executable specific operation steps.

[0029] Step S104: Execute the target task and output the task execution result of the target task to the multimodal output layer corresponding to the target interaction space through the electronic device to respond to the target operation for the target object.

[0030] In some embodiments, the target task can be used to: call the corresponding system API (Application Programming Interface), cloud service, or local application to complete the operation. The task execution result may include: processed text content, generated image data, and retrieved information entries, etc. Different target tasks correspond to different task execution results; this application embodiment does not limit specific target tasks or task execution results. The multimodal output layer corresponding to the target interaction space may include at least one of: image output on the display interface and sound output through an audio device. This application embodiment does not limit specific output methods.

[0031] As an example, a paper book printed in English is placed flat on a table. The system continuously captures image data of the tabletop using an RGB camera, identifies the bounding box coordinates of the paper book using the YOLO object detection algorithm, determines a unique identifier, and establishes an index corresponding to the paper book. The user then uses their finger to circle a specific paragraph on a page within the target interaction space corresponding to the RGB camera. The system extracts the hand keypoint coordinate sequence using the MediaPipe algorithm (a hand keypoint detection model) and calculates the gesture trajectory. Based on the gesture trajectory and the identified object bounding box, the system identifies the paper book as the target object and marks its state as active, while recording the gesture completion timestamp. The user then issues a voice command, "Translate this content," and the system captures the voice command through a microphone and uses ASR (Automatic Speech) to process it. The speech recognition engine converts speech into text, where the operation intent includes "translate" and the referent includes "this". The system records the speech reception timestamp and, based on the speech reception timestamp and gesture completion timestamp, determines the temporal correlation to meet the association conditions. The referent "this" is then bound to the activated target object. Based on the operation intent "translate" and the target object attribute "English text", a translation operation intent containing a unique identifier for the target object is generated. The system selects an interaction strategy combining OCR recognition and machine translation from the strategy set based on the gesture selection action type, translation operation intent, and book text attributes. This generates the target task, calls the local OCR service interface to extract the English text from the selected area, and then calls the cloud translation API to convert the English to Chinese. The resulting translated Chinese text is used as the task execution result. A semi-transparent Chinese translation text box is overlaid on the corresponding selected area in the real-time video stream through the electronic device's application interface, while simultaneously outputting a "translation complete" voice prompt through the audio device.

[0032] The method in this application establishes a spatiotemporal correlation between image data, voice data, and gesture data, binding voice pronouns to target objects in a physical scene. This enables the understanding of the user's gestures in physical space, achieving collaborative understanding of the three modalities (image, voice, and gesture) at the semantic level and improving the depth of multimodal fusion. By dynamically selecting interaction strategies from a preset set based on the target operation intent and target object attribute information, scene adaptability is enhanced. By executing the target task and outputting the results to the multimodal output layer corresponding to the target interaction space, a complete task loop is formed, from physical object information acquisition and digital processing to physical feedback of results.

[0033] In some embodiments, the processing flow of the data processing method is illustrated. Figure 2 ,like Figure 2As shown, step S102, which involves binding pronouns in the voice data to target objects in the target interaction space based on the correlation between image data, voice data, and gesture data, and generating target operation intentions for the target objects, may include: Step S201: Based on the image data, determine the position information of each object in the target interaction space.

[0034] As an example, image data of the target interaction space is continuously acquired using an RGB camera. This image data could be a book and a mobile phone placed on a desktop scene. The YOLO object detection algorithm is used to identify the book and mobile phone on the desktop, generating corresponding bounding box coordinates for each. These bounding box coordinates are then used as location information, and a spatial index is established for each item.

[0035] Step S202: Based on the spatial correlation between the spatial coordinates and position information corresponding to the gesture data, determine the target object within the target interaction space.

[0036] In this embodiment, spatial coordinates can represent the user's specific pointing position in the target interaction space. Spatial correlation can be used to map gesture actions in the target interaction space to selection operations on target objects. Temporal correlation can be used to determine whether voice commands and gesture operations are temporally related.

[0037] In some embodiments, step S202 may include: determining a sequence of hand key point coordinates based on the spatial coordinates corresponding to the gesture data; determining the gesture motion trajectory based on the sequence of hand key point coordinates; determining the bounding box coordinates of each object in the target interaction space as position information; determining objects with a spatial relationship between the gesture motion trajectory and the position information as candidate objects; determining the priority of each candidate object, and determining the candidate object with the highest priority as the target object; the priority is associated with the attribute information and / or voice data of the candidate object.

[0038] In this embodiment, the hand keypoint coordinate sequence may include a set of coordinate positions reflecting hand movement extracted by the MediaPipe algorithm. The gesture trajectory can be obtained by fitting the hand keypoint coordinate sequence. The gesture trajectory represents the user's specific gesture operation in physical space. The bounding box coordinates can represent the spatial position of an object in the target interaction space. Position information can be used to establish a spatial index of the object in the system. Candidate objects may include multiple objects that have a spatial intersection with the gesture trajectory. Priorities may include: priorities determined based on the candidate object's visual confidence score, priorities determined based on the object's size, priorities determined based on the distance between the candidate object and the gesture center point, and priorities determined after adjustment using clarification instructions from the voice data. Priorities can be used to further filter candidate objects to obtain the target object when there is ambiguity in the gesture direction or multiple objects overlap.

[0039] As an example, the user makes a selection gesture within the target interaction space. A closed gesture trajectory is generated based on a continuous sequence of hand keypoint coordinates. Simultaneously, the YOLO object detection algorithm identifies two books (book A and book B) and a mobile phone placed on the table within the target interaction space. The bounding box coordinates of the three books and the mobile phone are determined and used as positional information. The spatial relationship between the gesture trajectory and the bounding boxes of each object is calculated, determining the intersection-union ratio (IU) of the trajectory with the bounding box of book A to be 0.8, with the bounding box of book B to be 0.3, and with the bounding box of the mobile phone to be 0. Book A and book B are identified as candidate objects. All candidate objects are previewed with half-highlighting to prompt the user for confirmation. Then, the priority of the two candidate objects is determined, including at least one of the following: The visual confidence score of book A is 0.9, and the visual confidence score of book B is 0.5. Book A, which has a higher visual confidence score, is given the highest priority and is identified as the target object.

[0040] Book A has a size of 10 square centimeters, and Book B has a size of 6 square centimeters. Book A, which has a larger size, is given the highest priority and is designated as the target object.

[0041] Upon receiving the user's clarification instruction "Translate that book", the system extracts the qualifier "Translate that book" from the voice data and determines from the context that the user is referring to book B. Book B is then given the highest priority and identified as the target object.

[0042] Step S203: Based on the temporal correlation between voice data and gesture data, determine the target operation intention for the target object.

[0043] As an example, image data of the target interaction space is continuously acquired using an RGB camera. This image data could be a table scene with a book and a mobile phone placed on it. The YOLO object detection algorithm is used to identify the book and the mobile phone on the table, generating corresponding bounding box coordinates for each. These bounding box coordinates are then used as the location information.

[0044] The MediaPipe algorithm is used to extract the key point coordinate sequence of the hand, calculate the coordinates of the index fingertip and generate the corresponding pointing extension vector. The intersection of the pointing extension vector with the bounding box coordinates is determined. The drawback is that it intersects with the bounding box coordinates of the book but not with the bounding box coordinates of the phone. The book is identified as the target object and marked as active.

[0045] Upon receiving the user's voice command "Summarize the contents of this book," the system determines the gesture completion timestamp as T1 and the voice reception timestamp as T2. The time difference between T1 and T2 is calculated to be 1.2 seconds, which is less than the preset 2-second effective time window. The system extracts "summarize" as the operation intent and "this" as the referent from the voice text, binding the referent to the activated target object to generate a target operation intent for the book. This target operation intent represents summarizing the entire contents of the book.

[0046] In some embodiments, step S203 may include: obtaining the gesture completion timestamp corresponding to each gesture data, the operation intent corresponding to each voice data, and the voice reception timestamp; determining the temporal association relationship based on the voice reception timestamp and the gesture completion timestamp; in response to the temporal association relationship satisfying the set association conditions, determining the pronouns included in the voice data, and binding the pronouns to the target object to obtain the binding relationship; verifying the semantic matching between the operation intent and the target object based on the attribute information of the operation intent and the target object; in response to the semantic matching satisfying the set matching conditions, replacing the corresponding pronoun with the unique identifier of the corresponding target object to generate the target operation intent for the target object.

[0047] In this embodiment, the gesture completion timestamp may include: the time when the motion trajectory terminates as determined by the gesture recognition algorithm. The voice reception timestamp may include: the time when the first valid word in the user's voice command is detected. The binding relationship may include: a mapping relationship established between pronouns in the voice data and unique identifiers of the target object. The operation intent may characterize the specific type and content of the operation that the user needs to perform on the target object. Semantic matching and matching conditions may be used to: determine whether the operation requested by the user matches the attributes of the target object. The matching conditions may include: a preset semantic matching degree threshold. The unique identifier may include: a unique ID (Identifier) ​​assigned by the system to each identified physical object.

[0048] As an example, the moment the user completes the gesture of selecting a book with their finger is recorded as the gesture completion timestamp T1, and the moment the microphone receives the user's voice "Please translate this book for me" is recorded as the voice reception timestamp T2. The time difference between T1 and T2 is calculated to be 0.5 seconds. The system detects that the hand is quickly removed after the gesture is completed, and the corresponding effective time window is 1 second. Since 0.5 seconds is less than 1 second, the association condition is met, and the time association relationship is determined to meet the set association condition. The pronoun "this book" is parsed from the voice data, the operation intent is "translate", and "this book" is bound to the activated target object (book A). The semantic matching score between the operation intent "translate" and the attribute information of the target object (object category is book, content language is English) is determined to be 0.95. The semantic matching score of 0.95 is greater than the set semantic matching score threshold of 0.7. The pronoun "this book" is replaced with the unique identifier of the target object "book A", and a target operation intent for book A is generated.

[0049] In some embodiments, determining a temporal correlation based on a voice reception timestamp and a gesture completion timestamp may include: determining a time difference between the voice reception timestamp and the gesture completion timestamp; determining spatial position information of the hand within a first region of the target object after the gesture is completed; determining a first duration for an effective time window in response to the spatial position information indicating that the hand is continuously located within the first region, otherwise determining a second duration for the effective time window, wherein the first duration is greater than the second duration; and determining that the temporal correlation satisfies a set correlation condition in response to the time difference being less than or equal to the effective time window.

[0050] In this embodiment, the spatial location information within the first region can be used to determine whether the hand remains near the target object after the gesture is completed. The effective time window may include the maximum time interval allowed for establishing a correlation between voice data and gesture data. The first duration may include a longer effective time window (e.g., 3 seconds) used when the hand remains continuously within the first region of the target object after the gesture is completed. The second duration may include a shorter effective time window (e.g., 1 second) used when the hand immediately leaves the target object after the gesture is completed.

[0051] As an example, the moment when the user completes the action of circling the book with their finger is determined as the gesture completion timestamp T1, and the start time when the microphone receives the voice is determined as the voice reception timestamp T2. The time difference between T1 and T2 is calculated to be 0.4 seconds. The system detects that the hand is quickly removed after the gesture is completed, and the effective time window is determined to be the second duration of 1 second. The time difference of 0.4 seconds is determined to be within the second duration of 1 second. Therefore, the time correlation is determined to meet the set correlation conditions.

[0052] As an example, the moment the user completes the gesture of circling the book is determined as the gesture completion timestamp T1, and the start time of the microphone receiving the voice is determined as the voice reception timestamp T2. The time difference between T1 and T2 is calculated to be 2.2 seconds. The system detects that the hand remains in the area above the book after the gesture is completed, i.e., within the first area. The effective time window is set to a first duration of 3 seconds. Since the time difference of 2.2 seconds is within the first 3-second duration, the time correlation is determined to meet the set correlation conditions.

[0053] As an example, the moment when the user completes the action of pointing at the book with their finger is determined as the gesture completion timestamp T1, and the start time when the microphone receives the voice is determined as the voice reception timestamp T2. The time difference between T1 and T2 is calculated to be 2.5 seconds. The system detects that the hand quickly leaves the first area after the gesture is completed, and determines the effective time window as the second duration of 1 second. It is determined that the time difference of 2.5 seconds is greater than the second duration of 1 second, so the time correlation does not meet the set correlation conditions.

[0054] As an example, the moment when the user completes the action of circling the document paragraph with their finger is determined as the gesture completion timestamp T1, and the moment when the microphone receives the long sentence voice command is determined as the voice reception timestamp T2. The time difference between T1 and T2 is calculated to be 4.2 seconds. The system detects that the hand remains in the first area after the gesture is completed and detects that there are complex instructions in the long sentence voice command. The effective time window is determined to be an extended duration of 5 seconds. The time difference of 4.2 seconds is determined to be within the extended duration of 5 seconds. Therefore, the time correlation is determined to meet the set correlation conditions.

[0055] The method in this application determines the position information of each object in the target interaction space based on image data, and determines the target object based on the spatial correlation between the spatial coordinates corresponding to the gesture data and the position information. At the same time, it determines the target operation intention based on the temporal correlation between the voice data and the gesture data. This achieves deep semantic binding and consistency verification between voice pronouns and physical entities, realizes collaborative understanding of the three modalities of image, voice and gesture at the semantic level, improves the depth of multimodal fusion, realizes intelligent disambiguation of multiple candidate objects in the physical interaction space, can intelligently process data, improves the perception capability of physical space, enhances the depth of multimodal fusion and improves the scene adaptation capability.

[0056] In some embodiments, step S103 may include: determining task content based on the action type corresponding to the gesture data, the target operation intention, and the attribute information of the target object; determining the task execution scenario based on the task content and the type of the target object; selecting a corresponding interaction strategy from a set of strategies based on the task execution scenario; and generating a target task based on the interaction strategy and the task content.

[0057] In this embodiment, the interaction strategy includes at least one of information retrieval processing, text processing, image processing, and content generation processing. The action types corresponding to gesture data may include: finger selection, pointing, drawing, hovering, and clicking. This embodiment does not limit the specific action types. The task content may include: a specific task description formed by combining the action type, the target operation intention, and the attribute information of the target object (e.g., translating English books, searching mobile phone information, enhancing blurred images, generating summaries, etc.). The type of target object may include: object categories classified based on attribute information (e.g., text objects, image objects, electronic device objects). The task execution scenario may include: a specific application scenario identified according to the task content and object type (e.g., information retrieval scenario, foreign language document translation scenario, image enhancement scenario, content generation scenario). The strategy set may include: a set of preset interaction strategies. Each interaction strategy corresponds to at least one task execution scenario.

[0058] As an example, the gesture data is identified as a "selection action," the target operation intent is "summarize," and the target object's attribute information includes the object category as a book, the content language as English, and the text type as a technical document. The corresponding task content is determined as "summarizing the content of an English technical document." Based on the determined task content and the type of the target object, the task execution scenario is determined as a "foreign language document summarization scenario." Based on the task execution scenario, the corresponding interaction strategy is determined from the strategy set as an interaction strategy chain of "OCR recognition → AI summarization generation → screen output." The target task is generated based on the determined interaction strategy and task content. The target task could be that the system automatically takes a picture of the selected area, calls the OCR service to extract the English text, sends it to the summarization model to generate a Chinese summary, and finally outputs the result to the screen in the form of a floating card.

[0059] The method in this application determines the task content and task execution scenario based on the action type corresponding to the gesture data, the target operation intention, and the attribute information of the target object. It then dynamically selects the corresponding interaction strategy from a preset strategy set based on the task execution scenario. This enables dynamic adjustment of the processing strategy according to the specific task type and scenario characteristics, optimizes the execution effect, and improves the scene adaptability.

[0060] In some embodiments, step S104, which involves outputting the task execution result of the target task to the multimodal output layer corresponding to the target interaction space via an electronic device, may include: receiving the task execution result corresponding to the target task; determining at least one output method based on the task execution result; and outputting the task execution result to the multimodal output layer via an electronic device based on the determined output method.

[0061] In this embodiment, the output method includes at least one of the following: displaying the image as a window overlay on the display interface of the electronic device, outputting text in the display interface, and outputting audio via an audio device of the electronic device. The multimodal output layer may include at least one of image output on the display interface and sound output via an audio device. The electronic device may include: VR (Virtual Reality) glasses, AR (Augmented Reality) glasses, smart TVs, and laptops, etc. This application embodiment does not limit the specific electronic device. The output method can be determined based on the content type of the task execution result and user needs. The display interface may include: the screen display area of ​​the electronic device, the graphical interface of the application, and a preview window capable of displaying the camera's captured image in real time. This application embodiment does not limit the specific display interface. Window overlay display may include: a semi-transparent window floating on the real-time video stream. The audio device may include: a speaker built into the electronic device, Bluetooth headphones, and external audio devices, etc. This application embodiment does not limit the specific audio device.

[0062] As an example, the system receives the task execution result for translating an English book. The result includes the translated Chinese text. Based on the content type of the result and preset output rules, the system determines two output methods: "window overlay display" and "audio output." The system overlays a semi-transparent translation text box onto the corresponding page in the real-time video stream on the AR glasses' display interface. Simultaneously, it controls the AR glasses' audio device to output a "translation complete" voice prompt. Thus, the task execution result is fed back to the user through a multimodal output layer.

[0063] The method in this application embodiment receives the task execution result corresponding to the target task, and determines the output method, including window overlay display, text output or audio output, based on the result. The task execution result is output to the multimodal output layer corresponding to the target interactive space through an electronic device, realizing multi-channel feedback of digital processing results to physical scene, improving the task execution capability across physical and digital boundaries, and supporting a complete task closed loop from physical object information collection, digital processing to result output.

[0064] refer to Figure 3 This application provides an example of an application scenario diagram of a data processing method, which is applied to the architecture of a multimodal intelligent desktop scene interaction system.

[0065] The perception input layer includes: image modality, which extracts object and text information through YOLO object detection and OCR text recognition using an RGB camera; gesture modality, which obtains gesture data through MediaPipe hand keypoint detection and gesture classification and spatial coordinate mapping using an RGB camera; and speech modality, which extracts operation intent through ASR speech recognition and NLP intent understanding using a microphone array.

[0066] The multimodal fusion and interaction engine includes a context alignment and reference resolution module and a semantic completion and task generation module.

[0067] The feedback and output layer drives three outputs through a task executor: calling system APIs to perform search / translation / save operations, broadcasting results through TTS (Text-to-Speech) speech synthesis, and updating AR overlay and pop-up display through a graphical interface.

[0068] The perception input layer continuously acquires desktop scene images through the visual modality to detect objects in real time and build a scene context index containing unique identifiers. It captures the spatial coordinates of the user's pointing or circling actions through the gesture modality to activate specific objects and provide visual feedback. At the same time, it receives natural language instructions through the voice modality to extract operation intentions, pronouns, and timestamps, thereby realizing the recognition of entity states, user actions, and voice semantics within the physical interaction space.

[0069] The multimodal fusion and interaction engine first verifies the temporal correlation between voice data and gesture data to bind pronouns in the voice to objects in the active state and performs semantic consistency verification. Then, based on user intent, object attributes and gesture action type, it identifies the current interaction scenario (such as foreign language document processing or physical object retrieval) and selects a targeted interaction strategy (such as translation, enhancement or search) from a preset strategy set to generate the target task.

[0070] The feedback and output layer receives the target tasks generated by the engine through the task executor and calls the corresponding system APIs, cloud services or local application interfaces to execute the tasks (such as information retrieval or content generation), obtains the task execution results, and displays the task execution results overlaid in the real-time video stream through a graphical interface and broadcasts feedback information through speech synthesis.

[0071] refer to Figure 4 This application provides another application scenario diagram of the data processing method, which is applied to a multimodal intelligent desktop scene interaction method.

[0072] The system continuously captures desktop scene image data containing the English documents "Text 1" and "Text 2" via a camera, while simultaneously listening to the user's voice via a microphone. When the user selects "Text 1" with their finger, the system determines the corresponding gesture trajectory based on the key hand coordinate sequence, performs spatial correlation calculations between the gesture trajectory and the document bounding box, identifies "Text 1" as the target object, records the gesture completion timestamp T1, determines that "Text 1" is active, and displays a highlighted border around "Text 1".

[0073] The user issues the voice command "Translate this". The system records the voice reception timestamp T2, calculates the time difference between T1 and T2 to be 1.8 seconds, and detects that the hand remains above text 1 after the gesture is completed. The effective time window is determined to be 3 seconds. Since the time difference of 1.8 seconds is less than the effective time window of 3 seconds, the time correlation condition is met. The system extracts the pronoun "this" from the voice and establishes a binding relationship with "text 1", generating the target operation intent of "Translate text 1". Based on the target object's attribute information and the target intent, the system determines the task execution scenario to be a foreign language document translation scenario. It then determines the interaction strategy chain "OCR recognition → Machine translation → Overlay display" from the strategy set and generates the corresponding target task.

[0074] When executing the target task, the system first calls the OCR service to extract the English content of the selected area, then calls the cloud-based translation API to convert text 1 into Chinese, and obtains the translated text of text 1 as the task execution result; finally, the system outputs the result to the multimodal output layer through electronic devices, and displays the translated text of text 1 as a floating card in the real-time video stream of the display interface below the area corresponding to text 1.

[0075] The exemplary structure of the data processing apparatus 90 provided in the embodiments of this application will be further described below. In some embodiments, such as... Figure 5 As shown, the data processing device 90 may include: a data acquisition module 901, used to acquire multimodal data corresponding to the target interaction space; the multimodal data includes at least image data, voice data, and gesture data; the target interaction space represents the physical interaction space containing objects; The determination module 902 is used to bind pronouns in the voice data to target objects in the target interaction space based on the correlation between image data, voice data and gesture data, and generate target operation intentions for the target objects; the correlation includes at least temporal correlation and spatial correlation; The generation module 903 is used to select the corresponding interaction strategy from a preset strategy set based on the target operation intention and the attribute information of the target object to generate the target task; The execution module 904 is used to execute the target task and output the task execution result to the multimodal output layer corresponding to the target interaction space through an electronic device to respond to the target operation on the target object.

[0076] In some embodiments, the determining module 902 is configured to: determine the position information of each object in the target interaction space based on image data; determine the target object in the target interaction space based on the spatial correlation between the spatial coordinates corresponding to the gesture data and the position information; and determine the target operation intention for the target object based on the temporal correlation between the voice data and the gesture data.

[0077] In some embodiments, the determining module 902 is configured to: acquire the gesture completion timestamp corresponding to each gesture data, the operation intent corresponding to each voice data, and the voice reception timestamp; determine the temporal association based on the voice reception timestamp and the gesture completion timestamp; in response to the temporal association satisfying the set association conditions, determine the pronouns included in the voice data, and bind the pronouns to the target object to obtain the binding relationship; verify the semantic matching between the operation intent and the target object based on the operation intent and the attribute information of the target object; in response to the semantic matching satisfying the set matching conditions, replace the corresponding pronoun with the unique identifier of the corresponding target object to generate the target operation intent for the target object.

[0078] In some embodiments, the determining module 902 is configured to: determine the time difference between the voice reception timestamp and the gesture completion timestamp; determine the spatial position information of the hand within a first region of the target object after the gesture is completed; in response to the spatial position information indicating that the hand is continuously located within the first region, determine the effective time window as a first duration, otherwise determine the effective time window as a second duration, wherein the first duration is greater than the second duration; and in response to the time difference being less than or equal to the effective time window, determine that the time correlation satisfies the set correlation conditions.

[0079] In some embodiments, the determining module 902 is configured to: determine a sequence of hand key point coordinates based on the spatial coordinates corresponding to the gesture data; determine the gesture motion trajectory based on the sequence of hand key point coordinates; determine the bounding box coordinates of each object in the target interaction space as position information; determine objects with a spatial relationship between the gesture motion trajectory and the position information as candidate objects; determine the priority of each candidate object, and determine the candidate object with the highest priority as the target object; the priority is associated with the attribute information and / or voice data of the candidate object.

[0080] In some embodiments, the generation module 903 is configured to: determine task content based on the action type corresponding to the gesture data, the target operation intention, and the attribute information of the target object; determine the task execution scenario based on the task content and the type of the target object; select a corresponding interaction strategy from a set of strategies based on the task execution scenario; the interaction strategy includes at least one of information retrieval processing, text processing, image processing, and content generation processing; and generate a target task based on the interaction strategy and the task content.

[0081] In some embodiments, the execution module 904 is configured to: receive the task execution result corresponding to the target task; determine at least one output method based on the task execution result; the output method includes at least one of displaying the result in a window overlay on the display interface of the electronic device, outputting text in the display interface, and outputting audio through the audio device of the electronic device; and outputting the task execution result to the multimodal output layer through the electronic device based on the determined output method; the multimodal output layer includes at least one of outputting an image on the display interface and outputting sound through the audio device.

[0082] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For any technical details not covered in the data processing apparatus provided in this application embodiment, please refer to... Figures 1 to 4 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0083] According to embodiments of this application, this application also provides an electronic device and a non-transitory computer-readable storage medium.

[0084] Figure 6 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0085] like Figure 6As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0086] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0087] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data processing methods. For example, in some embodiments, the data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0088] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0089] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0090] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0091] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0092] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0093] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0094] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0095] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0096] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, the method comprising: Collect multimodal data corresponding to the target interaction space; The multimodal data includes at least image data, voice data, and gesture data; The target interaction space represents the physical interaction space containing objects; Based on the correlation between image data, voice data and gesture data, pronouns in the voice data are bound to target objects in the target interaction space to generate target operation intentions for the target objects. The aforementioned relationships include at least temporal relationships and spatial relationships; Based on the target operation intention and the attribute information of the target object, a corresponding interaction strategy is selected from a preset strategy set to generate the target task; The target task is executed, and the task execution result is output to the multimodal output layer corresponding to the target interaction space via an electronic device to respond to the target operation for the target object.

2. The method according to claim 1, wherein the step of binding pronouns in the voice data to target objects in the target interaction space based on the correlation between image data, voice data, and gesture data, and generating target operation intentions for the target objects, includes: Based on the image data, determine the position information of each object within the target interaction space; Based on the spatial correlation between the spatial coordinates corresponding to the gesture data and the location information, the target object within the target interaction space is determined. Based on the temporal correlation between the voice data and the gesture data, the target operation intention for the target object is determined.

3. The method according to claim 2, wherein determining the target operation intention for the target object based on the temporal correlation between the voice data and the gesture data includes: Obtain the gesture completion timestamp corresponding to each gesture data, the operation intent corresponding to each voice data, and the voice reception timestamp; The time correlation is determined based on the voice reception timestamp and the gesture completion timestamp; In response to the time association relationship satisfying the set association conditions, the pronouns included in the voice data are determined, and the pronouns are bound to the target object to obtain the binding relationship; Based on the operation intention and the attribute information of the target object, verify the semantic matching between the operation intention and the target object; In response to the semantic matching satisfying the set matching conditions, the corresponding pronoun is replaced with the unique identifier of the target object, and a target operation intent for the target object is generated.

4. The method according to claim 3, wherein determining the time correlation based on the voice reception timestamp and the gesture completion timestamp includes: Determine the time difference between the voice reception timestamp and the gesture completion timestamp; After the gesture is completed, determine the spatial position information of the hand within the first region of the target object; In response to the spatial location information indicating that the hand is continuously located within the first area, the effective time window is determined as a first duration; otherwise, the effective time window is determined as a second duration, wherein the first duration is longer than the second duration. In response to the time difference being less than or equal to the effective time window, it is determined that the time correlation satisfies the set correlation condition.

5. The method according to claim 2, wherein determining the target object within the target interaction space based on the spatial correlation between the spatial coordinates corresponding to the gesture data and the location information comprises: Based on the spatial coordinates corresponding to the gesture data, determine the coordinate sequence of key hand points; Based on the hand key point coordinate sequence, the gesture movement trajectory is determined; The bounding box coordinates of each object within the target interaction space are determined as the position information; Objects that have a spatial relationship between the gesture trajectory and the location information are identified as candidate objects; Determine the priority of each candidate object, and identify the candidate object with the highest priority as the target object; The priority is associated with the attribute information of the candidate object and / or the voice data.

6. The method according to claim 1, wherein selecting a corresponding interaction strategy from a preset strategy set based on the target operation intention and the attribute information of the target object to generate a target task includes: The task content is determined based on the action type corresponding to the gesture data, the target operation intention, and the attribute information of the target object; Based on the task content and the type of the target object, determine the task execution scenario; Based on the task execution scenario, a corresponding interaction strategy is selected from a set of strategies; the interaction strategy includes at least one of information retrieval processing, text processing, image processing, and content generation processing. The target task is generated based on the interaction strategy and the task content.

7. The method according to claim 1, wherein outputting the task execution result of the target task to the multimodal output layer corresponding to the target interaction space via an electronic device comprises: Receive the task execution result corresponding to the target task; Based on the task execution results, at least one output method is determined; The output method includes at least one of the following: displaying the electronic device's display interface in a window format, outputting text in the display interface, and outputting audio based on the electronic device's audio device. Based on the determined output method, the task execution result is output to the multimodal output layer through the electronic device; the multimodal output layer includes at least one of image output through the display interface and sound output through the audio device.

8. A data processing apparatus, the apparatus comprising: The acquisition module is used to acquire multimodal data corresponding to the target interaction space; The multimodal data includes at least image data, voice data, and gesture data; The target interaction space represents the physical interaction space containing objects; The determination module is used to bind pronouns in the voice data to target objects in the target interaction space based on the correlation between image data, voice data and gesture data, and generate target operation intentions for the target objects. The aforementioned relationships include at least temporal relationships and spatial relationships; The generation module is used to select a corresponding interaction strategy from a preset strategy set based on the target operation intention and the attribute information of the target object, and generate a target task. An execution module is used to execute the target task and output the task execution result of the target task to the multimodal output layer corresponding to the target interaction space through an electronic device, in response to the target operation on the target object.

9. An electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform: Collect multimodal data corresponding to the target interaction space; the multimodal data includes at least image data, voice data, and gesture data; the target interaction space represents the physical interaction space containing objects; Based on the correlation between image data, voice data, and gesture data, pronouns in the voice data are bound to target objects in the target interaction space to generate target operation intentions for the target objects; the correlation includes at least temporal correlation and spatial correlation. Based on the target operation intention and the attribute information of the target object, a corresponding interaction strategy is selected from a preset strategy set to generate the target task; The target task is executed, and the task execution result is output to the multimodal output layer corresponding to the target interaction space via an electronic device to respond to the target operation for the target object.

10. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform: Collect multimodal data corresponding to the target interaction space; the multimodal data includes at least image data, voice data, and gesture data; the target interaction space represents the physical interaction space containing objects; Based on the correlation between image data, voice data and gesture data, pronouns in the voice data are bound to target objects in the target interaction space to generate target operation intentions for the target objects. The aforementioned relationships include at least temporal relationships and spatial relationships; Based on the target operation intention and the attribute information of the target object, a corresponding interaction strategy is selected from a preset strategy set to generate the target task; The target task is executed, and the task execution result is output to the multimodal output layer corresponding to the target interaction space via an electronic device to respond to the target operation for the target object.

Images