Large model-based video interaction method, device and product

Abstract

The present disclosure provides a large model-based video interaction method and device, electronic equipment, storage medium and computer program product, relates to the technical field of artificial intelligence, in particular to the technical field of large model, natural language understanding, video understanding, and can be applied to video calls and shared screen scenarios. The specific implementation scheme is as follows: in the video interaction process with the large model, the target object to which the spatial directional action associated with the video picture in the video interaction process is directed is determined; according to the input information associated with the spatial directional action, a data processing instruction for the target object is determined; using the large model, the target object is processed according to the data processing instruction to obtain a data processing result. The present disclosure allows users to express intentions in an intuitive way by combining spatial directional actions and information input, such as "pointing" and "speaking", thereby reducing the communication cost in the human-computer interaction process and improving the understanding efficiency and processing accuracy of user intentions in the human-computer interaction process.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of large models, natural language understanding, and video understanding, and particularly to a video interaction method, device, electronic device, storage medium, and computer program product based on a large model, which can be applied to video calls and screen sharing scenarios. Background Technology

[0002] Existing AI (Artificial Intelligence) systems can achieve a certain degree of human-computer interaction in scenarios such as video calls and screen sharing. However, AI systems have limitations in understanding user intentions during human-computer interaction. Summary of the Invention

[0003] This disclosure provides a video interaction method, apparatus, electronic device, storage medium, and computer program product based on a large model.

[0004] According to the first aspect, a video interaction method based on a large model is provided, including: in the process of video interaction with the large model, determining the target object targeted by the spatially directional action associated with the video frame in the video interaction process; determining the data processing instruction for the target object based on the input information associated with the spatially directional action; using the large model, performing data processing on the target object according to the data processing instruction, and obtaining the data processing result.

[0005] According to the second aspect, a video interaction device based on a large model is provided, comprising: an object determination unit configured to determine, during video interaction with the large model, a target object targeted by a spatially directional action associated with a video frame during the video interaction process; an instruction determination unit configured to determine a data processing instruction for the target object based on input information associated with the spatially directional action; and a data processing unit configured to use the large model to perform data processing on the target object according to the data processing instruction, and obtain a data processing result.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first aspect.

[0008] According to a fifth aspect, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0009] According to the technology disclosed herein, a video interaction method and apparatus based on a large model are provided. During video interaction with the large model, the target object of the spatially directional action associated with the video frame during the video interaction process is determined; based on the input information associated with the spatially directional action, data processing instructions for the target object are determined; using the large model, data processing is performed on the target object according to the data processing instructions to obtain the data processing result. This provides a human-computer interaction method that combines multimodal data of spatially directional actions, video frames, and input information, allowing users to express their intentions in an intuitive way by combining spatially directional actions and information input, such as "pointing" and "speaking," reducing communication costs during human-computer interaction and improving the efficiency of understanding and processing user intentions.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0012] Figure 1 This is an exemplary system architecture diagram that can be applied to an embodiment of this disclosure;

[0013] Figure 2 This is a flowchart of an embodiment of the video interaction method based on a large model according to the present disclosure;

[0014] Figure 3 This is a schematic diagram illustrating an application scenario of the video interaction method based on a large model according to this embodiment;

[0015] Figure 4 This is a schematic diagram of data processing instructions combining text and graphics according to this embodiment;

[0016] Figure 5A-5G This is a schematic diagram of the data processing process based on multiple spatial directional actions according to this embodiment;

[0017] Figure 6 This is a flowchart of yet another embodiment of the large-model-based video interaction method according to the present disclosure;

[0018] Figure 7This is a flowchart of yet another embodiment of the large-model-based video interaction method according to the present disclosure;

[0019] Figure 8 This is a structural diagram of one embodiment of a large-model-based video interaction device according to the present disclosure;

[0020] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0023] Figure 1 An exemplary architecture 100 is shown that can be applied to the large-model-based video interaction method and apparatus disclosed herein.

[0024] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. The communication connections between terminal devices 101, 102, and 103 form a network topology. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0025] Terminal devices 101, 102, and 103 can be hardware or software that supports network connectivity for data interaction and processing. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices that support network connectivity, information acquisition, interaction, display, and processing functions, including but not limited to smartphones, tablets, e-book readers, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. They can be implemented as, for example, multiple software programs or software modules to provide distributed services, or as a single software program or software module. No specific limitations are imposed here.

[0026] Server 105 can be a server that provides various services, such as a background processing server that acquires multimodal data, including video feeds displayed by terminal devices 101, 102, and 103, user input information related to the video feeds, and spatial directional actions, and performs human-computer interaction based on the multimodal data. As an example, server 105 can be a cloud server.

[0027] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (such as software programs or software modules used to provide distributed services), or as a single software program or software module. No specific limitations are made here.

[0028] It should also be noted that the large-model-based video interaction method provided in the embodiments of this disclosure is generally executed by a server, but the possibility of it being executed by a terminal device, or by the server and terminal device cooperating with each other, is not excluded. Accordingly, the various parts (e.g., various units) included in the large-model-based video interaction device can be all located in the server, all located in the terminal device, or separately located in the server and the terminal device.

[0029] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Any number of terminal devices, networks, and servers can be included depending on implementation needs. When the electronic devices running the large-model-based video interaction method do not require data transmission with other electronic devices, the system architecture may consist only of the electronic devices (e.g., terminal devices or servers) running the large-model-based video interaction method.

[0030] Please refer to Figure 2 , Figure 2 A flowchart illustrating a video interaction method based on a large model, provided for embodiments of this disclosure. Flowchart 200 includes the following steps:

[0031] Step 201: During the video interaction with the large model, determine the target object of the spatially directional action associated with the video frame during the video interaction process.

[0032] In this embodiment, the execution entity of the video interaction method based on the large model (e.g., Figure 1 The server can obtain video footage and spatially directional actions associated with the video footage from the user's terminal device via wired or wireless network connection. It can also identify the target object of the spatially directional actions associated with the video footage.

[0033] Video feeds are the real-time video frames displayed on the user's terminal device. These can be from a video call or a shared screen. They can be captured in real-time through screen recording, camera access, or direct video streaming via a media player. It's important to note that any of these methods requires user authorization before they can capture the video feed in real-time.

[0034] Spatial directional actions associated with video footage can be actions performed by the user on the display screen, such as clicking, touching, selecting, drawing, and dragging target objects on the display screen based on mouse pointers or touchscreens. They can also be eye movements representing the user's gaze point or gaze trajectory in the video footage (in which case eye-tracking technology, such as an eye tracker, is needed to acquire spatial directional actions). Alternatively, they can be spatial directional actions expressed by the user based on gestures or other physical movements in the video footage. For example, a user can make physical gestures such as pointing, selecting, or grabbing at a target object in the real environment, and the camera can capture the real-world environment including the physical gestures to obtain the video footage in real time.

[0035] The target object can be any object in the video frame. It can be a real physical object such as flowers or objects, or a non-physical object presented in the form of text, graphics, or symbols. It can also be a virtual object generated by a computer with the development of technologies such as virtual reality and augmented reality.

[0036] Users can engage in video interactions with the large model, such as video calls and screen sharing. Large models (artificial intelligence large models) refer to a class of artificial intelligence models with a large number of parameters built from artificial neural networks, such as large language models, large vision models, multimodal large models, and basic science large models. Taking a large language model as an example, it is a large-scale language model built based on deep learning technology, primarily used for natural language processing tasks. Through training on large-scale data, it learns language patterns and structures, enabling it to generate natural language text or understand natural language input. This embodiment may specifically employ a multimodal large language model, which typically includes the following modules:

[0037] Input module: Receives multimodal data such as text, images, videos, and input information from users, as well as text generation requests such as questions, instructions, or dialogue content.

[0038] Preprocessing module: preprocesses the input multimodal data. For example, text data preprocessing includes word segmentation, stop word removal, text cleaning, etc., to convert the text into a form that the model can process.

[0039] The encoding module encodes the preprocessed multimodal data into vector form so that the model can understand and process it. Common encoding methods include word embeddings and encoders in the Transformer architecture. Word embeddings include, for example, Word2Vec (Words to Vector) and GloVe (Global Vectors for Word Representation).

[0040] Model module: The core component, typically based on a deep learning architecture (such as Transformer), responsible for processing encoded data vectors and performing language understanding and generation. The model learns complex patterns and semantic relationships of language through multi-layered neural network structures.

[0041] Decoding module: Decodes the model's output vectors into natural language text, images, videos, and input information, generating responses or processing results for user input. Decoding methods can include greedy decoding, beam search, etc.

[0042] Output module: Outputs the decoded text in a user-readable format, such as text, images, videos, and input information displayed on the screen.

[0043] As an example, firstly, the target object of the spatial directional action is determined according to the type of implementation of the spatial directional action. For example, if the spatial directional action is an action performed by the user on the display screen of the video, the position targeted by the spatial directional action is determined based on the display screen; if the spatial directional action is a spatial directional action expressed by the user based on gestures or other body movements in the video, the target object of the spatial directional action is determined based on the recognition of the spatial directional action in the video.

[0044] As another example, the type of spatially directional action and video footage are input into a pre-trained target object recognition model. The target object recognition model then determines the target object targeted by the spatially directional action. This model characterizes the object relationships between the type of spatially directional action, video footage, and the target objects targeted by the associated spatially directional actions. It can be obtained by training recurrent neural networks, convolutional neural networks, or other neural networks using supervised learning and other machine learning algorithms. The target object recognition model can be implemented using a large-scale model.

[0045] In the process of identifying target objects, data processing can be carried out in at least one of the following ways, but is not limited to:

[0046] Object recognition: Using pre-trained visual models to identify visual objects covered or near the mouse pointer or touch point, such as user interface elements (buttons, text boxes, scroll bars), specific text content (titles, paragraphs, keywords), specific areas in images, specific objects or people in videos, etc.

[0047] Semantic segmentation or instance segmentation: Based on the context of user dialogue, perform fine semantic segmentation or instance segmentation on video screens to determine the semantic category (e.g., "chart area", "text area") or specific instance object (e.g., "third quarter bar chart in sales trend chart") of the area where the mouse pointer or touch point is located.

[0048] Contextual understanding: By combining the surrounding visual elements and overall screen layout of a spatially directional action, the target object pointed to by the spatially directional action is determined. For example, if a spatially directional action points to an interactive button, the target is identified as "button"; if a spatially directional action points to a specific text paragraph, the target object is identified as "text paragraph"; if a spatially directional action points to a person's face, it is identified as "person's face".

[0049] Step 202: Determine the data processing instructions for the target object based on the input information associated with the spatial directional action.

[0050] In this embodiment, the aforementioned execution entity can determine data processing instructions for the target object based on the input information associated with spatially directional actions.

[0051] Input information associated with spatially directional actions refers to the input information emitted by a user when performing a spatially directional action. This input information can be represented in at least one of the following forms: text, voice, or image. For example, when performing a spatially directional action, a user can simultaneously input text data into the aforementioned execution entity. For instance, in response to detecting the user's spatially directional action, the execution entity displays a text input box on the terminal screen, and the text information entered by the user through this box constitutes the input information associated with the spatially directional action. Similarly, when performing a spatially directional action, a user can simultaneously emit speech. In this case, the user can intuitively express their intention by combining "pointing" and "speaking," with "pointing" specifically representing a spatially directional action and "speaking" specifically representing speech. For example, a user can point to a "flower" in a video frame and simultaneously say, "What kind of flower is this?" Finally, when performing an image action, a user can simultaneously input image data into the aforementioned execution entity. For instance, in response to detecting the user's spatially directional action, the execution entity displays an image upload box on the terminal screen, and the image data uploaded by the user through this box constitutes the input information associated with the spatially directional action.

[0052] Taking the input information as represented by voice and image as an example, the user makes a spatial directional action while uttering the voice command "process this into the style shown in the uploaded image", and uploads the image data to the aforementioned execution entity through the image upload box.

[0053] As an example, the semantics represented by the input information can be identified. For instance, speech-to-text technology can be used to obtain the recognized text of the speech, which can then be directly used as a data processing instruction for the target object.

[0054] As another example, deep understanding can be achieved from video footage and spatial directional actions to determine the user's true operational intent towards the target object; based on this true operational intent, data processing instructions for the target object can be generated. For instance, video footage, spatial directional actions, and the recognized text can be input into a large model, which can then determine the user's true operational intent towards the target object and generate data processing instructions.

[0055] Data processing instructions can be various data processing instructions used in scenarios such as office work, study, and entertainment, including but not limited to:

[0056] The purpose is to obtain data processing instructions for static attributes of a target object at the geometric, physical, and semantic levels. Static attributes include, but are not limited to, size, material, color, texture, quality, transparency, reflectivity, and category.

[0057] Data processing instructions used to retrieve prior knowledge or external knowledge base information related to the target object, covering historical background, cultural connotations, functional uses, structural principles, ecological significance, related events, and multilingual interpretations;

[0058] Data processing commands that allow real-time modification of the spatial pose, appearance features, and dynamic behavior of target objects in the image. Modifications include, for example, displacement, rotation, scaling, deformation, color mapping, transparency adjustment, lighting condition changes, texture replacement, and redefinition of interactive relationships with other objects such as collision, occlusion, snapping, and linkage.

[0059] Step 203: Using a large model, perform data processing on the target object according to the data processing instructions to obtain the data processing results.

[0060] In this embodiment, the aforementioned execution entity can adopt a large model to perform data processing on the target object according to the data processing instructions, and obtain the data processing result.

[0061] In this embodiment, data processing instructions are input into a large model. Based on its powerful natural language understanding and logical analysis capabilities, the large model performs a deep understanding of the instructions and processes the target object based on the understanding results. The resulting data is then displayed on the user's terminal device. The data processing results can be represented in various forms, such as text, input information, images, and videos, or a combination of these forms.

[0062] Continuing with the data processing instructions above used to analyze the health status of leaves, the large model extracts features from the image of the target object (leaves), determines its health status based on the feature extraction results, and provides targeted suggestions based on the health status.

[0063] As another example, the data processing instruction is "turn the yellow leaves of this plant into green leaves". The large model extracts features from the target object (plant) and determines the yellow leaves on it based on the feature extraction results, and adjusts the yellow leaves on the plant in the video to green leaves.

[0064] See also Figure 3 , Figure 3 This is a schematic diagram 300 illustrating an application scenario of the large-model-based video interaction method according to this embodiment. User 301 is processing a presentation via terminal device 302 and requests assistance from the AI ​​system set up on server 303 during the presentation processing by sharing their screen. During the presentation processing, the user finds they do not understand the meaning of a technical term, "classical computing bottleneck," and points the mouse pointer at the technical term 304 in the currently displayed video frame on the terminal device, uttering the question 305, "What does this term mean?" The AI ​​system acquires the video frame and identifies the technical term (target object) associated with the spatially directional action. Based on the question associated with the spatially directional action, it determines the data processing instruction for the target object, "What does the term 'classical computing bottleneck' mean?" Using a large model, it processes the data of the target object according to the data processing instruction, obtaining the specific meaning of the technical term (data processing result) and displaying it through the terminal device.

[0065] This embodiment provides a video interaction method based on a large model. During video interaction with the large model, the target object of the spatially directional action associated with the video frame is determined. Based on the input information associated with the spatially directional action, data processing instructions for the target object are determined. Using the large model, data processing is performed on the target object according to the data processing instructions to obtain the data processing result. This provides a human-computer interaction method that combines multimodal data of spatially directional actions, video frames, and input information. It allows users to express their intentions intuitively by combining spatially directional actions and information input, such as "pointing" and "speaking," reducing communication costs during human-computer interaction and improving the efficiency and accuracy of understanding and processing user intentions.

[0066] In some optional implementations of this embodiment, the execution entity can perform step 202 as follows:

[0067] The first step is to generate a semantic description text of the target object in response to the fact that the description part of the target object in the recognition text of the input information is an implicit reference description.

[0068] First, the semantics of the input information are identified to obtain its recognized text. For example, based on speech-to-text technology, the recognized text of the speech is obtained, and the timestamp of the recognized text is retained from the input information. Based on the timestamp of the recognized text and the timestamp of the spatially directional action, the descriptive portion of the target object in the recognized text is determined. For example, the portion of the recognized text with the same timestamp as the spatially directional action is used as the descriptive portion of the target object. To improve the accuracy of determining the descriptive portion of the target object in the recognized text, the semantic understanding and timestamps of the recognized text can be combined to determine the descriptive portion of the target object.

[0069] Then, based on the semantic understanding of the description of the target object, it is determined whether it is an implicit referential description. Implicit referential description refers to the use of language to indirectly imply the target object without directly stating the specific person, thing, or object, but through the context, mutually known information, or other related data. Words like "this" and "which" are examples of implicit referential descriptions.

[0070] Finally, in response to the recognition that the description of the target object in the text is an implicit referential description, a semantic description text that can represent the target object is generated. For example, the semantic description text represents what kind of object the target object is, such as the type or name of the target object.

[0071] The second step involves combining the semantic description text and the recognized text to determine the data processing instructions.

[0072] In this implementation, the description of the target object in the recognition text is replaced with semantic description text, resulting in a fused text that adjusts the implicit referential description to an explicit referential description. This fused text is then used as a data processing instruction. For example, in the recognition text "What kind of thing is this?", where the implicit referential description of the target object is "this", and the semantic description text of the target object is "a pot of flowers", then the fused text is "What kind of thing is this pot of flowers?".

[0073] In some implementations, to improve the semantic integrity and accuracy of data processing instructions, richer semantic description text can be generated. This semantic description text includes basic attribute information such as morphological features and appearance attributes; spatial relationships such as location and interaction with other objects; state and action descriptions; functional and attribute association information (for targets with specific functions, this includes functional descriptions, such as "This device is a portable charger used to power electronic devices"; it may also include attribute associations with other objects, such as "This key matches the lock on the right side of the screen"); and scene and contextual information (such as "The book in the image is placed on a library shelf, surrounded by similar books").

[0074] In this implementation, a large model can be used to perform deep understanding of the recognized text, and the target information required to perform the data processing task represented by the recognized text can be determined by combining the deep understanding results. Then, video understanding or image understanding can be performed on the target object to determine the target information of the target object, thereby generating semantic description text.

[0075] In this implementation, when the description of the target object in the identified text is an implicit reference, the data processing instructions are determined by combining the semantic description text of the target object and the identified text, thereby improving the accuracy and completeness of the data processing instructions.

[0076] In some optional implementations of this embodiment, the execution entity can perform the second step as follows: First, combine the semantic description text and the recognition text to determine the fused text; then, combine the visual data of the target object into the description part of the target object in the fused text to determine the data processing instructions.

[0077] Visual data can be image data or video data of the target object. Taking image data as an example, the image portion corresponding to the target object in a video frame can be cropped to obtain visual data in the form of an image of the target object. Taking video data as an example, target video frames containing the target object can be selected from multiple video frames displayed on the terminal device screen and used as visual data in the form of a video of the target object. To further improve the targeting of the visual data in the form of a video of the target object, the target object in each target video frame containing the target object can be cropped to a fixed size to obtain cropped frames. Multiple cropped frames can then be combined according to the temporal relationship between the target video frames to obtain visual data in the form of a video containing only the target object.

[0078] For example, for static target objects, visual data in the form of images can be used; for dynamic target objects, visual data in the form of videos can be used.

[0079] In this implementation, firstly, the description of the target object in the recognized text is replaced with semantic description text, resulting in fused text that adjusts implicit referential descriptions to explicit referential descriptions. Then, the visual data of the target object is combined with the description of the target object in the fused text, generating data processing instructions that combine text and images or video and text. (Continue to refer to...) Figure 4 The diagram illustrates a data processing instruction combining text and images. The data processing instruction 400 includes an image portion 401, a potted peony, and a text portion 402, "What kind of flower is this?".

[0080] In this implementation, after clarifying the implicit references in the identified text, the visual data of the target object is further combined to obtain data processing instructions that combine images and text or video and text. This further improves the data richness and semantic expressiveness of the data processing instructions, which helps to further improve the accuracy of the large model's intent understanding based on the data processing instructions and the efficiency of data processing.

[0081] In some optional implementations of this embodiment, the execution entity can perform step 202 as follows: in response to the fact that the description part of the target object in the recognition text of the input information is an explicit referential description, the visual data of the target object is combined with the description part of the target object in the recognition text to determine the data processing instruction.

[0082] Explicit referential description refers to a descriptive method in language that uses explicit words or phrases to directly and clearly point to a specific object, making the identity, scope, or characteristics of the referred object directly identifiable and understandable. For example, for a target object like a peony, the descriptive part would be "this pot of peonies".

[0083] In this implementation, the visual data of the target object is combined with the description of the target object in the fused text, and then a data processing instruction combining images and text or video and text is generated.

[0084] In this implementation, after determining that the description of the target object in the identified text is an explicit referential description, the visual data of the target object is further combined to obtain data processing instructions that combine images and text or video and text. This further improves the data richness and semantic expressiveness of the data processing instructions, which helps to further improve the accuracy of the large model's intent understanding based on the data processing instructions and the efficiency of data processing.

[0085] In some optional implementations of this embodiment, a single user input is associated with multiple spatially directional actions. For example, a user may perform multiple spatially directional actions while uttering a voice command. In terms of action type, the multiple spatially directional actions can be of the same type, such as two spatially directional actions both representing the user clicking a target object in a video frame using a mouse pointer; or they can be actions of different types, such as one spatially directional action representing the user clicking a target object in a video frame using a mouse pointer, and another spatially directional action representing the user selecting a target object in a video frame using eye movements. Generally, multiple spatially directional actions target different target objects; however, it is not excluded that multiple spatially directional actions target the same target object.

[0086] In the temporal relationship dimension, multiple spatial directional actions can be actions performed by the user simultaneously, such as touching two target objects on the video screen with two fingers at the same time; or they can be actions performed at different times. For example, the first spatial directional action represents the user touching the first target object in the video screen with their finger, and the second spatial directional action represents the user touching the second target object in the video screen with their finger.

[0087] In terms of the relationship dimension, multiple spatial directional actions can be actions with a relationship, such as swapping the positions of the target objects targeted by two spatial directional actions; or they can be actions without a relationship, such as two spatial directional actions that simply point to two independent target objects that have no relationship, and the subsequent data processing processes corresponding to the two target objects are completely unrelated.

[0088] In this implementation, the aforementioned execution entity can perform step 202 as follows:

[0089] The first step is to generate a semantic description text of the target object in the recognition text of the input information for multiple spatially directional actions, in response to the implicit referential description of the target object targeted by the spatially directional action in the recognition text.

[0090] First, the semantics of the input information are identified to obtain its recognized text. For example, based on speech-to-text technology, the recognized text of the speech is obtained, and the timestamp of the input information is used in the recognized text. Then, for each spatially directional action among multiple spatially directional actions, the descriptive part of the target object targeted by the spatially directional action in the recognized text is determined based on the timestamp of the recognized text and the timestamp of the spatially directional action. For example, the part of the recognized text with the same timestamp as the spatially directional action is taken as the descriptive part of the target object. To improve the accuracy of determining the descriptive part of the target object in the recognized text, the semantic understanding of the recognized text and the timestamp can be combined to determine the descriptive part of the target object in the recognized text. Based on the semantic understanding of the descriptive part of the target object targeted by the spatially directional action, it is determined whether it is an implicit referential description. In response to the fact that the descriptive part of the target object targeted by the spatially directional action in the recognized text is an implicit referential description, a semantic descriptive text that can represent the target object is generated.

[0091] The second step is to perform temporal alignment between the input information's recognized text and multiple spatially directional actions to determine the temporal correspondence between the multiple spatially directional actions and the recognized text.

[0092] Based on the timestamp of the recognized text and the timestamps of each of the multiple spatially directional actions, the recognized text and the multiple spatially directional actions of the input information are temporally aligned to determine the description parts in the recognized text that have the same timestamp as the multiple spatially directional actions, thus obtaining the temporal correspondence between the multiple spatially directional actions and the recognized text.

[0093] As an example, the identified text is "swap this object with this object," where the first "this object" matches the timestamp of the first spatially directional action (pointing to the peony), and the second "this object" matches the timestamp of the second spatially directional action (pointing to the rose). It can be understood that the descriptive part that has a temporal correspondence with a spatially directional action is the descriptive part of the target object that the spatially directional action targets.

[0094] The third step is to determine the data processing instructions based on the temporal correspondence, combined with the semantic description text and the recognized text.

[0095] As an example, for each spatially directional action whose description of the target object is an implicit referential description, the description text of the target object targeted by the spatially directional action replaces the description part of the identification text that has a temporal correspondence with the spatially directional action, thus obtaining a data processing instruction. That is, for each implicit referential description in at least one implicit referential description associated with the input information, the semantic description text of the target object corresponding to the implicit referential description replaces the implicit referential description in the identification text.

[0096] This implementation provides a data processing instruction generation method for multiple spatial directional actions. Based on the support for multiple spatial directional actions, it further improves the flexibility and convenience of users combining spatial directional actions with information input, such as "pointing" and "speaking," to express their intentions, while also enriching the data processing methods for target objects.

[0097] In some optional implementations of this embodiment, the execution entity can perform the third step as follows: First, based on the temporal correspondence, the semantic description text and the recognition text are combined to determine the fused text; then, for multiple spatially directional actions, the visual data of the target object targeted by the spatially directional action is combined with the description part of the target object in the fused text to obtain data processing instructions.

[0098] In this implementation, firstly, for each spatially directional action whose description of the target object is an implicit referential description, the semantic description text of the target object targeted by the spatially directional action replaces the description part of the recognized text that has a temporal correspondence with the spatially directional action, resulting in a fused text that adjusts the implicit referential description to an explicit referential description. Then, for each target object, the visual data of the target object is combined with the description part of the target object in the fused text to generate a data processing instruction combining text and images, video and text, or images, videos, and text. In the same data processing instruction, the visual data of the target objects targeted by multiple spatially directional actions can adopt the same data format, such as images; or they can adopt different data formats, for example, some target objects have visual data in the form of images, while others have visual data in the form of videos.

[0099] In this implementation, after clarifying the implicit referential descriptions in the identified text, the visual data of the target objects targeted by multiple spatial directional actions are further combined to obtain data processing instructions. This not only improves the data richness of the data processing instructions but also enriches the data processing methods for the target objects.

[0100] Continue to refer to Figure 5A-5G This diagram illustrates the data processing procedure based on multiple spatial directional actions.

[0101] The user's voice message was, "I want to put this painting here on the windowsill. Can you generate a mockup for me?" Figure 5A In the video shown, the user's first spatial directional action was clicking the position corresponding to "drawing" 501 with the mouse pointer; corresponding to Figure 5BIn the video shown, the user's second spatial pointing action was to select the target location range 502 on the "windowsill" using the mouse pointer, and finally the mouse pointer stopped at... Figure 5C At the location shown.

[0102] For the first spatial directional action, the aforementioned executing entity combines the location pointed to by the first spatial directional action with the corresponding part of the speech-recognized text "this painting" to identify the target object it is pointing to. The recognition effect is as follows: Figure 5D As shown.

[0103] For second-space directional actions, the aforementioned executing entity sets the action trajectory of the second-space directional action and the corresponding partial speech recognition text "this position on the windowsill" to identify its target location range. The recognition effect is as follows: Figure 5E As shown.

[0104] For each target object, after combining the visual data of the target object with the description of the target object in the fused text, a data processing instruction for combining text and images is generated, such as... Figure 5F As shown.

[0105] The large model processes the target object based on data processing instructions to obtain the data processing results, such as... Figure 5G As shown.

[0106] In some optional implementations of this embodiment, the execution entity can perform step 203 as follows: using a large model, performing data processing on the target object according to the data processing instructions, the context of the input information, and the video frame, to obtain the data processing result.

[0107] As an example, the aforementioned executing entity can use a multimodal large model to deeply analyze data processing instructions that integrate visual and textual information of the target object, understanding the complex intentions contained within. The large model can accurately distinguish between different types of instructions, such as asking questions, requesting specific actions, seeking explanations, or engaging in emotional interaction. Furthermore, the executing entity can input data processing instructions, the context of input information, and video footage into the large model. The large model combines the context of historical dialogues, the overall semantics of the current video footage, and the precise target object pointed to by the user to perform deeper reasoning, and then process the data of the target object based on the deep understanding results.

[0108] In this implementation, the large model combines data processing instructions, the context of input information, and video footage to process the target object, further improving the accuracy of the data processing results.

[0109] In some optional implementations of this embodiment, the execution entity can perform step 201 as follows:

[0110] The first step is to determine the position of spatially directional actions on the video screen during video interaction.

[0111] In this implementation, during the user's video interaction with the large model, the aforementioned execution entity collects the user's input information, the video feed from the user's terminal device, and the user's spatial directional actions in response to the video feed in real time, and performs temporal alignment on the multimodal data, including input information, video feed, and spatial directional actions. It should be noted that the collection of this multimodal data is performed with the user's authorization.

[0112] For spatial pointing actions made by the user with the mouse, firstly, the absolute coordinates of the mouse pointer on the screen are captured by the terminal device. Then, based on the actual display area of ​​the current video on the screen, the absolute coordinates are mapped to pixel coordinates relative to the original resolution of the video by the horizontal and vertical scaling ratios, thereby determining the position of the mouse pointer in the video.

[0113] For spatial directional actions performed by users through touch screen, the terminal device first captures the absolute coordinates of the user's touch on the screen. Then, based on the transformation matrix of the video image caused by stretching, cropping, or rotation, and combined with the proportional relationship between the original size of the video and the display area, the absolute coordinates are back-calculated back to the original coordinate system of the video image, thereby determining the accurate position of the touch point in the video image.

[0114] The second step is to determine the location of the target object in the video frame.

[0115] As an example, the aforementioned execution entity can input video footage and location into the target object recognition model, and the target object recognition model can output the target object located in the video footage.

[0116] This implementation provides a target object recognition method that combines spatially directional actions. Based on the explicit direction of the action, it improves the recognition efficiency and accuracy of the target object.

[0117] In some optional implementations of this embodiment, the execution entity can perform the second step as follows: determine the target object at the above-mentioned position in the video frame based on the description portion of the recognition text of the input information that is at the same time as the spatial directional action.

[0118] As an example, the recognition text and spatial directional action can be temporally aligned to determine the description portion of the recognition text that occurs at the same time as the spatial directional action; based on the semantic understanding of the description portion, the semantic information of the target object can be determined; based on the semantic information, the target object at the aforementioned location in the video frame can be determined.

[0119] In this implementation, based on spatial directional actions, the target object in the video frame is further determined by the description portion of the recognition text in the input information that occurs at the same time as the spatial directional actions. This helps to further improve the recognition efficiency and accuracy of the target object.

[0120] In some optional implementations of this embodiment, the execution entity can perform the first step as follows: during video interaction, determine the target location range based on the position points of the action trajectory represented by the spatial directional action on multiple video frames.

[0121] In this implementation, the aforementioned execution entity can determine whether the position of the same spatial directional action in the video frame has moved. In response to determining that movement has occurred, the spatial directional action is characterized by an action trajectory, and the position points on multiple video frames are determined, so as to determine the target position range represented by multiple position points.

[0122] For example, for each video frame touched by a user's spatial directional action, the position point of the spatial directional action in that video frame is determined, and multiple position points are connected according to the temporal relationship between video frames, thereby determining the motion trajectory of the spatial directional action in the video and determining the target position range indicated by spatial directional actions such as circle selection and line drawing.

[0123] In this implementation, the aforementioned execution entity can perform the second step as follows: identify the target object within the target location range in the video frame.

[0124] In response to identifying an object within a target location range in a video frame, the object is identified as the target object; in response to identifying multiple objects within a target location range in a video frame, semantic understanding is performed on the description portion of the recognition text that occurs at the same time as the spatial directional action to determine the semantic information of the target object, and the target object is identified from multiple objects based on the semantic information.

[0125] When the semantic information of the target object is relatively ambiguous, a confirmation request can be sent to the user for multiple identified objects, so that the target object can be filtered out based on the user's instructions.

[0126] This implementation provides a method for identifying target objects within a target location range, allowing users to indicate target objects through action sequences, thereby improving user action flexibility and the user experience during human-computer interaction.

[0127] In some optional implementations of this embodiment, the execution entity can perform the process of determining the target location range in the following manner:

[0128] First, during video interaction, the hot zone targeted by the spatially directional action is determined based on the position points of the action trajectory represented by the spatially directional action on multiple video frames and the attribute information of the action trajectory.

[0129] The attributes of the motion trajectory include, but are not limited to, shape, speed, and dwell time. The motion trajectory data represented by the captured location points is input into a pre-trained directional intent judgment model. This model can understand the directional intent represented by different trajectory patterns. For example, the model can identify whether the user is performing spatial directional actions such as circling or drawing lines. Furthermore, the model analyzes the shape, speed, and dwell time of the pointer's motion trajectory, and, combined with video footage, predicts the hotspot of the user's intended direction.

[0130] Then, based on the heat zone, determine the target location range.

[0131] As an example, the location range represented by the hot zone can be used as the target location range.

[0132] In this implementation, the target location range is determined based on the position points of the motion trajectory on multiple video frames and the attribute information of the motion trajectory, thereby improving the accuracy of the target location range.

[0133] In some optional implementations of this embodiment, the execution entity may also perform the first step as follows: during video interaction, determine the position point of the instantaneous action represented by the spatial directional action on the video screen.

[0134] Instantaneous actions include actions such as clicking a mouse or touching the screen.

[0135] In this implementation, the aforementioned executing entity can determine whether the position of the spatial directional action in the video frame has moved. In response to determining that no movement has occurred, the spatial directional action is determined to be an instantaneous action, and its position point in the video frame is determined.

[0136] In this implementation, the aforementioned execution entity can perform the first step as follows: perform object recognition at the location point in the video frame to determine the target object.

[0137] A location point is a point on or near the target object. Based on the location point, object recognition can be performed on the corresponding object to obtain the target object.

[0138] This implementation provides a method for identifying target objects based on location points, allowing users to indicate target objects through instantaneous actions, thereby improving the user's action flexibility and the experience during human-computer interaction.

[0139] In some optional implementations of this embodiment, the video interaction process includes a video call process, a screen sharing process, and a video dialogue process. That is, the video screen can be the video screen displayed in the video call interface, the video screen displayed in the screen sharing interface, or the video screen of the user communicating with the large model in video form.

[0140] Taking the video screen displayed in the video call interface as an example, users can make video calls with other users or with a large model. The aforementioned execution entity collects the video screen displayed in the video call interface, the user's spatial directional actions, and input information in real time.

[0141] In AI video call or video stream analytics applications, users can point directly to real-world objects, people, or scenes, just as if communicating with a real person. Large models can understand the target object the user is pointing to and engage in precise dialogue.

[0142] In one example, when a user displays a plant, they can precisely point to a leaf of the plant with their finger or mouse pointer and ask, "Is this leaf sick?" The execution entity can recognize that the user is pointing to the target object "leaf" rather than the entire plant, and determine the data processing instructions used to analyze the health status of the leaf.

[0143] Taking the video screen displayed in the shared screen interface as an example, the user can share the screen of their terminal with the aforementioned execution entity, enabling the execution entity to capture the video screen displayed in the video call interface in real time, as well as the user's spatial directional actions and input information.

[0144] In some AI screen-sharing collaboration tools, users share the screen content of their terminal device (computer or mobile phone) with a large model. The large model can understand any UI (User Interface) elements, text, images, charts, etc. that the user points to on the screen and determine the data processing instructions for the target object on the shared screen.

[0145] In one example, a user hovers their mouse pointer over a specific word on a page in a shared presentation and asks via voice, "What is the definition of this word?" The aforementioned execution entity can then generate data processing instructions to determine the definition of the word.

[0146] Taking video footage from a conversation as an example, users can send pre-recorded videos to a large model. During video recording, users make spatial pointing gestures and input information. For instance, while recording, a user points to a target object in a real-world scene and speaks accordingly. The user's device captures the scene's visuals and the user's speech to obtain the recorded video. Another example is screen recording, where the user records the content displayed on their device's screen, including spatial pointing gestures using the mouse pointer, and also captures the user's speech during the recording process to obtain the recorded video.

[0147] In video calls, screen sharing, and video conversations, users can express their intentions in an intuitive way by combining spatial directional actions and information input, such as "pointing" and "speaking." This reduces communication costs in the human-computer interaction process and improves the efficiency of understanding user intentions and the accuracy of data processing.

[0148] Continue to refer to Figure 6 The illustration shows a schematic flow 600 of yet another embodiment of the large-model-based video interaction method according to this disclosure. Flow 600 includes the following steps:

[0149] Step 601: During the video interaction with the large model, determine the action type of the spatially directional action associated with the video frame during the video interaction process.

[0150] Action types include sequential actions and instantaneous actions.

[0151] Step 602: In response to the action type being a sequence action, determine the target location range based on the position points of the action trajectory represented by the spatial directional action on multiple video frames.

[0152] Step 603: Perform object recognition within the target location range in the video frame to determine the target object.

[0153] Step 604: In response to the action type being instantaneous action, determine the position point of the instantaneous action represented by the spatially directional action on the video screen.

[0154] Step 605: Perform object recognition at the location points in the video frame to determine the target object.

[0155] Step 606: Determine the description type of the description portion of the target object in the recognition text of the input information associated with spatial directional actions.

[0156] Description types include implicit reference descriptions and explicit reference descriptions.

[0157] Step 607: In response to the fact that the description part of the target object in the recognition text of the input information is an implicit referential description, generate a semantic description text of the target object.

[0158] Step 608: Combine the semantic description text and the recognized text to determine the fused text.

[0159] Step 609: Combine the visual data of the target object with the description of the target object in the fused text to determine the data processing instructions.

[0160] Step 610: In response to the fact that the description of the target object in the recognition text of the input information is an explicit referential description, the visual data of the target object is combined with the description of the target object in the recognition text to determine the data processing instructions.

[0161] Step 611: Using a large model, perform data processing on the target object according to the data processing instructions to obtain the data processing results.

[0162] The process 600 of the video interaction method based on the large model in this embodiment, compared with the above process 200, specifically describes the object recognition process based on spatial directional actions and the data processing instruction generation process. It allows users to express their intentions in an intuitive way by combining spatial directional actions and information input, such as "pointing" and "speaking," which further reduces the communication cost in the human-computer interaction process and improves the efficiency of understanding and processing user intentions in the human-computer interaction process.

[0163] Continue to refer to Figure 7 The illustration shows a schematic flow 700 of yet another embodiment of the large-model-based video interaction method according to this disclosure. Flow 700 includes the following steps:

[0164] Step 701: During the video interaction with the large model, for multiple spatially directional actions associated with the received input information, perform the following operations:

[0165] Step 7011: Determine the action type of the spatially directional action.

[0166] Step 7012: In response to the action type being a sequence action, determine the target location range based on the position points of the action trajectory represented by the spatial directional action on multiple video frames.

[0167] Step 7013: Perform object recognition within the target location range in the video frame to determine the target object.

[0168] Step 7014: In response to the action type being instantaneous action, determine the position point of the instantaneous action represented by the spatially directional action on the video screen.

[0169] Step 7015: Perform object recognition at the location points in the video frame to determine the target object.

[0170] Step 7016: Determine the description type of the description portion of the target object in the recognition text of the input information.

[0171] Step 7017: In response to the recognition that the description part of the target object in the text is an implicit referential description, generate a semantic description text of the target object.

[0172] Step 702: Perform temporal alignment between the recognized text and multiple spatially directional actions to determine the temporal correspondence between the multiple spatially directional actions and the recognized text.

[0173] Step 703: Based on the temporal correspondence, combine the semantic description text and the recognized text to determine the fused text.

[0174] Step 704: For multiple spatial directional actions, combine the visual data of the target object targeted by the spatial directional action into the description part of the target object in the fused text to obtain data processing instructions.

[0175] Step 705: Using a large model, perform data processing on the target object according to the data processing instructions to obtain the data processing results.

[0176] The process 700 of the video interaction method based on a large model in this embodiment, compared with the above-mentioned process 200, specifically describes the object recognition process of multiple spatial directional actions and the generation process of data processing instructions. It allows users to express their intentions in an intuitive way by combining spatial directional actions and information input, such as "pointing" and "speaking," which further reduces the communication cost in the human-computer interaction process and improves the efficiency of understanding and processing user intentions in the human-computer interaction process.

[0177] Continue to refer to Figure 8 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a video interaction device based on a large model. This system embodiment is similar to... Figure 2 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0178] like Figure 8As shown, the large-model-based video interaction device 800 includes: an object determination unit 801, configured to determine the target object of the spatially directional action associated with the video frame during the video interaction process with the large model; an instruction determination unit 802, configured to determine a data processing instruction for the target object based on the input information associated with the spatially directional action; and a data processing unit 803, configured to use the large model to process the data of the target object according to the data processing instruction to obtain a data processing result.

[0179] In some optional implementations of this embodiment, the instruction determination unit 802 is further configured to: generate a semantic description text of the target object in response to the fact that the description part of the target object in the recognition text of the input information is an implicit referential description; and determine a data processing instruction by combining the semantic description text and the recognition text.

[0180] In some optional implementations of this embodiment, the instruction determination unit 802 is further configured to: combine the semantic description text and the recognition text to determine the fused text; combine the visual data of the target object to the description part of the target object in the fused text to determine the data processing instruction.

[0181] In some optional implementations of this embodiment, the instruction determination unit 802 is further configured to: in response to the fact that the description part of the target object in the recognition text of the input information is an explicit referential description, combine the visual data of the target object with the description part of the target object in the recognition text to determine the data processing instruction.

[0182] In some optional implementations of this embodiment, the input information is associated with multiple spatially directional actions, and the instruction determination unit 802 is further configured to: for multiple spatially directional actions, in response to the implicit referential description of the target object targeted by the spatially directional action in the recognition text of the input information, generate a semantic description text of the target object; perform temporal alignment between the recognition text of the input information and the multiple spatially directional actions to determine the temporal correspondence between the multiple spatially directional actions and the recognition text; and determine the data processing instruction based on the temporal correspondence, combined with the semantic description text and the recognition text.

[0183] In some optional implementations of this embodiment, the instruction determination unit 802 is further configured to: determine the fused text based on the temporal correspondence, combining the semantic description text and the recognition text; and for multiple spatially directional actions, combine the visual data of the target object targeted by the spatially directional action into the description part of the target object in the fused text to obtain data processing instructions.

[0184] In some optional implementations of this embodiment, the data processing unit 803 is further configured to: use a large model to perform data processing on the target object according to the data processing instructions, the context of the input information, and the video frame, and obtain the data processing result.

[0185] In some optional implementations of this embodiment, the object determination unit 801 is further configured to: determine the position of the spatial directional action on the video screen during video interaction; and determine the target object located in the video screen.

[0186] In some optional implementations of this embodiment, the object determination unit 801 is further configured to: determine the target object in the video frame based on the description portion of the video recognition text that is at the same time as the spatial directional action.

[0187] In some optional implementations of this embodiment, the object determination unit 801 is further configured to: determine the target location range based on the position points of the action trajectory represented by the spatial directional action on multiple video frames during video interaction; and perform object recognition within the target location range in the video frames to determine the target object.

[0188] In some optional implementations of this embodiment, the object determination unit 801 is further configured to: during video interaction, determine the hot zone targeted by the spatial directional action based on the position points of the action trajectory represented by the spatial directional action on multiple video frames and the attribute information of the action trajectory; and determine the target location range based on the hot zone.

[0189] In some optional implementations of this embodiment, the object determination unit 801 is further configured to: determine the position point of the instantaneous action represented by the spatial directional action on the video screen during video interaction; and perform object recognition at the position point on the video screen to determine the target object.

[0190] In some optional implementations of this embodiment, the video interaction process includes a video call process, a screen sharing process, and a video dialogue process.

[0191] This embodiment provides a video interaction device based on a large model. During video interaction with the large model, the target object of the spatially directional action associated with the video frame is determined. Based on the input information associated with the spatially directional action, data processing instructions for the target object are determined. Using the large model, data processing is performed on the target object according to the data processing instructions to obtain the data processing result. This provides a human-computer interaction method that combines multimodal data of spatially directional actions, video frames, and input information. It allows users to express their intentions in an intuitive way by combining spatially directional actions and information input, such as "pointing" and "speaking," reducing communication costs during human-computer interaction and improving the efficiency of understanding and processing user intentions.

[0192] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the large-model-based video interaction method described in any of the above embodiments.

[0193] According to embodiments of this disclosure, this disclosure also provides a readable storage medium storing computer instructions that enable a computer to implement the large-model-based video interaction method described in any of the above embodiments when executed.

[0194] This disclosure provides a computer program product that, when executed by a processor, can implement the large-model-based video interaction method described in any of the above embodiments.

[0195] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0196] like Figure 9As shown, device 900 includes a computing unit 901, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 902 or a computer program loaded from storage unit 908 into random access memory (RAM) 903. RAM 903 may also store various programs and data required for the operation of device 900. The computing unit 901, ROM 902, and RAM 903 are interconnected via bus 904. Input / output (I / O) interface 905 is also connected to bus 904.

[0197] Multiple components in device 900 are connected to I / O interface 905, including: input unit 906, such as keyboard, mouse, etc.; output unit 907, such as various types of monitors, speakers, etc.; storage unit 908, such as disk, optical disk, etc.; and communication unit 909, such as network card, modem, wireless transceiver, etc. Communication unit 909 allows device 900 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0198] The computing unit 901 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 901 performs the various methods and processes described above, such as the large-model-based video interaction method. For example, in some embodiments, the large-model-based video interaction method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on device 900 via ROM 902 and / or communication unit 909. When the computer program is loaded into RAM 903 and executed by the computing unit 901, one or more steps of the large-model-based video interaction method described above can be performed. Alternatively, in other embodiments, the computing unit 901 can be configured to perform the large-model-based video interaction method by any other suitable means (e.g., by means of firmware).

[0199] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0200] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable large-scale video interactive device, such that when executed by the processor or controller, the program code enables the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0201] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0202] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0203] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0204] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, also known as cloud computing servers or cloud hosts, which are hosting products within the cloud computing service system to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services; they can also be servers for distributed systems or servers incorporating blockchain technology.

[0205] According to the technical solution of the embodiments of this disclosure, a video interaction method and apparatus based on a large model are provided. During video interaction with the large model, the target object of the spatially directional action associated with the video frame during the video interaction process is determined; based on the input information associated with the spatially directional action, a data processing instruction for the target object is determined; using the large model, data processing is performed on the target object according to the data processing instruction to obtain the data processing result. This provides a human-computer interaction method that combines multimodal data of spatially directional actions, video frames, and input information, allowing users to express their intentions in an intuitive way by combining spatially directional actions and information input, such as "pointing" and "speaking," reducing communication costs during human-computer interaction and improving the efficiency of understanding and processing user intentions.

[0206] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution provided in this disclosure can be achieved, and this is not limited herein.

[0207] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video interaction method based on a large model, comprising: During video interaction with a large model, the target object of the spatial directional action is determined based on the implementation type of the spatial directional action associated with the video frame during the video interaction. The implementation type includes a first type representing the display interface of the video frame to which the spatial directional action is directed and a second type representing the spatial directional action being located in the video frame. For the first type of spatial directional action, the absolute coordinates of the spatial directional action on the display interface are mapped to pixel coordinates relative to the video frame to determine the target object. For the second type of spatial directional action, the target object is determined based on the recognition of the spatial directional action in the video frame. Based on the input information associated with the spatial directional action, determine the data processing instructions for the target object; Using the large model, the target object is processed according to the data processing instructions to obtain the data processing result.

2. The method according to claim 1, wherein, The step of determining the data processing instructions for the target object based on the input information associated with the spatial directional action includes: In response to the fact that the description portion of the target object in the recognition text of the input information is an implicit referential description, a semantic description text of the target object is generated; The data processing instructions are determined by combining the semantic description text and the recognition text.

3. The method according to claim 2, wherein, The step of combining the semantic description text and the identified text to determine the data processing instruction includes: By combining the semantic description text and the identified text, the fused text is determined; The visual data of the target object is combined with the description portion of the target object in the fused text to determine the data processing instructions.

4. The method according to claim 1, wherein, The step of determining the data processing instructions for the target object based on the input information associated with the spatial directional action includes: In response to the fact that the description of the target object in the recognition text of the input information is an explicit referential description, the visual data of the target object is combined with the description of the target object in the recognition text to determine the data processing instruction.

5. The method according to any one of claims 1-4, wherein, The input information is associated with multiple spatial directional actions, and The step of determining the data processing instructions for the target object based on the input information associated with the spatial directional action includes: For multiple spatially directional actions, in response to the implicit referential description of the target object targeted by the spatially directional action in the recognition text of the input information, a semantic description text of the target object is generated; The recognized text of the input information and the multiple spatial directional actions are temporally aligned to determine the temporal correspondence between the multiple spatial directional actions and the recognized text; Based on the temporal correspondence, and in combination with the semantic description text and the recognition text, the data processing instruction is determined.

6. The method according to claim 5, wherein, The step of determining the data processing instruction based on the temporal correspondence, combined with the semantic description text and the identified text, includes: Based on the temporal correspondence, and combining the semantic description text and the identified text, the fused text is determined; For multiple spatial directional actions, the visual data of the target object targeted by the spatial directional action is combined with the description part of the target object in the fused text to obtain the data processing instruction.

7. The method according to claim 1, wherein, The process of using the large model and processing the target object according to the data processing instructions to obtain data processing results includes: Using the large model, the target object is processed according to the data processing instructions, the context of the input information, and the video frame to obtain the data processing result.

8. The method according to claim 1, wherein, During video interaction with a large model, the target object of the spatially directional actions associated with the video frames during the video interaction process is determined, including: During the video interaction, the position of the spatial directional action on the video screen is determined; Determine the target object at the location in the video frame.

9. The method according to claim 8, wherein, Determining the target object at the location in the video frame includes: Based on the description portion of the input information that occurs at the same time as the spatial directional action, the target object at the location in the video frame is determined.

10. The method according to claim 8, wherein, Determining the position of the spatial directional action on the video screen during the video interaction includes: During the video interaction, the target location range is determined based on the position points of the motion trajectory represented by the spatial directional action on multiple video frames; and Determining the target object at the location in the video frame includes: Object recognition is performed within the target location range in the video frame to determine the target object.

11. The method according to claim 10, wherein, During the video interaction, determining the target location range based on the position points of the motion trajectory represented by the spatial directional action on multiple video frames includes: During the video interaction, the hot zone targeted by the spatial directional action is determined based on the position points of the action trajectory represented by the spatial directional action on multiple video frames and the attribute information of the action trajectory. The target location range is determined based on the hot zone.

12. The method according to claim 8, wherein, Determining the position of the spatial directional action on the video screen during the video interaction includes: During the video interaction, the position point of the instantaneous action represented by the spatial directional action on the video frame is determined; and Determining the target object at the location in the video frame includes: Object recognition is performed at the location point in the video frame to determine the target object.

13. The method according to any one of claims 1-4 and 6-12, wherein, The video interaction process includes video call process, screen sharing process, and video dialogue process.

14. A video interactive device based on a large model, comprising: An object determination unit is configured to, during video interaction with a large model, determine the target object of a spatially directional action based on the implementation type of the spatially directional action associated with the video frame during the video interaction. The implementation type includes a first type representing the display interface of the video frame to which the spatially directional action is directed and a second type representing the spatially directional action being located within the video frame. For the first type of spatially directional action, the absolute coordinates of the spatially directional action on the display interface are mapped to pixel coordinates relative to the video frame to determine the target object. For the second type of spatially directional action, the target object is determined based on the recognition of the spatially directional action in the video frame. The instruction determination unit is configured to determine a data processing instruction for the target object based on the input information associated with the spatial directional action. The data processing unit is configured to use a large model to process the target object according to the data processing instructions and obtain the data processing result.

15. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-13.

16. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-13.

17. A computer program product comprising: A computer program that, when executed by a processor, implements the method according to any one of claims 1-13.

Images