Video language model training method and human body interaction behavior recognition method

Abstract

The invention provides a video language model training method and a human body interaction behavior recognition method, and relates to the technical field of computer vision recognition, and the method comprises the steps: obtaining a video sample and motion description text data for a human body interaction behavior in the video sample; determining a first video feature and a first object position feature corresponding to the video sample; determining visual joint features output by each layer of multi-head self-attention blocks in the L layers of multi-head self-attention blocks based on the first video features and the first object position features; and based on the action description text data and the visual joint features, determining visual representation, text representation and multi-modal representation output by the last layer of multi-modal refined learning module in the L layers of multi-modal refined learning modules, and based on the visual representation, the text representation and the multi-modal representation, updating model parameters of the video language model. And obtaining the trained target video language model. According to the invention, the accuracy of human body interaction behavior recognition can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer vision recognition technology, and in particular to a method for training a video language model and a method for recognizing human interaction behavior. Background Art

[0002] With the rapid development and widespread adoption of wearable devices like smart glasses and head-mounted cameras, coupled with breakthroughs in deep learning technology, research on first-person action recognition has become a hot topic in both academia and industry. Compared to third-person perspective, the first-person perspective can more directly record users' daily activities and capture richer information about human-object interactions. It offers unique advantages in recognizing fine-grained actions such as hand manipulation and object interaction.

[0003] However, action recognition from a first-person perspective also faces numerous technical challenges. First, there's the issue of video stability. Because the camera is fixed to the body (usually the head), the wearer's movement can cause noticeable video jitter, resulting in blurry and distorted images, which severely impacts the accuracy of feature extraction. Second, there's the limited field of view. The first-person perspective often only captures parts of the human body (such as the hands), and the center of vision shifts dynamically as the observer's attention shifts, increasing the difficulty of action recognition. Finally, there's fine-grained feature extraction. Human-object interactions from a first-person perspective often involve complex hand movements and object manipulation, requiring the model to accurately recognize and understand subtle differences in movement while also considering multi-dimensional features such as the context of object interactions, spatial positional relationships, and temporal variations. Accurately extracting these features is crucial for high-quality action recognition.

[0004] Currently, interactive behavior recognition can be performed from a first-person perspective based on the extended application of traditional video understanding models and multimodal learning methods based on video language models. However, both of the above methods have certain limitations and cannot guarantee accurate recognition of interactive behaviors. Summary of the Invention

[0005] The present invention provides a video language model training method and a human interaction behavior recognition method, which can improve the accuracy of human interaction behavior recognition.

[0006] In a first aspect, a method for training a video language model is provided. The video language model includes a video feature extraction network, an object position feature extraction network, an L-layer multi-head self-attention block, and an L-layer multimodal refinement learning module. The L-layer multi-head self-attention block and the L-layer multimodal refinement learning module are connected in a one-to-one correspondence. The method for training the video language model includes: Obtaining video samples and action description text data for human interaction behaviors in the video samples, where the action description text data includes action description text and action classification labels corresponding to the action description text; Inputting the video sample into the video feature extraction network to extract video features and obtain a first video feature; Performing object position labeling processing on the video sample to obtain object position labeling information, and inputting the object position labeling information into the object position feature extraction network to extract object features to obtain a first object position feature; Input the first video feature and the first object position feature into L layers of multi-head self-attention blocks for multi-head self-attention learning, and obtain the visual joint feature output by each layer of multi-head self-attention blocks; Performing feature separation processing on the visual joint features to obtain the second video features and second object position features after feature optimization at each layer, inputting the action description text data, the second video features, and the second object position features into the L-layer multimodal refinement learning module for feature learning, and obtaining the visual representation, text representation, and multimodal representation output by each layer of the multimodal refinement learning module; Based on the action description text data and the visual representation, text representation and multimodal representation output by the last layer of multimodal refinement learning module in the L-layer multimodal refinement learning module, the model parameters of the video language model are updated until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

[0007] In a second aspect, a method for identifying human interaction behavior is provided, comprising: Determining video data for human interaction behavior recognition; Input the video data into the trained target video language model to obtain the action classification prediction results of human interaction behavior; Input the video data into the trained target video language model to obtain the action recognition results of human interaction behaviors, including: When the video data includes a target video, the target video is input into a trained target video language model to obtain a classification confidence corresponding to each preset classification action, and the preset classification action with the highest classification confidence among multiple preset classification actions is determined as the target recognition action of the human interaction behavior; When the video data also includes multiple action description texts to be matched corresponding to the target video, the target video language model is used to determine the target comparative similarity and target matching degree between the target video and each action description text to be matched, and the classification confidence, target comparative similarity and target matching degree are weightedly calculated to obtain the final confidence degree between the target video and each action description text to be matched. The action description text to be matched with the highest final confidence degree among the multiple action description texts to be matched is determined as the action description text that matches the human body interaction behavior.

[0008] In a third aspect, a video language model training device is provided. The video language model includes a video feature extraction network, an object position feature extraction network, an L-layer multi-head self-attention block, and an L-layer multimodal refinement learning module. The L-layer multi-head self-attention block and the L-layer multimodal refinement learning module are connected in a one-to-one correspondence. The video language model training device includes: An acquisition module is used to acquire video samples and action description text data for human interaction behaviors in the video samples, where the action description text data includes action description text and action classification labels corresponding to the action description text; A first feature extraction module is used to input the video sample into the video feature extraction network to extract the video feature and obtain the first video feature; The second feature extraction module is used to perform object position labeling processing on the video sample to obtain object position labeling information, and input the object position labeling information into the object position feature extraction network to extract object features to obtain the first object position feature; A first processing module is configured to input the first video feature and the first object position feature into L layers of multi-head self-attention blocks for multi-head self-attention learning, thereby obtaining a visual joint feature output by each layer of multi-head self-attention blocks; a second processing module for performing feature separation processing on the visual joint features to obtain second video features and second object position features after feature optimization at each layer, inputting the action description text data, the second video features, and the second object position features into an L-layer multimodal refinement learning module for feature learning, and obtaining a visual representation, a text representation, and a multimodal representation output by each layer of the multimodal refinement learning module; The updating module is used to update the model parameters of the video language model based on the action description text data and the visual representation, text representation and multimodal representation output by the last layer of multimodal refinement learning module in the L-layer multimodal refinement learning module, until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

[0009] In a fourth aspect, a human interaction behavior recognition device is provided, comprising: A determination module, used to determine video data for human interaction behavior recognition; The input module is used to input video data into the trained target video language model to obtain the action classification prediction results of human interaction behavior; Input modules are specifically used for: When the video data includes a target video, the target video is input into a trained target video language model to obtain a classification confidence corresponding to each preset classification action, and the preset classification action with the highest classification confidence among multiple preset classification actions is determined as the target recognition action of the human interaction behavior; When the video data also includes multiple action description texts to be matched corresponding to the target video, the target video language model is used to determine the target comparative similarity and target matching degree between the target video and each action description text to be matched, and the classification confidence, target comparative similarity and target matching degree are weightedly calculated to obtain the final confidence degree between the target video and each action description text to be matched. The action description text to be matched with the highest final confidence degree among the multiple action description texts to be matched is determined as the action description text that matches the human body interaction behavior.

[0010] In a fifth aspect, an electronic device is provided, comprising: a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the first aspect or the second aspect.

[0011] In a sixth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method in the first aspect or the second aspect.

[0012] Through the technical solution provided by the present invention, a video language model including a video feature extraction network, an object position feature extraction network, an L-layer multi-head self-attention block and an L-layer multimodal refinement learning module can be constructed. When training the video language model, a video sample and text data describing the action of human interaction behavior in the video sample can be first obtained; then the video sample is input into the video feature extraction network for video feature extraction to obtain a first video feature; and the video sample is subjected to object position labeling processing to obtain object position labeling information, and the object position labeling information is input into the object position feature extraction network for object feature extraction to obtain a first object position feature; the first video feature and the first object position feature are further input into the L-layer multi-head self-attention block for Multi-head self-attention learning is performed to obtain the visual joint features output by each layer of multi-head self-attention blocks; and feature separation processing is performed on the visual joint features to obtain the second video features and second object position features after feature optimization of each layer. The action description text data, the second video features and the second object position features are input into the L-layer multimodal refinement learning module for feature learning to obtain the visual representation, text representation and multimodal representation output by each layer of the multimodal refinement learning module; finally, based on the action description text data and the visual representation, text representation and multimodal representation output by the last layer of the multimodal refinement learning module in the L-layer multimodal refinement learning module, the model parameters of the video language model are updated until it is determined that the preset training conditions are met, and the trained target video language model is obtained. The technical solution in this application, by designing two novel model enhancement components (i.e., object position feature extraction network and multimodal refinement learning module) in the video language model, can integrate object position annotation information as explicit information into the model feature extraction process, and guide the fine-grained alignment of visual features and text features, thereby realizing the joint fine-grained modeling of object interaction dynamic information and its semantic description, which can enhance the video language model's ability to understand actions from the first-person perspective, thereby improving the accuracy of human interactive behavior recognition.

[0013] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. Other features and advantages of the present application will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0015] Figure 1This is a schematic diagram of an example of human interaction behavior recognition based on a traditional video understanding model provided in an embodiment of the present application; Figure 2 An application scenario diagram provided for an embodiment of the present application; Figure 3 A model architecture diagram of a video language model provided in an embodiment of the present application; Figure 4 A flowchart of a method for training a video language model provided in an embodiment of the present application; Figure 5 A flowchart of a method for identifying human interaction behavior provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a video language model training device provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a human interaction behavior recognition device provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] It should be noted that the terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0018] With the rapid development and widespread adoption of wearable devices like smart glasses and head-mounted cameras, coupled with breakthroughs in deep learning technology, research on first-person action recognition has become a hot topic in both academia and industry. Compared to third-person perspective, the first-person perspective can more directly record users' daily activities and capture richer information about human-object interactions. It offers unique advantages in recognizing fine-grained actions such as hand manipulation and object interaction.

[0019] However, action recognition from a first-person perspective also faces numerous technical challenges. First, there's the issue of video stability. Because the camera is fixed to the body (usually the head), the wearer's movement can cause noticeable video jitter, resulting in blurry and distorted images, which severely impacts the accuracy of feature extraction. Second, there's the limited field of view. The first-person perspective often only captures parts of the human body (such as the hands), and the center of vision shifts dynamically as the observer's attention shifts, increasing the difficulty of action recognition. Finally, there's fine-grained feature extraction. Human-object interactions from a first-person perspective often involve complex hand movements and object manipulation, requiring the model to accurately recognize and understand subtle differences in movement while also considering multi-dimensional features such as the context of object interactions, spatial positional relationships, and temporal variations. Accurately extracting these features is crucial for high-quality action recognition.

[0020] Currently, in the field of first-person action recognition, existing technologies are mainly developed along two technical routes: The first technical approach is an extended application of traditional video understanding models. This approach builds on the proven experience of third-person video analysis and primarily employs a deep neural network architecture based on the Transformer framework. It uses visual information such as RGB video sequences and optical flow as model input. By converting action descriptions into fixed vectors of verb-noun combinations, such as "pick up-cup," various learning paradigms, such as supervised and unsupervised learning, are then employed to train the model to directly learn human-object interaction features from first-person videos.

[0021] The second technical approach is a multimodal learning method based on video language models. With the rise of large-scale pre-trained video language models, some work has begun to introduce this technology to the task of first-person action recognition. The core idea of this method is to construct a shared feature space for video and text, using the rich semantic information in the text description to guide the visual model to extract key action features in the video. Specifically, the model first extracts features from the video sequence and the corresponding text description through their respective video and text feature extraction networks. Then, a cross-modal attention mechanism or an additional self-attention fusion layer is used to align and complement the visual and language features.

[0022] However, both of the above methods have certain limitations and cannot guarantee accurate identification of interactive behaviors, specifically: While the first technical approach inherits the mature framework of traditional video analysis, it still has some flaws. First, this method overly relies on visual appearance information such as RGB video sequences and optical flow as model input, which results in model performance being severely limited by the quality of the input data. Especially in the first-person perspective, because the camera is fixed to the user, it will vibrate violently with human movement, resulting in severe blurring and distortion of the captured video. This decline in data quality directly affects the accuracy of feature extraction and reduces the accuracy of behavior recognition. Furthermore, the simplified label representation methods used in this technical approach, such as verb-noun combinations, ignore the rich semantic information contained in natural language descriptions.

[0023] For example, Figure 1 As shown in the figure, consider two similar but different actions: "pushing the model car and letting it fall" and "pushing the model car onto the box." Distinguishing between these two actions requires not only identifying the objects (hand, car, box) but also a deep understanding of their dynamic spatiotemporal relationships. In the first action, the hand can be seen to move slightly, followed by the model car falling downward independently. In the second action, the hand and model car move synchronously toward the box. These examples highlight two key insights. First, the action descriptions in the text (such as "push," "put down," and "onto") directly correspond to changes in the position of objects and the spatial relationships between objects. Second, these actions share common textual components (such as "push the model car") and can be learned from multiple action categories simultaneously.

[0024] The second technical approach typically still relies primarily on RGB video information for visual feature input, while ignoring other forms of visual information, such as object position annotations. Some research has attempted to clean video samples by retaining image pixels in the object position annotation region, directing the model to focus more on important pixels in the foreground (such as interacting objects and hands), while reducing the impact of background interference. While these works demonstrate the importance of position annotations in video understanding tasks, they still haven't fully explored the potential of object bounding box annotations as a data source. Although object bounding box annotations are relatively simple in terms of data volume, they intuitively reflect the motion trajectories and interactions of objects in spatiotemporal dimensions. Furthermore, existing technologies for visual-text feature alignment often use conventional feature space alignment methods and lack fine-grained feature refinement mechanisms. This crude approach makes it difficult for models to establish fine-grained correspondences between visual content and text descriptions. This often results in an inability to accurately identify and distinguish similar interactions, particularly in scenarios where subtle differences in movement must be understood.

[0025] In order to solve the above technical problems, the inventive concept of the present invention is to effectively integrate object position annotation information into the video language modeling framework in the behavior recognition task under the first-person perspective, so as to accurately capture and understand the subtle interaction relationships described in the action description, thereby enhancing the accuracy and robustness of the model in recognizing complex human-object interaction behaviors.

[0026] It should be understood that the technical solution of this application can be applied to the following scenarios, but is not limited to: In some implementations, Figure 2 An application scenario diagram provided in an embodiment of the present application, such as Figure 2 As shown, the application scenario may include an electronic device 210 and a network device 220. The electronic device 210 may establish a connection with the network device 220 via a wired network or a wireless network.

[0027] For example, electronic device 210 may be a desktop computer, a laptop computer, a tablet computer, etc., but is not limited thereto. Network device 220 may be a terminal device or a server, but is not limited thereto. In one embodiment of the present application, electronic device 210 may send a request message to network device 220, where the request message may be used to request obtaining a video sample and text data describing actions of human body interaction behaviors in the video sample. Furthermore, electronic device 210 may receive a response message sent by network device 220, where the response message includes the video sample and text data describing actions of human body interaction behaviors in the video sample.

[0028] also, Figure 2 An electronic device 210 and a network device 220 are exemplarily provided. In practice, other numbers of electronic devices and network devices may be included, and this application does not impose any limitation thereto.

[0029] In other possible implementations, the technical solution of the present application may also be executed by the above-mentioned electronic device 210, or the technical solution of the present application may also be executed by the above-mentioned network device 220, and the present application does not impose any restrictions on this.

[0030] After introducing the application scenarios of the embodiments of the present application, the technical solutions of the present application will be described in detail below: like Figure 3As shown, the video language model includes a video feature extraction network, an object position feature extraction network, an L-layer multi-head self-attention block and an L-layer multimodal refinement learning module. The L-layer multi-head self-attention block includes L layers of multi-head self-attention blocks connected in series (such as multi-head self-attention block 1, multi-head self-attention block 2, ..., multi-head self-attention block L), and the L-layer multimodal refinement learning module includes L layers of multimodal refinement learning modules connected in series. Each layer of multimodal refinement learning module includes a single modal refinement module and a cross-modal refinement module, that is, the L-layer multimodal refinement learning module correspondingly includes L layers of single modal refinement modules connected in series (such as single modal refinement module 1, single modal refinement module 2, ... single modal refinement module L) and L layers of cross-modal refinement modules connected in series (such as cross-modal refinement module 1, cross-modal refinement module 2, ... cross-modal refinement module L). The L-layer multi-head self-attention blocks are connected one-to-one with the L-layer multimodal refinement learning modules, that is, the multi-head self-attention block 1 is connected to the unimodal refinement module 1 and the cross-modal refinement module 1 respectively, the multi-head self-attention block 2 is connected to the unimodal refinement module 2 and the cross-modal refinement module 2 respectively, ..., the multi-head self-attention block L is connected to the unimodal refinement module L and the cross-modal refinement module L respectively.

[0031] Figure 4 A flowchart of a method for training a video language model provided in an embodiment of the present application, which can be performed as follows Figure 2 The electronic device 210 shown is executed, but not limited to this. Figure 4 As shown, the method may include the following steps: Step 410: Obtain video samples and action description text data for human interaction behaviors in the video samples. The action description text data includes action description text and action classification labels corresponding to the action description text.

[0032] In a specific application scenario, after obtaining sample video data corresponding to human interaction behavior, a segmented random sampling method can be used to divide the indefinite length video into a fixed number of video sequences, and the video sequence can be used as a video sample. Specifically, the sample video data can be divided into T segments at equal intervals. When the number of video frames is insufficient, the fixed length is maintained by padding with zero vectors. For the human interaction behavior in the video sample, corresponding action description text data is generated. The action description text data may include action description text, such as "push the model car and let it fall", and for the human interaction behavior described by the action description text, the action description text is assigned a corresponding action classification label. For the embodiment of the present disclosure, for the action description text, the action description text can first be segmented and the sentence can be divided into independent word units. Then, a text sequence containing multiple word units is constructed, and all unique word units are mapped to integer indexes. Next, the text sequence is padded and truncated to unify the length. Finally, the word units are converted into word embedding vector representations to facilitate the subsequent feature input to the video language model.

[0033] Step 420: Input the video sample into a video feature extraction network to extract video features and obtain a first video feature.

[0034] In specific application scenarios, the video feature extraction network may include a pre-trained video encoder. This video encoder has its parameters frozen during model training and inference, serving solely as a feature extractor and not participating in gradient updates to reduce computational complexity. In the disclosed embodiments, the video feature extraction network may utilize the video encoder to extract video features from video samples and use these as first video features.

[0035] Step 430: perform object position labeling on the video sample to obtain object position labeling information, and input the object position labeling information into the object position feature extraction network to extract object features to obtain the first object position feature.

[0036] The object position annotation information is used to identify and locate the rectangular box position information of the target object position, which is usually represented by four numerical values: the coordinates x, y of the center point of the rectangular box and the length and width h, w of the rectangular box.

[0037] In a specific application scenario, the object position feature extraction network includes a graph neural network, a spatial feature extraction network, and a temporal feature extraction network. The graph neural network is connected to the spatial feature extraction network, and the spatial feature extraction network is connected to the temporal feature extraction network. The object position annotation information includes the bounding box coordinates of each entity in each video frame. Accordingly, in step 430 of the embodiment, inputting the object position annotation information into the object position feature extraction network to extract object features and obtain the first object position feature can include the following steps: Step 430-1: Input the object location annotation information into the graph neural network to obtain a knowledge graph corresponding to the object location annotation information. The graph nodes in the knowledge graph represent entities, and the edges in the knowledge graph represent the mutual connections between entities.

[0038] Consider a video sample consisting of T frames, each containing N instances. The location information of these instances can be expressed as ,in, represents the bounding box coordinates of the i-th instance in the t-th frame. When there are few instances detected in each video frame, a zero vector is used as a feature. Therefore, a knowledge graph corresponding to the object location annotation information can be constructed, which is recorded as , where the graph nodes V and edges E represent the identified entities and the connections between entities, respectively.

[0039] Step 430-2: Input the knowledge graph into the spatial feature extraction network to capture the interactions between graph nodes and edges within the same video frame to obtain the global spatial features of each video frame.

[0040] Step 430 - 3 : Input the global spatial features of each video frame into the temporal feature extraction network to extract the contextual relationship between different video frames, and obtain the first object position feature corresponding to the object position annotation information.

[0041] Step 440: Input the first video feature and the first object position feature into L-layer multi-head self-attention blocks for multi-head self-attention learning to obtain the visual joint feature output by each layer of multi-head self-attention blocks.

[0042] In a specific application scenario, the L-layer multi-head self-attention block includes L layers of multi-head self-attention blocks connected in series, and each layer of multi-head self-attention block includes a multi-head self-attention layer, a normalization layer, and a feedforward neural network layer. After the embodiment steps 420 and 430, the first video feature can be obtained. and the first object position feature , where T is the number of video frames, D is the dimension of the feature vector, and and The head of the first video feature and the first object position feature are inserted with the learnable special vector. When the first video feature and the first object position feature are input into the L-layer multi-head self-attention block, the first video feature and the first object position feature after the head is inserted with the learnable special vector can be merged to obtain a visual feature vector. In order to facilitate the L-layer multi-head self-attention block to distinguish the merged visual feature vector, the frame index and feature vector category embedding can be added to the visual feature vector, which are expressed as and . The visual feature vector with the two embedded information added is then input into the L-layer multi-head self-attention block, specifically into the first-layer multi-head self-attention block of the L-layer multi-head self-attention block. In the entire technical solution, each layer of multi-head self-attention blocks in the L-layer multi-head self-attention block is responsible for processing and converting the input features, and ultimately realizes complex feature extraction and fusion functions layer by layer to serve subsequent behavior recognition tasks. The processing method of each layer is basically similar, except that the feature data at different stages are processed, and the output of the previous layer serves as the input of the next layer.

[0043] Accordingly, in step 440 of the embodiment, inputting the first video feature and the first object position feature into L-layer multi-head self-attention blocks for multi-head self-attention learning to obtain the visual joint feature output by each layer of multi-head self-attention blocks may include the following steps: Step 440-1: For any layer of the L-layer multi-head self-attention blocks, use the multi-head self-attention layer, normalization layer and feedforward neural network layer configured therein to perform multi-head self-attention processing, normalization processing and dimensionality transformation processing on the first input feature in sequence to obtain the visual joint features corresponding to the output of the multi-head self-attention block in the current layer.

[0044] In each multi-head self-attention block, the multi-head self-attention layer is the first component to take effect. It receives the first input feature. The multi-head self-attention layer divides the input feature into multiple "heads" for parallel processing. Each "head" focuses on a different aspect of the first input feature. By calculating attention weights, it can capture long-range dependencies within different subspaces of the first input feature, thereby better extracting complex relationships between features. For example, when processing the interaction between people and objects in a video, different "heads" can focus on different aspects of feature information, such as hand movements, object position changes, and body posture. After processing by the multi-head self-attention layer, the resulting features enter the normalization layer. The normalization layer normalizes the data by calculating the mean and variance of the input features and normalizing the features based on these statistics. This ensures a more stable data distribution, facilitating subsequent neural network training and learning, and avoiding training difficulties or unstable model performance caused by large data distribution variations. The normalized features then enter the feedforward neural network layer. A feedforward neural network layer typically consists of at least two neural network layers. It performs nonlinear transformations on the input features, which can lead to dimensionality changes. For example, the feature dimensions may be expanded and then contracted to a suitable dimension. This dimensionality change and nonlinear operation further enhances the model's ability to express features, enabling it to learn more complex feature patterns.

[0045] After the aforementioned multi-head self-attention processing, normalization, and dimensionality transformation, the resulting visual joint feature is generated for the corresponding output of the multi-head self-attention block in the current layer. This visual joint feature, which incorporates various information from the first input features and is processed by the multi-head self-attention block in this layer, serves as the input to the next multi-head self-attention block (if there is one), or is used in subsequent single-modal and cross-modal refinement modules for deeper feature fusion and analysis, ultimately enabling accurate recognition of complex human-object interactions from a first-person perspective.

[0046] It should be noted that when the multi-head self-attention block in the current layer is the first multi-head self-attention block in the L-layer multi-head self-attention block, the first input feature is the feature vector after adding the frame index and vector category embedding on the basis of the splicing feature corresponding to the first video feature and the first object position feature; when the multi-head self-attention block in the current layer is any multi-head self-attention block in the L-layer multi-head self-attention block except the first multi-head self-attention block, the first input feature is the visual joint feature output by the multi-head self-attention block in the previous layer corresponding to the multi-head self-attention block in the current layer. In this way, the visual joint feature output by each multi-head self-attention block in the L-layer multi-head self-attention block can be obtained (such as the visual joint feature output by multi-head self-attention block 1). , visual joint features output by multi-head self-attention block 2 , ..., the visual joint features output by the multi-head self-attention block L ).

[0047] For example, when the value of L is 4, the L-layer multi-head self-attention block may include multi-head self-attention block 1, multi-head self-attention block 2, multi-head self-attention block 3, and multi-head self-attention block 4; for multi-head self-attention block 1, its corresponding first input feature is the feature vector after adding the frame index and vector category embedding on the basis of the splicing feature corresponding to the first video feature and the first object position feature; for multi-head self-attention block 2, its corresponding first input feature is the visual joint feature output by the multi-head self-attention block in the previous layer (i.e., multi-head self-attention block 1) of the multi-head self-attention block in the current layer; for multi-head self-attention block 3, its corresponding first input feature is the visual joint feature output by the multi-head self-attention block in the previous layer (i.e., multi-head self-attention block 2) of the multi-head self-attention block in the current layer; for multi-head self-attention block 4, its corresponding first input feature is the visual joint feature output by the multi-head self-attention block in the previous layer (i.e., multi-head self-attention block 3) of the multi-head self-attention block in the current layer.

[0048] Step 450: perform feature separation processing on the visual joint features to obtain the second video features and second object position features after feature optimization of each layer, input the action description text data, the second video features and the second object position features into the L-layer multimodal refinement learning module for feature learning, and obtain the visual representation, text representation and multimodal representation output by each layer of the multimodal refinement learning module.

[0049] In a specific application scenario, an L-layer multimodal refinement learning module includes L layers of multimodal refinement learning modules. Each layer of multimodal refinement learning modules includes a unimodal refinement module and a cross-modal refinement module. That is, the L-layer multimodal refinement learning module correspondingly includes L layers of unimodal refinement modules connected in series and L layers of cross-modal refinement modules connected in series. The purpose of the L-layer multimodal refinement learning module is to further refine and fuse features to enhance the model's understanding and recognition of complex human-object interactions. Each layer performs similar operations, but targets feature data at different stages, and the output of the previous layer affects the input of the next layer.

[0050] Accordingly, step 450 of inputting the action description text data, the second video features, and the second object position features into L-layer multimodal refinement learning modules for feature learning to obtain visual representations, textual representations, and multimodal representations output by each layer of the multimodal refinement learning module may include the following steps: Step 450-1: For any layer of the L-layer multimodal refinement learning modules, use the current layer unimodal refinement module configured in the multimodal refinement learning module under the current layer to perform unimodal refinement learning on the second input feature to obtain the visual representation and text representation output by the current layer unimodal refinement module, and use the current layer cross-modal refinement module configured in the multimodal refinement learning module under the current layer to perform cross-modal refinement learning on the third input feature to obtain the multimodal representation output by the current layer cross-modal refinement module.

[0051] Among them, when the multimodal refinement learning module in the current layer is the first multimodal refinement learning module in the L-layer multimodal refinement learning module, the second input feature and the third input feature both include action description text data, and the second video feature and the second object position feature after feature separation of the target visual joint feature. The target visual joint feature is the visual joint feature output by the multi-head self-attention block corresponding to the multimodal refinement learning module in the current layer; when the multimodal refinement learning module in the current layer is any multimodal refinement learning module in the L-layer multimodal refinement learning module except the first multimodal refinement learning module, the second input feature includes the second video feature and the second object position feature after feature separation of the target visual joint feature, and the visual representation and text representation output by the current layer single-modal refinement module corresponding to the previous layer single-modal refinement module; the third input feature includes the second video feature and the second object position feature after feature separation of the target visual joint feature, and the multimodal representation output by the current layer cross-modal refinement module corresponding to the previous layer cross-modal refinement module.

[0052] For example, when the value of L is 4, the L-layer multimodal refinement learning module may include multimodal refinement learning module 1, multimodal refinement learning module 2, multimodal refinement learning module 3, and multimodal refinement learning module 4. The multimodal refinement learning module 1 includes the single-modal refinement module 1 and the cross-modal refinement module 1, the multimodal refinement learning module 2 includes the single-modal refinement module 2 and the cross-modal refinement module 2, the multimodal refinement learning module 3 includes the single-modal refinement module 3 and the cross-modal refinement module 3, and the multimodal refinement learning module 4 includes the single-modal refinement module 4 and the cross-modal refinement module 4. For the unimodal refinement module 1, its corresponding second input features include the action description text data, the target visual joint features output by the corresponding connected multi-head self-attention block 1, the second video features and the second object position features after feature separation; for the unimodal refinement module 2, its corresponding second input features include the target visual joint features output by the corresponding connected multi-head self-attention block 2, the second video features and the second object position features after feature separation, and the visual representation and text representation output by the unimodal refinement module 1; for the unimodal refinement module 3, its corresponding second input features include the target visual joint features output by the corresponding connected multi-head self-attention block 3, the second video features and the second object position features after feature separation, and the visual representation and text representation output by the unimodal refinement module 2; for the unimodal refinement module 4, its corresponding second input features include the target visual joint features output by the corresponding connected multi-head self-attention block 4, the second video features and the second object position features after feature separation, and the visual representation and text representation output by the unimodal refinement module 3. For the cross-modal refinement module 1, its corresponding third input feature includes the action description text data, and the target visual joint feature output by the corresponding connected multi-head self-attention block 1, the second video feature and the second object position feature after feature separation; for the cross-modal refinement module 2, its corresponding third input feature includes the target visual joint feature output by the corresponding connected multi-head self-attention block 2, the second video feature and the second object position feature after feature separation, and the multimodal representation output by the cross-modal refinement module 1; for the cross-modal refinement module 3, its corresponding third input feature includes the target visual joint feature output by the corresponding connected multi-head self-attention block 3, the second video feature and the second object position feature after feature separation, and the multimodal representation output by the cross-modal refinement module 2; for the cross-modal refinement module 4, its corresponding third input feature includes the target visual joint feature output by the corresponding connected multi-head self-attention block 4, the second video feature and the second object position feature after feature separation, and the multimodal representation output by the cross-modal refinement module 3.

[0053] In specific application scenarios, each unimodal refinement module consists of a first self-attention layer, a first cross-attention layer, and a feedforward network layer. Each cross-modal refinement module consists of a second self-attention layer and a second cross-attention layer. After unimodal refinement learning, the current unimodal refinement module outputs both a visual representation and a textual representation. Taking the action of a person picking up a cup in a video as an example, the visual representation highlights the details of the spatial and temporal interaction between the hand and the cup, such as changes in hand movement and the cup's position. The textual representation is a feature representation of the action description (e.g., "pick up the cup") processed by the self-attention layer and the feedforward network, which incorporates an understanding of the action's semantics. After cross-modal refinement learning, the current cross-modal refinement module outputs a multimodal representation. This multimodal representation integrates visual and textual information. For example, when determining whether a person's action in a video matches the text description "pick up the cup," the multimodal representation comprehensively considers visual features (e.g., the person's hand movement and the cup's state) and textual features (e.g., the semantics of "pick up" and "cup") to arrive at a more accurate judgment. During the training phase, the cross-modal refinement module also optimizes the model by determining whether a video-text sample pair is a positive sample (match) or a negative sample (mismatch) through binary cross-entropy loss.

[0054] Correspondingly, for step 450-1 of the embodiment, when performing unimodal refinement learning on the second input feature using the current layer unimodal refinement module configured in the multimodal refinement learning module under the current layer to obtain the visual representation and text representation output by the current layer unimodal refinement module, the embodiment steps may include: inputting the second input feature into the first self-attention layer in the current layer unimodal refinement module, performing self-attention learning processing of visual representation and text representation on the second input feature respectively, to obtain a first intermediate hidden state of visual representation and a second intermediate hidden state of text representation; inputting the first intermediate hidden state, the second video feature after feature separation of the target visual joint feature, and the second object position feature into the first cross-attention layer in the current layer unimodal refinement module for cross-attention learning processing, and calculating the visual representation output by the current layer unimodal refinement module based on the output result of the first cross-attention layer; inputting the second intermediate hidden state into the feedforward network layer to obtain the text representation output by the current layer unimodal refinement module.

[0055] Correspondingly, for step 450-1 of the embodiment, when performing cross-modal refinement learning on the third input feature using the current-layer cross-modal refinement module configured in the multimodal refinement learning module under the current layer to obtain the multimodal representation output by the current-layer cross-modal refinement module, the embodiment steps may include: inputting the third input feature into the second self-attention layer in the current-layer cross-modal refinement module, performing self-attention learning processing of the multimodal representation on the third input feature, and obtaining a third intermediate hidden state of the multimodal representation; inputting the third intermediate hidden state, the second video feature after feature separation of the target visual joint feature, and the second object position feature into the second cross-attention layer in the current-layer cross-modal refinement module for cross-attention learning processing, and calculating the multimodal representation output by the current-layer cross-modal refinement module based on the output result of the second cross-attention layer.

[0056] Step 460: Based on the action description text data and the visual representation, text representation, and multimodal representation output by the last layer of multimodal refinement learning modules in the L-layer multimodal refinement learning module, update the model parameters of the video language model until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

[0057] For the embodiment of the present disclosure, step 460 may include the following steps: Step 460 - 1 : Calculate the classification loss of the video language model based on the visual representation and action classification label output by the last layer of multimodal refinement learning modules in the L-layer multimodal refinement learning modules.

[0058] For the embodiment of the present disclosure, the visual representation output by the last layer of multimodal refinement learning module in the L-layer multimodal refinement learning module can be And the action classification label is input into the fully connected layer to obtain the classification confidence, and the classification loss of the video language model is obtained through the cross entropy loss function .

[0059] Step 460-2: Calculate the maximum contrast similarity between the video sample and the action description text based on the visual representation and text representation output by the last layer of multimodal refinement learning modules in the L-layer multimodal refinement learning modules, and calculate the contrast loss of the video language model based on the maximum contrast similarity.

[0060] For the embodiment of the present disclosure, the visual representation of the output of the last layer of multimodal refinement learning module in the L-layer multimodal refinement learning module is and text representation , which may include a visual representation and a text representation corresponding to each of the multiple video frames. For the embodiment of the present disclosure, the visual representation and the text representation can be extracted in pairs according to the order of the video frames, and the pairwise similarities between the video sample and the action description text can be multiplied accordingly, and the highest value can be selected as the similarity of the video-text pair of this video sample. The similarity of the video-text pair can then be input into the InfoNCE loss function to calculate the contrast loss of the video language model. .

[0061] Step 460-3: Calculate the matching degree between the video sample and the action description text based on the multimodal representation output by the last layer of multimodal refinement learning modules in the L-layer multimodal refinement learning modules, and calculate the matching loss of the video language model based on the matching degree.

[0062] For the embodiment of the present disclosure, the multimodal representation output by the last multimodal refinement learning module in the L-layer multimodal refinement learning module can be used. , calculate the prediction confidence in the matching task. Then input the prediction confidence into the binary cross entropy loss function to obtain the matching loss of the video language model .

[0063] Step 460 - 4 : Perform weighted calculation on the classification loss, contrast loss, and matching loss to obtain the final loss function of the video language model.

[0064] For the embodiment of the present disclosure, the classification loss can be expressed as follows: , contrast loss and matching loss By performing weighted calculation, we can calculate the final loss function L of the video language model:

[0065] in, and These are all hyperparameters used to control the contribution of different terms. and The value can be determined according to the actual application. For example, The value can be 0.4, The acceptable value is 0.6.

[0066] Step 460 - 5 : Update the model parameters of the video language model according to the final loss function until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

[0067] In the disclosed embodiments, after obtaining the final loss function, the model parameters of the video language model can be updated using the back gradient propagation algorithm. Back gradient propagation begins with the loss function, calculates the gradient of the loss with respect to each parameter in the model, and then adjusts the parameter values based on the direction and magnitude of the gradient, causing the loss function value to decrease. Preset training conditions serve as criteria for determining whether model training is complete. Common conditions include reaching a specified number of training rounds, loss function convergence (e.g., the change in the loss function value for multiple consecutive rounds is less than a certain threshold), and achieving a certain level of validation set accuracy.

[0068] By continuously updating model parameters based on the final loss function, the model gradually learns the relationship between the interactions between characters in the video and the text descriptions. When the preset training conditions are met, the training process ends, and the resulting model is the trained target video language model. This model can more accurately recognize human-object interactions from a first-person perspective, achieving higher recognition accuracy for similar actions such as "pushing a model car and causing it to fall" and "pushing the model car onto a box."

[0069] In summary, according to the training method of the video language model provided by the present invention, by designing two novel model enhancement components in the video language model (namely, the object position feature extraction network and the multimodal refinement learning module), the object position annotation information can be integrated into the model feature extraction process as explicit information, and the visual features and text features can be guided to be fine-grainedly aligned, thereby realizing the joint fine-grained modeling of the dynamic information of object interaction and its semantic description, which can enhance the video language model's ability to understand actions from the first-person perspective, thereby improving the accuracy of human interactive behavior recognition.

[0070] Figure 5 This is a flow chart of a method for identifying human interaction behavior according to an embodiment of the present invention. The method can be performed as follows: Figure 2 The electronic device 210 shown is executed, but not limited to this. Figure 5 As shown, the following steps may be included: Step 510: Determine video data for human interaction behavior recognition.

[0071] Video data can come from real-world scenes, such as footage captured by wearable devices like smart glasses and head-mounted cameras during daily activities. These devices can record a user's various activities from a first-person perspective, including interactions with objects, such as using tools, operating electronic devices, and picking up items.

[0072] In specific application scenarios, once video data is obtained, it can be preprocessed as needed. This includes converting it to the format required by the model. Operations such as random segmentation and sampling of variable-length videos, object location annotation, and text data preprocessing (word segmentation, vocabulary building, padding and truncation, and conversion to word embedding vectors) may also be performed. The processed video data contains rich visual information (such as human movements, object appearance, and location) as well as features related to the corresponding text descriptions. This data serves as the model input, providing the basis for model analysis and behavior recognition.

[0073] Step 520: Input the video data into the trained target video language model to obtain the action classification prediction result of the human interaction behavior.

[0074] The trained target video language model has already learned the relationships between human interactions and corresponding text descriptions in a large number of videos, and possesses powerful feature extraction and analysis capabilities. When video data enters the target video language model, it processes it according to the network structure and parameter settings established during training. For example, the model's feature extraction networks (such as the position feature extraction network and the video feature extraction network) first extract object position features and video appearance features in the video. These visual features are then integrated using a multi-head self-attention block, and further fused and refined using the unimodal refinement module and the cross-modal refinement module.

[0075] For the embodiment of the present disclosure, the trained target video language model can be applied to the action recognition task of human interaction behavior and the matching task of video and action description text, depending on the input type of video data.

[0076] Accordingly, in step 520 of the embodiment, when inputting the video data into the trained target video language model to obtain the action recognition result of the human interaction behavior, the following steps may be included: As a possible implementation method, when the video data includes a target video, the target video language model can be used to perform the action recognition task of human interactive behavior. Specifically, the target video can be input into the trained target video language model to obtain the classification confidence corresponding to each preset classification action. The preset classification action with the highest classification confidence among multiple preset classification actions is determined as the target recognition action of human interactive behavior.

[0077] Based on the first possible implementation method mentioned above, as another possible implementation method, when the video data also includes multiple action description texts to be matched corresponding to the target video, the target video language model can also be used to perform the matching task of the video and the action description text. Specifically, the target video language model can be used to determine the target comparison similarity and target matching degree between the target video and each action description text to be matched, and the classification confidence, target comparison similarity and target matching degree are weightedly calculated to obtain the final confidence between the target video and each action description text to be matched. The action description text to be matched with the highest final confidence among the multiple action description texts to be matched is determined as the action description text that matches the human body interaction behavior.

[0078] In summary, the human interaction behavior recognition method provided by the present invention can input video data for human interaction behavior recognition into a trained target video language model to obtain action classification prediction results for human interaction behaviors. Given that the target video language model is a fine-grained joint modeling of the dynamic information of object interactions and their semantic descriptions, it can enhance the video language model's ability to understand actions from a first-person perspective, thereby improving the accuracy of human interaction behavior recognition.

[0079] Based on the above Figure 4 A detailed description of the training method of the video language model is provided, such as Figure 6 As shown, Figure 6 This is a block diagram of a video language model training device according to an exemplary embodiment. The video language model includes a video feature extraction network, an object position feature extraction network, an L-layer multi-head self-attention block, and an L-layer multimodal refinement learning module. The L-layer multi-head self-attention block and the L-layer multimodal refinement learning module are connected one-to-one. The video language model training device. Figure 6 As shown, the device includes: An acquisition module 61 is configured to acquire video samples and action description text data for human interaction behaviors in the video samples, wherein the action description text data includes action description text and action classification labels corresponding to the action description text; A first feature extraction module 62 may be configured to input a video sample into a video feature extraction network to extract video features and obtain first video features; The second feature extraction module 63 may be used to perform object position labeling processing on the video sample to obtain object position labeling information, and input the object position labeling information into the object position feature extraction network to extract object features to obtain the first object position feature; A first processing module 64 may be configured to input the first video feature and the first object position feature into L layers of multi-head self-attention blocks for multi-head self-attention learning, thereby obtaining a visual joint feature output by each layer of multi-head self-attention blocks; A second processing module 65 may be configured to perform feature separation processing on the visual joint features to obtain second video features and second object position features after feature optimization at each layer, input the action description text data, the second video features, and the second object position features into an L-layer multimodal refinement learning module for feature learning, and obtain visual representations, textual representations, and multimodal representations output by each layer of the multimodal refinement learning module; The updating module 66 can be used to update the model parameters of the video language model based on the action description text data and the visual representation, text representation and multimodal representation output by the last layer of multimodal refinement learning module in the L-layer multimodal refinement learning module, until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

[0080] In some embodiments of the present application, the object position feature extraction network includes a graph neural network, a spatial feature extraction network and a temporal feature extraction network, and the object position annotation information includes the bounding box coordinates of each entity in each video frame; the second feature extraction module 63 can be specifically used to input the object position annotation information into the graph neural network to obtain a knowledge graph corresponding to the object position annotation information, the graph nodes in the knowledge graph represent entities, and the edges in the knowledge graph represent the mutual connections between entities; the knowledge graph is input into the spatial feature extraction network to capture the interaction between the graph nodes and edges in the same video frame to obtain the global spatial features of each video frame; the global spatial features of each video frame are input into the temporal feature extraction network to extract the contextual relationship between different video frames to obtain the first object position features corresponding to the object position annotation information.

[0081] In some embodiments of the present application, each layer of multi-head self-attention blocks includes a multi-head self-attention layer, a normalization layer, and a feedforward neural network layer; the first processing module 64 can be specifically used to perform multi-head self-attention processing, normalization processing, and dimensionality transformation processing on the first input features in sequence for any layer of multi-head self-attention blocks in the L-layer multi-head self-attention blocks, using the multi-head self-attention layer, normalization layer, and feedforward neural network layer configured therein, to obtain the visual joint features corresponding to the output of the multi-head self-attention block in the current layer; wherein, when the multi-head self-attention block in the current layer is the first layer of multi-head self-attention blocks in the L-layer multi-head self-attention blocks, the first input feature is the feature vector after adding the frame index and vector category embedding on the basis of the splicing features corresponding to the first video features and the first object position features; when the multi-head self-attention block in the current layer is any layer of multi-head self-attention blocks in the L-layer multi-head self-attention blocks except the first layer of multi-head self-attention blocks, the first input feature is the visual joint features output by the multi-head self-attention block in the previous layer corresponding to the multi-head self-attention block in the current layer.

[0082] In some embodiments of the present application, each layer of multimodal refinement learning module includes a unimodal refinement module and a cross-modal refinement module; the second processing module 65 can be specifically used to perform unimodal refinement learning on the second input feature for any layer of multimodal refinement learning modules in the L-layer multimodal refinement learning module using the current-layer unimodal refinement module configured in the multimodal refinement learning module under the current layer to obtain the visual representation and text representation output by the current-layer unimodal refinement module, and to perform cross-modal refinement learning on the third input feature using the current-layer cross-modal refinement module configured in the multimodal refinement learning module under the current layer to obtain the multimodal representation output by the current-layer cross-modal refinement module; wherein, when the multimodal refinement learning module under the current layer is the first-layer multimodal refinement learning module in the L-layer multimodal refinement learning module, the second input feature and the third input feature both include action descriptions Text data, and the second video feature and the second object position feature after feature separation of the target visual joint feature, the target visual joint feature is the visual joint feature output by the multi-head self-attention block corresponding to the multimodal refinement learning module in the current layer; when the multimodal refinement learning module in the current layer is an L-layer multimodal refinement learning module, in any layer of multimodal refinement learning module except the first layer of multimodal refinement learning module, the second input feature includes the second video feature and the second object position feature after feature separation of the target visual joint feature, and the visual representation and text representation output by the current layer single modal refinement module corresponding to the previous layer single modal refinement module, the third input feature includes the second video feature and the second object position feature after feature separation of the target visual joint feature, and the multimodal representation output by the current layer cross-modal refinement module corresponding to the previous layer cross-modal refinement module.

[0083] In some embodiments of the present application, each layer of the single-modal refinement module includes a first self-attention layer, a first cross-attention layer and a feedforward network layer, and each layer of the cross-modal refinement module includes a second self-attention layer and a second cross-attention layer; when the current layer single-modal refinement module configured in the multimodal refinement learning module under the current layer is used to perform single-modal refinement learning on the second input feature to obtain the visual representation and text representation output by the current layer single-modal refinement module, the second processing module 65 can be specifically used to input the second input feature into the first self-attention layer in the current layer single-modal refinement module, and perform single-modal refinement learning on the second input feature. The self-attention learning process of visual representation and text representation is performed respectively to obtain the first intermediate hidden state of visual representation and the second intermediate hidden state of text representation; the first intermediate hidden state, the second video feature after feature separation of the target visual joint feature and the second object position feature are input into the first cross-attention layer in the current layer single-modal refinement module for cross-attention learning process, and the visual representation output by the current layer single-modal refinement module is calculated based on the output result of the first cross-attention layer; the second intermediate hidden state is input into the feedforward network layer to obtain the text representation output by the current layer single-modal refinement module.

[0084] Correspondingly, when the current-layer cross-modal refinement module configured in the multimodal refinement learning module under the current layer is used to perform cross-modal refinement learning on the third input feature to obtain the multimodal representation output by the current-layer cross-modal refinement module, the second processing module 65 can be specifically used to input the third input feature into the second self-attention layer in the current-layer cross-modal refinement module, perform self-attention learning processing of multimodal representation on the third input feature, and obtain a third intermediate hidden state of the multimodal representation; the third intermediate hidden state, the second video feature after feature separation of the target visual joint feature, and the second object position feature are input into the second cross-attention layer in the current-layer cross-modal refinement module for cross-attention learning processing, and the multimodal representation output by the current-layer cross-modal refinement module is calculated based on the output result of the second cross-attention layer.

[0085] In some embodiments of the present application, the update module 66 can be specifically used to calculate the classification loss of the video language model based on the visual representation and action classification label output by the last layer of multimodal refinement learning modules in the L-layer multimodal refinement learning modules; calculate the maximum contrast similarity between the video sample and the action description text based on the visual representation and text representation output by the last layer of multimodal refinement learning modules in the L-layer multimodal refinement learning modules, and calculate the contrast loss of the video language model based on the maximum contrast similarity; calculate the matching degree between the video sample and the action description text based on the multimodal representation output by the last layer of multimodal refinement learning modules in the L-layer multimodal refinement learning modules, and calculate the matching loss of the video language model based on the matching degree; perform weighted calculation on the classification loss, contrast loss and matching loss to obtain the final loss function of the video language model; update the model parameters of the video language model according to the final loss function until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

[0086] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0087] The embodiments of the present application, by designing two novel model enhancement components (i.e., object position feature extraction network and multimodal refinement learning module) in the video language model, can integrate object position annotation information as explicit information into the model feature extraction process, and guide the fine-grained alignment of visual features and text features, thereby realizing the joint fine-grained modeling of object interaction dynamic information and its semantic description, which can enhance the video language model's ability to understand actions from a first-person perspective, thereby improving the accuracy of human interaction behavior recognition.

[0088] Based on the above Figure 5 The specific description of the human interaction behavior recognition method provided is as follows: after training the target video language model, the use of the target video language model can refer to the use of the normal deep learning model, and specifically provides a human interaction behavior recognition device, such as Figure 7 As shown, the device includes: A determination module 71 is configured to determine video data for human interaction behavior recognition; An input module 72 is configured to input video data into a trained target video language model to obtain action classification prediction results of human interaction behaviors; The input module 72 may be used to: When the video data includes a target video, the target video is input into a trained target video language model to obtain a classification confidence corresponding to each preset classification action, and the preset classification action with the highest classification confidence among multiple preset classification actions is determined as the target recognition action of the human interaction behavior; When the video data also includes multiple action description texts to be matched corresponding to the target video, the target video language model is used to determine the target comparative similarity and target matching degree between the target video and each action description text to be matched, and the classification confidence, target comparative similarity and target matching degree are weightedly calculated to obtain the final confidence degree between the target video and each action description text to be matched. The action description text to be matched with the highest final confidence degree among the multiple action description texts to be matched is determined as the action description text that matches the human body interaction behavior.

[0089] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0090] In the embodiments of this application, video data for human interaction behavior recognition can be input into a trained target video language model to obtain action classification prediction results for human interaction behaviors. Given that the target video language model is a fine-grained joint modeling of dynamic information about object interactions and their semantic descriptions, this can improve the video language model's ability to understand actions from a first-person perspective, thereby increasing the accuracy of human interaction behavior recognition.

[0091] The above description, in conjunction with the accompanying drawings, describes the video language model training device and human interaction behavior recognition device according to the embodiments of the present invention from the perspective of functional modules. It should be understood that the functional modules can be implemented in hardware, software, or a combination of hardware and software modules. Specifically, the various steps of the video language model training device and human interaction behavior recognition device embodiments according to the present invention can be completed by hardware integrated logic circuits and / or software instructions in a processor. The steps of the video language model training method and human interaction behavior recognition method according to the embodiments of the present invention can be directly implemented as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in combination with its hardware, completes the steps of the above-mentioned video language model training method and human interaction behavior recognition method embodiments.

[0092] Figure 8is a schematic block diagram of an electronic device 800 according to an embodiment of the present invention.

[0093] like Figure 8 As shown, the electronic device 800 may include: The memory 810 and the processor 820 are configured to store computer programs and transmit the program code to the processor 820. In other words, the processor 820 can call and execute the computer program from the memory 810 to implement the method in the embodiment of the present invention.

[0094] For example, the processor 820 may be configured to execute the above method embodiments according to instructions in the computer program.

[0095] In some embodiments of the present invention, the electronic device 820 may include but is not limited to: General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0096] In some embodiments of the present invention, the memory 810 includes but is not limited to: Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0097] In some embodiments of the present invention, the computer program may be divided into one or more modules, which are stored in the memory 810 and executed by the processor 820 to implement the method provided by the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the controller.

[0098] like Figure 8 As shown, the electronic device 800 may further include: The transceiver 830 may be connected to the processor 820 or the memory 810 .

[0099] The processor 820 may control the transceiver 830 to communicate with other devices. Specifically, it may send information or data to other devices or receive information or data sent by other devices. The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include one or more antennas.

[0100] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0101] The present invention also provides a computer storage medium having a computer program stored thereon. When the computer program is executed by a computer, the computer is enabled to perform the method of the above-mentioned method embodiment. Alternatively, one embodiment of the present invention further provides a computer program product containing instructions. When the computer is executed by the instructions, the computer is enabled to perform the method of the above-mentioned method embodiment.

[0102] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiments of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, magnetic tape), optical media (e.g., Digital Video Disc (DVD)), or semiconductor media (e.g., Solid State Drive (SSD)).

[0103] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments applied for herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0104] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0105] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0106] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for training a video language model, characterized in that: The video language model includes a video feature extraction network, an object position feature extraction network, an L-layer multi-head self-attention block, and an L-layer multimodal refinement learning module, wherein the L-layer multi-head self-attention block is connected to the L-layer multimodal refinement learning module in a one-to-one correspondence. The training method of the video language model includes: Obtaining a video sample and action description text data for human interaction behavior in the video sample, wherein the action description text data includes an action description text and an action classification label corresponding to the action description text; Inputting the video sample into the video feature extraction network to extract video features to obtain a first video feature; Performing object position labeling processing on the video sample to obtain object position labeling information, and inputting the object position labeling information into the object position feature extraction network to perform object feature extraction to obtain a first object position feature; Inputting the first video feature and the first object position feature into the L-layer multi-head self-attention block for multi-head self-attention learning, and obtaining a visual joint feature output by each layer of the multi-head self-attention block; Performing feature separation processing on the visual joint features to obtain second video features and second object position features after feature optimization of each layer, inputting the action description text data, the second video features, and the second object position features into the L-layer multimodal refinement learning module for feature learning, and obtaining a visual representation, a text representation, and a multimodal representation output by each layer of the multimodal refinement learning module; Based on the action description text data, and the visual representation, text representation, and multimodal representation output by the last layer of the multimodal refinement learning module in the L-layer multimodal refinement learning module, the model parameters of the video language model are updated until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

2. The video language model training method according to claim 1, characterized in that The object position feature extraction network includes a graph neural network, a spatial feature extraction network, and a temporal feature extraction network, and the object position annotation information includes the bounding box coordinates of each entity in each video frame; Inputting the object position annotation information into the object position feature extraction network to extract object features to obtain a first object position feature, including: Inputting the object position annotation information into the graph neural network to obtain a knowledge graph corresponding to the object position annotation information, wherein the graph nodes in the knowledge graph represent the entities, and the edges in the knowledge graph represent the mutual connections between the entities; Inputting the knowledge graph into the spatial feature extraction network to capture the interactions between graph nodes and edges within the same video frame to obtain the global spatial features of each video frame; The global spatial features of each video frame are input into the temporal feature extraction network to extract contextual relationships between different video frames, and a first object position feature corresponding to the object position annotation information is obtained.

3. The video language model training method according to claim 1, characterized in that Each layer of the multi-head self-attention block includes a multi-head self-attention layer, a normalization layer and a feedforward neural network layer; Inputting the first video feature and the first object position feature into the L-layer multi-head self-attention block for multi-head self-attention learning, and obtaining a visual joint feature output by each layer of the multi-head self-attention block, including: For any one layer of the L-layer multi-head self-attention blocks, using the multi-head self-attention layer, the normalization layer, and the feedforward neural network layer configured therein, sequentially perform multi-head self-attention processing, normalization processing, and dimensionality transformation processing on the first input feature to obtain the visual joint feature corresponding to the output of the multi-head self-attention block in the current layer; Among them, when the multi-head self-attention block under the current layer is the first layer of multi-head self-attention blocks in the L layers of multi-head self-attention blocks, the first input feature is the feature vector after adding the frame index and vector category embedding on the basis of the corresponding splicing features of the first video feature and the first object position feature; when the multi-head self-attention block under the current layer is any layer of multi-head self-attention blocks in the L layers of multi-head self-attention blocks except the first layer of multi-head self-attention blocks, the first input feature is the visual joint feature output by the multi-head self-attention block under the current layer corresponding to the previous layer of multi-head self-attention block.

4. The video language model training method according to claim 1, characterized in that The multimodal refinement learning module in each layer includes a single-modal refinement module and a cross-modal refinement module; Inputting the action description text data, the second video features, and the second object position features into the L-layer multimodal refinement learning module for feature learning, and obtaining a visual representation, a text representation, and a multimodal representation output by each layer of the multimodal refinement learning module, including: For any one of the L-layer multimodal refinement learning modules, perform unimodal refinement learning on the second input feature using the current-layer unimodal refinement module configured in the multimodal refinement learning module under the current layer, obtaining a visual representation and a textual representation output by the current-layer unimodal refinement module, and perform cross-modal refinement learning on the third input feature using the current-layer cross-modal refinement module configured in the multimodal refinement learning module under the current layer, obtaining a multimodal representation output by the current-layer cross-modal refinement module; Among them, when the multimodal refinement learning module in the current layer is the first multimodal refinement learning module in the L-layer multimodal refinement learning module, the second input feature and the third input feature both include the action description text data, and the second video feature and the second object position feature after feature separation of the target visual joint feature, and the target visual joint feature is the visual joint feature output by the multi-head self-attention block corresponding to the multimodal refinement learning module in the current layer; when the multimodal refinement learning module in the current layer is any multimodal refinement learning module in the L-layer multimodal refinement learning module except the first multimodal refinement learning module, the second input feature includes the second video feature and the second object position feature after feature separation of the target visual joint feature, and the visual representation and text representation output by the current-layer unimodal refinement module corresponding to the previous-layer unimodal refinement module; the third input feature includes the second video feature and the second object position feature after feature separation of the target visual joint feature, and the multimodal representation output by the current-layer cross-modal refinement module corresponding to the previous-layer cross-modal refinement module.

5. The video language model training method according to claim 4, characterized in that: Each layer of the single-modal refinement module includes a first self-attention layer, a first cross-attention layer and a feedforward network layer, and each layer of the cross-modal refinement module includes a second self-attention layer and a second cross-attention layer; The method of performing unimodal refinement learning on the second input feature using the current-layer unimodal refinement module configured in the current-layer multimodal refinement learning module to obtain a visual representation and a textual representation output by the current-layer unimodal refinement module includes: Inputting the second input feature into the first self-attention layer in the current-layer unimodal refinement module, performing self-attention learning of visual representation and textual representation on the second input feature, respectively, to obtain a first intermediate hidden state of the visual representation and a second intermediate hidden state of the textual representation; Inputting the first intermediate hidden state, the second video feature obtained by feature separation of the target visual joint feature, and the second object position feature into the first cross-attention layer in the current-layer unimodal refinement module for cross-attention learning processing, and calculating the visual representation output by the current-layer unimodal refinement module based on the output result of the first cross-attention layer; Inputting the second intermediate hidden state into the feedforward network layer to obtain the text representation output by the unimodal refinement module of the current layer; Performing cross-modal refinement learning on the third input feature using the current-layer cross-modal refinement module configured in the multimodal refinement learning module under the current layer to obtain a multimodal representation output by the current-layer cross-modal refinement module, including: Inputting the third input feature into the second self-attention layer in the cross-modal refinement module of the current layer, performing self-attention learning of multimodal representation on the third input feature, and obtaining a third intermediate hidden state of the multimodal representation; The third intermediate hidden state, the second video feature after feature separation of the target visual joint feature, and the second object position feature are input into the second cross-attention layer in the current layer cross-modal refinement module for cross-attention learning processing, and the multimodal representation output by the current layer cross-modal refinement module is calculated based on the output result of the second cross-attention layer.

6. The video language model training method according to claim 1, characterized in that The method includes updating the model parameters of the video language model based on the action description text data and the visual representation, text representation, and multimodal representation output by the last layer of the multimodal refinement learning module in the L-layer multimodal refinement learning module until it is determined that a preset training condition is met, thereby obtaining a trained target video language model, including: Calculating the classification loss of the video language model based on the visual representation output by the last layer of the multimodal refinement learning module in the L layers of multimodal refinement learning modules and the action classification label; Calculating the maximum contrast similarity between the video sample and the action description text based on the visual representation and text representation output by the last layer of multimodal refinement learning modules in the L layers of multimodal refinement learning modules, and calculating the contrast loss of the video language model based on the maximum contrast similarity; Calculating a matching degree between the video sample and the action description text according to the multimodal representation output by the last layer of the multimodal refinement learning module in the L layers of multimodal refinement learning modules, and calculating a matching loss of the video language model based on the matching degree; Performing weighted calculation on the classification loss, the contrast loss, and the matching loss to obtain a final loss function of the video language model; The model parameters of the video language model are updated according to the final loss function until it is determined that a preset training condition is met, thereby obtaining a trained target video language model.

7. A method for identifying human interaction behavior, characterized in that: include: Determining video data for human interaction behavior recognition; Inputting the video data into the trained target video language model to obtain action classification prediction results of human interaction behavior; The video data is input into the trained target video language model to obtain the action recognition results of human interaction behavior, including: When the video data includes a target video, the target video is input into a trained target video language model to obtain a classification confidence corresponding to each preset classification action, and a preset classification action corresponding to the highest classification confidence among the plurality of preset classification actions is determined as the target recognition action of the human interaction behavior; When the video data also includes multiple action description texts to be matched corresponding to the target video, the target video language model is used to determine the target comparative similarity and target matching degree between the target video and each of the action description texts to be matched, and the classification confidence, the target comparative similarity and the target matching degree are weightedly calculated to obtain the final confidence degree between the target video and each of the action description texts to be matched. The action description text to be matched that corresponds to the highest final confidence degree among the multiple action description texts to be matched is determined as the action description text that matches the human body interaction behavior.

8. A video language model training device, characterized in that: The video language model includes a video feature extraction network, an object position feature extraction network, an L-layer multi-head self-attention block, and an L-layer multimodal refinement learning module, wherein the L-layer multi-head self-attention block is connected to the L-layer multimodal refinement learning module in a one-to-one correspondence. The training device of the video language model includes: An acquisition module is used to acquire video samples and action description text data for human interaction behaviors in the video samples, wherein the action description text data includes action description text and action classification labels corresponding to the action description text; A first feature extraction module, configured to input the video sample into the video feature extraction network to extract video features and obtain a first video feature; a second feature extraction module, configured to perform object position labeling processing on the video sample to obtain object position labeling information, and input the object position labeling information into the object position feature extraction network to perform object feature extraction to obtain a first object position feature; A first processing module is configured to input the first video feature and the first object position feature into the L-layer multi-head self-attention block to perform multi-head self-attention learning, thereby obtaining a visual joint feature output by each layer of the multi-head self-attention block; a second processing module, configured to perform feature separation processing on the visual joint features to obtain second video features and second object position features after feature optimization at each layer, input the action description text data, the second video features, and the second object position features into the L-layer multimodal refinement learning module for feature learning, and obtain visual representation, text representation, and multimodal representation output by each layer of the multimodal refinement learning module; An updating module is used to update the model parameters of the video language model based on the action description text data and the visual representation, text representation and multimodal representation output by the last layer of multimodal refinement learning module in the L-layer multimodal refinement learning module, until it is determined that the preset training conditions are met, thereby obtaining a trained target video language model.

9. A human interaction behavior recognition device, characterized in that: include: A determination module, used to determine video data for human interaction behavior recognition; An input module, configured to input the video data into a trained target video language model to obtain an action classification prediction result of human interaction behavior; The input module is specifically used for: When the video data includes a target video, the target video is input into a trained target video language model to obtain a classification confidence corresponding to each preset classification action, and a preset classification action corresponding to the highest classification confidence among the plurality of preset classification actions is determined as the target recognition action of the human interaction behavior; When the video data also includes multiple action description texts to be matched corresponding to the target video, the target video language model is used to determine the target comparative similarity and target matching degree between the target video and each of the action description texts to be matched, and the classification confidence, the target comparative similarity and the target matching degree are weightedly calculated to obtain the final confidence degree between the target video and each of the action description texts to be matched. The action description text to be matched that corresponds to the highest final confidence degree among the multiple action description texts to be matched is determined as the action description text that matches the human body interaction behavior.

10. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the video language model training method described in any one of claims 1 to 6 or the human interaction behavior recognition method described in claim 7.

Images