Method, device and medium for recognizing human interaction based on monocular camera

Abstract

The application provides a method and device for recognizing human interaction based on a monocular camera, equipment and a medium. It relates to the technical field of computers. The method comprises: acquiring a plurality of continuous first images, the first images including a human-object interaction scene; determining first human features corresponding to human images of each first image and first object features corresponding to human-object interaction images; performing feature fusion on each first human feature and the corresponding first object feature to obtain a plurality of second human features and a plurality of second object features; determining first time sequence information between each second human feature and second time sequence information between each second object feature; determining human motion parameters according to the first time sequence information and object motion parameters according to the second time sequence information; and recognizing human-object interaction information in each first image according to the human motion parameters and the object motion parameters to improve the accuracy of human interaction recognition.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, device and medium for recognizing human interaction based on a monocular camera. Background Technology

[0002] With the rapid development of computer vision technology, Human-Object Interaction (HOI) tracking and recognition, as a core technology, is widely used in fields such as virtual reality, robotics, and gaming applications. The accuracy of its recognition directly affects the experience and actual effect of downstream applications.

[0003] In related technologies, model prediction is usually used for tracking and recognition. This involves inputting a single captured image into a preset model to predict human posture and object trajectory, and then outputting the tracking and recognition results of the human body and the object.

[0004] However, in the above methods, the model prediction process relies on human body trajectory to determine the object trajectory, which can lead to stationary objects being misjudged as moving objects. Furthermore, when an object is occluded, the trajectory can easily become distorted or even lost, resulting in low accuracy in recognizing human interactions. Summary of the Invention

[0005] This application provides a method, apparatus, device, and medium for recognizing human interaction based on a monocular camera, in order to solve the technical problem of low accuracy in recognizing human interaction.

[0006] In a first aspect, this application provides a method for recognizing human interactions based on a monocular camera, comprising:

[0007] Acquire multiple consecutive first images, each of which includes a scene of human interaction with an object. The first images are acquired using a monocular camera.

[0008] Determine the first human features corresponding to the human body image of each first image, and the first object features corresponding to the interaction image between the human body and the object.

[0009] Feature fusion is performed on each first human body feature and the corresponding first object feature to obtain multiple second human body features and multiple second object features;

[0010] Determine the first temporal information between each second human body feature and the second temporal information between each second object feature;

[0011] Based on the first time series information, determine the human motion parameters, and based on the second time series information, determine the object motion parameters;

[0012] Based on human motion parameters and object motion parameters, the interaction information between the human body and the object in each first image is identified.

[0013] In one possible implementation, determining the first human feature corresponding to the human image of each first image, and the first object feature corresponding to the interaction image between the human body and the object, includes:

[0014] Obtain the first mask corresponding to the object in each first image. The first mask is used to indicate the visible pixel area of ​​the object in the current first image.

[0015] Obtain a pre-trained recognition model, which is used to track and recognize the interaction information between the human body and objects. The recognition model is trained based on the first loss function.

[0016] By using the first feature extraction module in the recognition model, features are extracted from each human body image to obtain multiple first human body features;

[0017] By using the second feature extraction module in the recognition model, features are extracted from each interactive image and the corresponding first mask to obtain multiple first object features.

[0018] In one possible implementation, a second feature extraction module in the recognition model extracts features from each interactive image and its corresponding first mask to obtain multiple first object features, including:

[0019] Determine the object feature map corresponding to the interactive image;

[0020] Based on the object feature map, predict the object coordinate map, the two-dimensional correspondence map, and the second mask in the interactive image. The object coordinate map is used to indicate the mapping relationship between the two-dimensional pixels of the interactive image and the three-dimensional coordinates of the object. The two-dimensional correspondence map is used to indicate the coordinate information of each two-dimensional pixel. The second mask is used to indicate the visible and invisible pixel areas of the object.

[0021] The first mask is multiplied element-wise with the stitched object coordinate map and the two-dimensional corresponding map to obtain the features corresponding to the visible pixel area of ​​the object.

[0022] The convolutional and fully connected layers of the second feature extraction module process the features corresponding to the visible pixel regions of the object to obtain the first object features. The loss function of the second feature extraction module is constructed based on the object coordinate map and the second mask.

[0023] In one possible implementation, determining first temporal information between each second human body feature and second temporal information between each second object feature includes:

[0024] By recognizing the first temporal module of the model, multiple second human body features are processed temporally to obtain first temporal information, which is used to indicate the motion correlation between each second human body feature.

[0025] By recognizing the second temporal module of the model, multiple second object features are processed temporally to obtain second temporal information. The second temporal information is used to indicate the motion correlation between each second object feature. The number of network layers in the first temporal module is less than the number of network layers in the second temporal module.

[0026] In one possible implementation, determining human motion parameters based on first timing information and determining object motion parameters based on second timing information includes:

[0027] The first regression module of the recognition model performs regression prediction on the first time series information to obtain human motion parameters. The human motion parameters include human posture parameters, human shape parameters and projection parameters. The human posture parameters are used to indicate the rotational posture of each joint of the human body, the human shape parameters are used to indicate the body shape characteristics of the human body, and the projection parameters are used to indicate the projection relationship of the human body on the image plane. The first regression module is used to map the human time series features into human motion parameters.

[0028] The second regression module of the identification model performs regression prediction on the second time series information to obtain the object motion parameters. The object motion parameters include object rotation parameters and translation parameters. The object rotation parameters are used to indicate the rotation state of the object in three-dimensional space, and the translation parameters are used to indicate the position offset of the object in three-dimensional space. The second regression module is used to map the object time series features to the object motion parameters.

[0029] In one possible implementation, based on human motion parameters and object motion parameters, interaction information between the human body and the object in each first image is identified, including:

[0030] Acquire the camera parameters of the monocular camera, the first image information of each human body image, and the second image information of each interactive image;

[0031] Based on the projection parameters, camera parameters, and first image information, determine the human body's motion trajectory information;

[0032] Based on the preset 3D template of the object, translation parameters, camera parameters, and second image information, the motion trajectory information of the object is determined;

[0033] Based on human posture parameters, human shape parameters, and human motion trajectory information, the three-dimensional vertex information and joint information of the human body in each first image are reconstructed.

[0034] Based on the object's rotation parameters and motion trajectory information, determine the three-dimensional vertex information of the object in each first image;

[0035] The three-dimensional vertex information and joint information of the human body and the three-dimensional vertex information of the object are determined as the interaction information between the human body and the object.

[0036] In one possible implementation, the first loss function of the identification model is obtained based on the following steps:

[0037] Determine the second loss function and the third loss function. The second loss function is used to indicate the relevant parameters of the human body to be adjusted, and the third loss function is used to indicate the relevant parameters of the object to be adjusted.

[0038] Determine the interaction loss function between the human body and the object. The interaction loss function is used to constrain the contact relationship between the human body and the object.

[0039] The first loss function is determined based on the second loss function, the third loss function, and the interaction loss function.

[0040] Secondly, this application provides a human interaction recognition device based on a monocular camera, comprising: an acquisition module, a first determining module, a fusion module, a second determining module, a third determining module, and a recognition module, wherein...

[0041] The acquisition module is used to acquire multiple consecutive first images, which include interactive scenes between the human body and objects. The first images are acquired based on a monocular camera.

[0042] The first determining module is used to determine the first human features corresponding to the human body image of each first image, and the first object features corresponding to the human body and object interaction image.

[0043] The fusion module is used to fuse each first human body feature and the corresponding first object feature to obtain multiple second human body features and multiple second object features.

[0044] The second determining module is used to determine the first temporal information between each second human body feature and the second temporal information between each second object feature;

[0045] The third determining module is used to determine human motion parameters based on the first time-series information and to determine object motion parameters based on the second time-series information.

[0046] The recognition module is used to identify the interaction information between the human body and the object in each first image based on the human body motion parameters and the object motion parameters.

[0047] In one possible implementation, the first determining module is specifically used for:

[0048] Obtain the first mask corresponding to the object in each first image. The first mask is used to indicate the visible pixel area of ​​the object in the current first image.

[0049] Obtain a pre-trained recognition model, which is used to track and recognize the interaction information between the human body and objects. The recognition model is trained based on the first loss function.

[0050] By using the first feature extraction module in the recognition model, features are extracted from each human body image to obtain multiple first human body features;

[0051] By using the second feature extraction module in the recognition model, features are extracted from each interactive image and the corresponding first mask to obtain multiple first object features.

[0052] In one possible implementation, the first determining module is specifically used for:

[0053] By using the second feature extraction module in the recognition model, features are extracted from each interactive image and its corresponding first mask to obtain multiple first object features, including:

[0054] Determine the object feature map corresponding to the interactive image;

[0055] Based on the object feature map, predict the object coordinate map, the two-dimensional correspondence map, and the second mask in the interactive image. The object coordinate map is used to indicate the mapping relationship between the two-dimensional pixels of the interactive image and the three-dimensional coordinates of the object. The two-dimensional correspondence map is used to indicate the coordinate information of each two-dimensional pixel. The second mask is used to indicate the visible and invisible pixel areas of the object.

[0056] The first mask is multiplied element-wise with the stitched object coordinate map and the two-dimensional corresponding map to obtain the features corresponding to the visible pixel area of ​​the object.

[0057] The convolutional and fully connected layers of the second feature extraction module process the features corresponding to the visible pixel regions of the object to obtain the first object features. The loss function of the second feature extraction module is constructed based on the object coordinate map and the second mask.

[0058] In one possible implementation, the second determining module is specifically used for:

[0059] By recognizing the first temporal module of the model, multiple second human body features are processed temporally to obtain first temporal information, which is used to indicate the motion correlation between each second human body feature.

[0060] By recognizing the second temporal module of the model, multiple second object features are processed temporally to obtain second temporal information. The second temporal information is used to indicate the motion correlation between each second object feature. The number of network layers in the first temporal module is less than the number of network layers in the second temporal module.

[0061] In one possible implementation, the third determining module is specifically used for:

[0062] The first regression module of the recognition model performs regression prediction on the first time series information to obtain human motion parameters. The human motion parameters include human posture parameters, human shape parameters and projection parameters. The human posture parameters are used to indicate the rotational posture of each joint of the human body, the human shape parameters are used to indicate the body shape characteristics of the human body, and the projection parameters are used to indicate the projection relationship of the human body on the image plane. The first regression module is used to map the human time series features into human motion parameters.

[0063] The second regression module of the identification model performs regression prediction on the second time series information to obtain the object motion parameters. The object motion parameters include object rotation parameters and translation parameters. The object rotation parameters are used to indicate the rotation state of the object in three-dimensional space, and the translation parameters are used to indicate the position offset of the object in three-dimensional space. The second regression module is used to map the object time series features to the object motion parameters.

[0064] In one possible implementation, the identification module is specifically used for:

[0065] Acquire the camera parameters of the monocular camera, the first image information of each human body image, and the second image information of each interactive image;

[0066] Based on the projection parameters, camera parameters, and first image information, determine the human body's motion trajectory information;

[0067] Based on the preset 3D template of the object, translation parameters, camera parameters, and second image information, the motion trajectory information of the object is determined;

[0068] Based on human posture parameters, human shape parameters, and human motion trajectory information, the three-dimensional vertex information and joint information of the human body in each first image are reconstructed.

[0069] Based on the object's rotation parameters and motion trajectory information, determine the three-dimensional vertex information of the object in each first image;

[0070] The three-dimensional vertex information and joint information of the human body and the three-dimensional vertex information of the object are determined as the interaction information between the human body and the object.

[0071] In one possible implementation, the first loss function of the identification model is obtained based on the following steps:

[0072] Determine the second loss function and the third loss function. The second loss function is used to indicate the relevant parameters of the human body to be adjusted, and the third loss function is used to indicate the relevant parameters of the object to be adjusted.

[0073] Determine the interaction loss function between the human body and the object. The interaction loss function is used to constrain the contact relationship between the human body and the object.

[0074] The first loss function is determined based on the second loss function, the third loss function, and the interaction loss function.

[0075] Thirdly, this application provides an electronic device, including: a memory and a processor;

[0076] The memory stores the instructions that the computer executes;

[0077] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0078] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible embodiments of the first aspect.

[0079] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0080] The present application provides a method, apparatus, device, and medium for recognizing human interaction based on a monocular camera. The method acquires multiple consecutive first images, each including a scene of human interaction with an object. The first images are acquired using a monocular camera. The method determines first human features corresponding to the human images in each first image, and first object features corresponding to the human-object interaction images. It then performs feature fusion on each first human feature and its corresponding first object feature to obtain multiple second human features and multiple second object features. The method determines first temporal information between each second human feature and second temporal information between each second object feature. Based on the first temporal information, it determines human motion parameters, and based on the second temporal information, it determines object motion parameters. Finally, based on the human motion parameters and object motion parameters, it identifies the human-object interaction information in each first image. In the aforementioned method, electronic devices can extract and fuse features of the human body and objects in parallel, avoiding the interdependence between human and object trajectories and effectively preventing human estimation errors from propagating to object tracking. This solves the problem of stationary objects being misjudged as moving. Furthermore, by determining the temporal information of the human body and objects, the complete motion trajectory can be recovered using temporal continuity when the object is occluded, effectively avoiding trajectory distortion or loss caused by object occlusion and improving the accuracy and stability of human interaction recognition. In addition, this method can complete recognition based on continuous images acquired by a monocular camera, without relying on depth sensors or multi-camera systems, reducing equipment costs and deployment complexity. It is applicable to different scenarios and shooting angles, and has high practical value and scalability. Attached Figure Description

[0081] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0082] Figure 1 This application provides an illustration of the application scenario.

[0083] Figure 2 A schematic diagram illustrating the process of a human interaction recognition method based on a monocular camera provided in this application;

[0084] Figure 3 A schematic diagram illustrating the process of a feature extraction method provided in this application;

[0085] Figure 4 A schematic diagram illustrating the process of another method for recognizing human interaction based on a monocular camera provided in this application;

[0086] Figure 5 A schematic diagram illustrating the flow of the end-to-end HOI tracing method provided in this application;

[0087] Figure 6This application provides a schematic diagram of the OANet and Transformer module structures.

[0088] Figure 7 A schematic diagram of the dual-branch architecture provided for this application;

[0089] Figure 8 A schematic diagram of the structure of a human interaction recognition device based on a monocular camera provided in this application;

[0090] Figure 9 A schematic diagram of the structure of the electronic device provided in this application.

[0091] The accompanying drawings have illustrated specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to specific embodiments. Detailed Implementation

[0092] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0093] The collection, storage, use, processing, transmission, provision, and disclosure of financial data or user data involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0094] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0095] First, let me explain the terms used in this application:

[0096] Human-Object Interaction (HOI) refers to the interactive behaviors between humans and objects, including actions such as touching, manipulating, and moving. It is an important task in computer vision for studying human-environment interaction.

[0097] Monocular RGB camera: It uses only a single ordinary color camera for shooting, without relying on depth sensors or multi-camera systems, making it the most economical and convenient way to acquire visual data.

[0098] End-to-end learning: refers to deep learning methods that learn directly from the original input to the final output. The intermediate process is automatically learned by the neural network, without the need for multi-stage manual design and optimization.

[0099] SMPL (Skinned Multi-Person Linear Model): A parametric 3D human body model that uses pose and shape parameters to represent the 3D geometry of the human body.

[0100] Occlusion-Aware Network (OANet): A neural network module specifically designed to handle situations where objects are occluded. It can learn the features of the visible part of the object and recover the motion information of the occluded part.

[0101] Dual-branch architecture: refers to a network structure that uses two independent processing branches to track the human body and objects respectively, avoiding mutual interference and maintaining the independence of their respective motion characteristics.

[0102] Temporal networks: Network modules that learn features by utilizing the temporal relationships between video frames, enabling them to capture the continuity and temporal consistency of motion.

[0103] Contact loss: A loss function used to constrain the contact relationship between a person and an object, ensuring that the reconstructed human body and object maintain geometric consistency at the contact position.

[0104] Transformer: A neural network architecture based on self-attention mechanism, which can effectively model long-distance dependencies in sequence data and is used for temporal feature learning in this invention.

[0105] 3D reconstruction: The process of recovering the geometry, position, and motion trajectory of an object in three-dimensional space from a two-dimensional image or video.

[0106] Figure 1 The application scenario diagram provided in this application is as follows: Figure 1 As shown, it includes a monocular camera and electronic devices.

[0107] Users can use a monocular camera to capture video or image data of human interaction scenes and send this data to an electronic device. The electronic device can process the received video or image data to obtain the processing result, namely the tracking and recognition result of human interaction information, thereby completing the human interaction recognition operation based on the monocular camera. The electronic device can be a server, computer, or other device with data processing capabilities.

[0108] As can be seen from the above scenarios, existing technologies typically employ model prediction for tracking and recognition. This involves inputting a single captured image into a pre-defined model to predict human posture and object trajectory, and then outputting the tracking and recognition results for the human and object. However, in this method, the model prediction process relies on the human trajectory to determine the object trajectory, which can lead to stationary objects being misidentified as moving objects. Furthermore, when objects are occluded, trajectory distortion or even loss can easily occur, resulting in low accuracy in recognizing human interactions.

[0109] The method, apparatus, device, and medium for recognizing human interaction based on a monocular camera provided in this application acquire multiple consecutive first images, extract the first human features of the human image and the first object features of the interaction image from each first image, fuse the two types of features to obtain optimized features, determine the temporal information corresponding to each feature, and determine the human motion parameters and object motion parameters respectively in combination with the temporal information, and finally recognize the human interaction information. In the aforementioned method, electronic devices can extract and fuse features of the human body and objects in parallel, achieving decoupled analysis of human body and object trajectories. Adaptive correlation at the feature level avoids trajectory dependence at the output layer, improving the accuracy of motion parameter prediction. Furthermore, by separately determining the temporal information of the human body and objects, the electronic device can recover the complete motion trajectory when the object is occluded, accurately distinguishing between stationary and moving objects and effectively avoiding trajectory distortion or loss caused by occlusion. This method eliminates the need for complex multi-stage iterative modeling, further reducing computational resource consumption and improving the processing efficiency of human interaction recognition. It also eliminates the need for depth sensors or multi-camera systems, reducing hardware dependence and deployment costs, and meeting real-time or near-real-time business requirements. Moreover, this method is applicable to different indoor and outdoor scenarios and shooting angles, enhancing the versatility and scalability of the solution. While improving the accuracy of human interaction recognition, it significantly improves the stability and real-time performance of 3D reconstruction.

[0110] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0111] Figure 2 A schematic diagram illustrating the process of a human interaction recognition method based on a monocular camera provided in this application is shown below. Figure 2 As shown, the method includes:

[0112] S201, Obtain multiple consecutive first images.

[0113] The execution subject of this application embodiment can be an electronic device or a person interaction recognition device based on a monocular camera installed in the electronic device. The person interaction recognition device based on a monocular camera can be implemented by software or by a combination of software and hardware.

[0114] The first image includes a scene of human interaction with objects, and the first image is acquired based on a monocular camera.

[0115] The first image can refer to each frame in a video frame sequence continuously captured by a monocular camera. The image contains interactive behaviors between the human body and objects, such as a person picking up a cup or interacting with a chair. Multiple first images are arranged in chronological order of acquisition time to form a coherent temporal image sequence.

[0116] In this embodiment of the application, multiple consecutive first images, i.e., the input video sequence, can be denoted as... Where i represents the i-th frame image, and n represents the total number of frames in the video sequence. The dimensions represent the image dimensions, corresponding to the three RGB color channels. H and W represent the image's height and width in pixels, respectively.

[0117] A monocular camera can refer to a device that uses only a single ordinary RGB camera to acquire images, without relying on depth sensors or multi-camera systems. It has the advantages of low cost, flexible deployment, and wide applicability.

[0118] In some embodiments, an electronic device can acquire video streams containing human-object interaction scenes in real time using a monocular camera, or read pre-recorded video files from local storage or network resources; by decoding the acquired or read video stream frame by frame, a continuous sequence of image frames is obtained, i.e., multiple consecutive first images.

[0119] Each frame of the image can be preprocessed according to actual needs, including but not limited to image size normalization and color space conversion, to make it meet the input requirements of the subsequent feature extraction network.

[0120] S202. Determine the first human body feature corresponding to the human body image of each first image, and the first object feature corresponding to the interaction image between the human body and the object.

[0121] A human body image can refer to a sub-image cropped from a first image that contains only the human body region.

[0122] The first human feature can refer to the high-dimensional feature vector obtained after encoding a human image through a feature extraction network. This feature vector contains semantic information such as the human body's posture, shape, and joint position, and is used to characterize the state of the human body in a single frame image.

[0123] Interactive images can refer to sub-images of regions cropped from a first image that contain both human figures and objects.

[0124] The first object feature can refer to the high-dimensional feature vector obtained after encoding the interactive image through a feature extraction network. This feature vector contains information such as the object's appearance, shape, and position, and is used to characterize the object's state in a single frame image.

[0125] In some embodiments, the electronic device may employ a target detection algorithm to perform human body detection and object detection on the first image to obtain human body bounding boxes and object bounding boxes; crop out a sub-image containing only the human body region from the first image based on the human body bounding box, i.e., a human body image; and expand outward by a preset ratio (e.g., 1.2 times) based on the object bounding box so that the cropped area simultaneously contains both the human body and the object, i.e., an interactive image.

[0126] In this way, through the above process, the electronic device can obtain a precise image region corresponding to the human body, as well as an extended image region containing the interaction context, providing accurate input for subsequent feature extraction.

[0127] In some embodiments, an electronic device can input a human image into a pre-trained feature extraction network. Through layer-by-layer transformation using convolutional layers, pooling layers, and fully connected layers, a high-dimensional feature vector representing human semantic information is extracted, which serves as the first human feature and can be denoted as... .

[0128] Furthermore, the interactive image is input into another pre-trained feature extraction network. Through operations such as convolution, pooling, and fully connected layers, a high-dimensional feature vector representing the object's appearance and location information is extracted. This vector serves as the first object feature, which can be denoted as... .

[0129] S203. Perform feature fusion on each first human body feature and the corresponding first object feature to obtain multiple second human body features and multiple second object features.

[0130] The second human feature can refer to the enhanced human feature vector obtained by interacting and fusing information from the first human feature and the corresponding first object feature, which can be denoted as: .

[0131] This feature vector incorporates the contextual information of the object while preserving the information of the human body itself. It retains the inherent attributes of the human body's posture and shape, and incorporates the spatial positional interaction between the human body and the object.

[0132] The second object feature can refer to the enhanced object feature vector obtained after information interaction and fusion between the first object feature and the corresponding first human body feature, which can be denoted as: .

[0133] This feature vector incorporates human contextual information while preserving the object's own information. It not only retains the object's basic information such as its outline, geometry, and texture, but also associates it with the spatial interaction effects brought about by human movement.

[0134] In some embodiments, a cross-module can be designed to fuse human features and object features. The cross-module consists of a multi-layer MLP and... and As input, the fused second human feature is output. Second object features .

[0135] Specifically, for the first human body feature and the first object feature corresponding to the same first image, the electronic device concatenates the two vectors to obtain a combined feature vector that integrates human and object information; subsequently, this combined feature vector is input into the first multilayer perceptron (MLP). After linear transformation and nonlinear activation, it is mapped to a second human feature.

[0136] Furthermore, the combined feature vector is input into the second multilayer perceptron (... After linear transformation and nonlinear activation, the combined features are mapped to second object features. The first and second multilayer perceptrons are two independent sets of network parameters, responsible for extracting human-related information and object-related information from the combined features, respectively. The relevant formula can be expressed as:

[0137]

[0138] in, Indicates the first human characteristic, The first object feature is represented by [;], which indicates a feature concatenation operation. and As an independent multilayer perceptron, Indicates a second human characteristic. It indicates the characteristics of the second object.

[0139] In this way, through the above process, the electronic device achieves the establishment of adaptive association between people at the feature level, so that human body features and object features contain each other's contextual information, but maintain their own independent representation forms, thus avoiding the generation of trajectory dependency relationships at the output level.

[0140] S204. Determine the first temporal information between each second human body feature and the second temporal information between each second object feature.

[0141] In some embodiments, the electronic device may determine the first temporal information between each second human body feature and the second temporal information between each second object feature based on the following implementation: by performing temporal processing on multiple second human body features through the first temporal module of the recognition model to obtain the first temporal information; by performing temporal processing on multiple second object features through the second temporal module of the recognition model to obtain the second temporal information, wherein the number of network layers of the first temporal module is less than the number of network layers of the second temporal module.

[0142] The first temporal information is used to indicate the motion correlation between each second human body feature. This can be understood as capturing the continuity and motion trend of human posture by analyzing the changes in human body features over multiple consecutive frames in the time dimension. Examples include dynamic information such as arm swings and body displacement. In this embodiment, the first temporal information can refer to human body temporal features. .

[0143] The second temporal information is used to indicate the motion correlation between various second object features. This can be understood as capturing the motion trajectory, rotation changes, and motion continuity under occlusion by analyzing the temporal variation patterns of object features across multiple consecutive frames. For example, it includes inferring the object's position before and after occlusion, and maintaining its motion direction. In this embodiment, the second temporal information can refer to the object's temporal features. .

[0144] The recognition model can refer to a pre-trained neural network model used for end-to-end tracking and recognition of human interactions. This model includes multiple sub-network components such as feature extraction module, time series module, and regression module.

[0145] The first temporal module can refer to the temporal network in the recognition model used to process human features. Its core is a Transformer architecture based on a multi-head self-attention mechanism, which is used to capture the motion dependencies of the human body in the time dimension.

[0146] The second temporal module can refer to the temporal network in the recognition model used to process object features. Its core is also a Transformer architecture based on a multi-head self-attention mechanism. However, due to the higher complexity of object motion and the need to handle occlusion, its network depth is greater than that of the first temporal module.

[0147] In some embodiments, the electronic device can fuse the single-frame feature sequence. and Temporal feature learning is performed by inputting human body Transformer (first temporal module) and object Transformer (second temporal module) respectively, to obtain first temporal information. Second timing information .

[0148] In some embodiments, the electronic device can arrange multiple consecutive frames of second human features in chronological order to form a human feature sequence, denoted as... Where T is the temporal window length, and T=64 is set during training; the sequence is input into the first temporal module, which adopts the Transformer architecture. First, the positional information of each frame in the sequence is encoded through Rotation Position Encoding (RoPE), enabling the network to perceive the relative positional relationship between frames; then, the correlation weights between the features of each frame are calculated through a multi-head self-attention mechanism, directly introducing positional information into the attention calculation. The relevant formula can be expressed as:

[0149]

[0150] in, The query vector after applying the rotation transformation. For key vectors, For each dimension of attention head, The scaling factor is used to prevent the softmax gradient from saturating due to excessively large dot product results. The softmax operation normalizes the attention score into a probability distribution, and the final output is the weighted sum of all value vectors V. RoPE can better capture relative positional relationships, is more robust to changes in sequence length, and outperforms traditional absolute position encoding for long sequences.

[0151] Through a multi-head mechanism, eight attention heads are used for parallel computation, each with a dimension of 512 / 8=64, enabling the model to learn different temporal dependencies in different representation subspaces. A gating mechanism then adaptively adjusts the fusion ratio of temporal information to avoid interference from invalid temporal information, ultimately outputting a 512-dimensional human enhancement feature sequence fused with temporal information, i.e., the first temporal information. The relevant formula can be expressed as:

[0152]

[0153] in, For the Sigmoid function, For multi-head attention output, As input features, For element-wise multiplication, the gating mechanism enables the network to adaptively determine the proportion of temporal information used based on the characteristics of the current frame.

[0154] In some embodiments, each Transformer layer comprises two sub-layers: a multi-head self-attention sub-layer and a feedforward network sub-layer. Each sub-layer employs a structure with residual connections followed by normalized layers to ensure the stability of network training and the smoothness of gradient propagation. The relevant formula can be expressed as:

[0155]

[0156] in, This represents the input features of the l-th sub-layer. This represents the transformation function of the current sublayer (i.e., multi-head self-attention mechanism or feedforward network). This indicates a residual connection, which directly adds the sub-layer input to the sub-layer output, allowing the gradient to bypass the sub-layer and propagate directly in the back, effectively alleviating the gradient vanishing problem in deep networks. The representation layer normalization operation normalizes the feature vectors to make their mean 0 and variance 1, thereby accelerating model convergence and enhancing training stability.

[0157] Specifically, in each Transformer layer, the input features are first... The intermediate features are obtained by inputting the multi-head self-attention sub-layer, passing through residual connections and layer normalization; then the intermediate features are input into the feedforward network sub-layer, and after passing through residual connections and layer normalization again, the output features of the layer are obtained.

[0158] In some embodiments, the electronic device arranges the second object features of multiple consecutive frames in chronological order to form an object feature sequence, denoted as . Where T is the temporal window length; this sequence is input into the second temporal module, which adopts the same Transformer architecture as the first temporal module. It also encodes the relative positions between frames through rotational position encoding, captures the motion dependencies between frames through a multi-head self-attention mechanism, and adaptively fuses temporal information through a gating mechanism; the output is a 512-dimensional object augmentation feature sequence fused with temporal information, i.e., the second temporal information. The second temporal module, through its deeper network structure, can better model the complex motion patterns of objects under occlusion, and infers the motion state of occluded frames using information from visible frames.

[0159] In some embodiments, the number of network layers in the first timing module is less than the number of network layers in the second timing module. The specific settings are as follows: Since human movement is constrained by the skeletal structure, the movement pattern is relatively simple and has a periodic regularity. Therefore, the first timing module adopts a shallower number of network layers, for example, a 6-layer Transformer structure with approximately 30M parameters, which can effectively capture the temporal dependencies of the human body by adapting to the relatively simple temporal patterns of human movement.

[0160] Objects have a higher degree of freedom of motion and can undergo complex movements such as rotation, throwing, and sliding. They are often occluded by human bodies during interaction, requiring deeper networks to model long-distance temporal dependencies and the transition rules between occlusion and visibility states. Therefore, the second temporal module adopts a deeper number of network layers, such as a 12-layer Transformer structure with approximately 60M parameters, to adapt to the high complexity of object tracking, especially motion recovery under occlusion states.

[0161] In this way, through this differentiated network depth design, while ensuring the accuracy of human body temporal modeling, the modeling ability for complex object motion and occlusion recovery is enhanced, thereby improving the overall recognition accuracy and stability.

[0162] It can be seen that the temporal modeling module (the first temporal module and the second temporal module) can make full use of the temporal continuity of the video, and obtain motion cues from adjacent frames when single-frame information is insufficient (e.g., object occlusion), thereby achieving stable trajectory estimation.

[0163] S205. Determine the human motion parameters based on the first time sequence information, and determine the object motion parameters based on the second time sequence information.

[0164] Human motion parameters refer to the set of parameters used to describe the posture, body shape, and projection relationship of the human body in three-dimensional space.

[0165] Object motion parameters can refer to the set of parameters used to describe the rotational state and positional offset of an object in three-dimensional space.

[0166] In some embodiments, the electronic device may determine human motion parameters based on first time-series information and object motion parameters based on second time-series information in the following manner: by performing regression prediction on the first time-series information through the first regression module of the recognition model to obtain human motion parameters; and by performing regression prediction on the second time-series information through the second regression module of the recognition model to obtain object motion parameters.

[0167] Among them, human motion parameters include human posture parameters, human shape parameters, and projection parameters.

[0168] Human pose parameters are used to indicate the rotational posture of each joint in the human body. They can be understood as describing the rotational angle of each joint in three-dimensional space, such as the angle of an arm raised or the degree of knee flexion. These parameters drive a parametric human model to present various poses and movements, and can be denoted as... .

[0169] Human body shape parameters are used to indicate the body shape characteristics of a person. They can be understood as low-dimensional vectors (usually 10-dimensional) describing differences in body shape such as height, weight, arm length, and leg length. These parameters control the generation of personalized human bodies with different body shapes using a parameterized human body model. In this embodiment, they can be denoted as... .

[0170] Projection parameters indicate the projection relationship of the human body onto the image plane. They can be understood as parameters describing the positional offset and scale of a 3D human body projected onto a 2D image plane, including the horizontal translation. Vertical translation amount and global scaling factor This is used to align the reconstructed 3D human body with the human body position in the image. In this embodiment, the weak perspective projection parameter can be denoted as... .

[0171] The first regression module is used to map human temporal features into human motion parameters. That is, it can be understood as transforming the first temporal information (temporally enhanced human features) into human pose parameters, human shape parameters, and projection parameters through a multilayer perceptron network.

[0172] The motion parameters of an object include its rotation parameters and translation parameters.

[0173] Object rotation parameters are used to indicate the rotational state of an object in three-dimensional space. They can be understood as describing the rotation angles of the object around the three coordinate axes. A 6-dimensional continuous rotation is used to represent the global rotation matrix of the object in SO(3) space to ensure the continuity of the rotation space and the stability of the regression. This can be denoted as... .

[0174] Translation parameters are used to indicate the positional offset of an object in three-dimensional space. That is, they can be understood as describing the three-dimensional spatial position of the object in the camera coordinate system. In this embodiment, the scale-invariant translation parameter can be denoted as... .

[0175] The second regression module is used to map the temporal features of an object to its motion parameters. This can be understood as transforming the second temporal information (temporally enhanced object features) into rotation and translation parameters using a multilayer perceptron network.

[0176] In some embodiments, the electronic device inputs the first time-series information output by the first time-series module into the first regression module. The first regression module adopts a multilayer perceptron (MLP) architecture, which includes multiple fully connected layers and nonlinear activation functions. Through the layer-by-layer transformation of the multilayer perceptron, the 512-dimensional human temporal features are mapped to human pose parameters. Human body shape parameters and projection parameters The first regression module learns the nonlinear mapping relationship from temporal features to motion parameters through end-to-end training, and outputs human motion parameters for subsequent 3D reconstruction.

[0177] In some embodiments, the electronic device inputs the second timing information output by the second timing module into the second regression module, which also adopts a multilayer perceptron (MLP) architecture; through the layer-by-layer transformation of the multilayer perceptron, the 512-dimensional object temporal features are mapped to object rotation parameters. Translation parameters The second regression module learns the nonlinear mapping relationship from temporal features to object motion parameters through end-to-end training, and outputs object motion parameters for subsequent 3D reconstruction.

[0178] S206. Based on human motion parameters and object motion parameters, identify the interaction information between the human body and the object in each first image.

[0179] Interactive information can refer to the complete information about the interaction between the human body and objects in three-dimensional space in each first image.

[0180] In some embodiments, the electronic device can identify the interaction information between the human body and the object in each first image based on the following implementation: acquiring camera parameters of a monocular camera, first image information of each human body image, and second image information of each interaction image; determining the motion trajectory information of the human body based on projection parameters, camera parameters, and first image information; determining the motion trajectory information of the object based on a preset three-dimensional object template, translation parameters, camera parameters, and second image information; reconstructing the three-dimensional vertex information and joint information of the human body in each first image based on human posture parameters, human shape parameters, and human motion trajectory information; determining the three-dimensional vertex information of the object in each first image based on object rotation parameters and object motion trajectory information; and determining the three-dimensional vertex information of the human body, joint information, and object as the interaction information between the human body and the object.

[0181] Camera parameters can refer to the internal parameters of a monocular camera, including the focal length f. These parameters describe the projection relationship between a point in three-dimensional space and the two-dimensional image plane, and are the basis for trajectory calculation and three-dimensional reconstruction.

[0182] The first image information can refer to the geometric information obtained from the human image cropping process, including image resolution, the size b of the cropping bounding box, and the center coordinates of a single frame image. and cropping image center It is used to convert weak perspective projection parameters into absolute three-dimensional trajectories.

[0183] The second image information can refer to the geometric information obtained from the interactive image cropping process, including the width of the cropped image. ,high Center coordinates of a single frame image Crop the image center And the resize ratio of the cropped image, used to convert scale-invariant translation parameters into absolute 3D trajectories.

[0184] Human motion trajectory information refers to the three-dimensional spatial position of the human body in the camera coordinate system, representing the three-dimensional coordinates of the human body's root node, which can be denoted as... .

[0185] A preset 3D object template can refer to a pre-defined 3D mesh model of an object, such as a CAD model of a cup or a 3D scanned model of a chair. This template defines the geometry of the object, and through rigid body transformation, it can be placed in the correct position in 3D space.

[0186] The trajectory information of an object can refer to the object's three-dimensional spatial position in the camera coordinate system, representing the three-dimensional coordinates of the object's center point, which can be denoted as... .

[0187] The three-dimensional vertex information of the human body can refer to the vertex coordinates of the human body surface mesh generated by a parametric human body model (e.g., the SMPL-H model), which constitute the complete geometric surface of the human body and can be denoted as... .

[0188] Joint information refers to the coordinates of the human skeleton joints generated through a parametric human model. It includes the three-dimensional positions of 24 key joints, used to represent the skeletal structure of the human body, and can be denoted as... .

[0189] The 3D vertex information of an object can refer to the vertex coordinates of the object's surface mesh obtained through rigid body transformation using the object's 3D template. The number of vertices depends on the complexity of the object template.

[0190] In some embodiments, when calculating the three-dimensional trajectory information of the human body, the electronic device may first calculate the depth coordinates of the human body in the camera coordinate system. The relevant formula can be expressed as:

[0191]

[0192] Where f is the camera focal length, s is the predicted scale factor, and b is the size of the cropping bounding box, this formula is based on a weak perspective projection model, which inversely calculates the scale of the human body in the image into its depth in three-dimensional space.

[0193] Then, the lateral coordinates of the human body in the camera coordinate system are calculated. and vertical coordinates The relevant formula can be expressed as:

[0194]

[0195] in, and For the predicted translation parameters, To crop the image center, To obtain the complete image center, this formula converts the translation in the cropped image coordinate system into absolute coordinates in the camera coordinate system. Through the above calculations, the three-dimensional motion trajectory information of the human body in the camera coordinate system is obtained.

[0196] In some embodiments, the depth coordinates of the object in the camera coordinate system are calculated. The relevant formula can be expressed as:

[0197]

[0198] in, The depth factor is the predicted depth factor, and the resize ratio is the scaling ratio of the cropped image. This formula directly regresses the depth, avoiding the instability problem of depth estimation caused by relying on 2D detection boxes.

[0199] Then, the lateral coordinates of the object in the camera coordinate system are calculated. and vertical coordinates The relevant formula can be expressed as:

[0200]

[0201] in, The pixel focal length of a monocular camera. and Convert the normalized offset to pixel coordinates in the cropped image, and add the cropped image center. This yields the pixel coordinates in the complete image, which are then subtracted from the center of the complete image. The offset relative to the image center is obtained, and then the three-dimensional spatial coordinates are solved by inversely using the perspective projection formula; through the above calculations, the three-dimensional motion trajectory information of the object in the camera coordinate system is obtained. .

[0202] In some embodiments, during the three-dimensional reconstruction of the human body, the electronic device acquires the human posture parameters output by the first regression module. and human body shape parameters and the calculated human motion trajectory information The pose and shape parameters are input into the parametric human body model (SMPL-H model) to generate vertices and joints of the human body in a standard pose. Then, the generated human body mesh and joints are translated and transformed according to the human body motion trajectory information to obtain the final human body 3D vertex information. and key point information This ensures that the reconstructed human body is positioned correctly in the camera coordinate system.

[0203] In some embodiments, during 3D reconstruction of an object, the electronic device acquires the object rotation parameters output by the second regression module. (Using 6-dimensional continuous rotation representation), and the calculated object motion trajectory information. Obtain a preset 3D template of the object, which defines the vertex coordinates of the 3D mesh of the object in its standard pose; apply a rigid body transformation to the 3D template of the object, that is, first perform a rotation transformation through rotation parameters, and then use motion trajectory information. Perform a translation transformation to obtain the 3D vertex information of the object in the camera coordinate system. .

[0204] The method, apparatus, device, and medium for recognizing human interaction based on a monocular camera provided in this application acquire multiple consecutive first images, extract the first human features of the human image and the first object features of the interaction image from each first image, fuse the two types of features to obtain optimized features, determine the temporal information corresponding to each feature, and determine the human motion parameters and object motion parameters respectively in combination with the temporal information, and finally recognize the human interaction information. In the above method, electronic devices can extract and fuse features of the human body and objects in parallel, achieving decoupled analysis of human body trajectory and object trajectory. Based on adaptive association at the feature level, the trajectory dependency of the output layer is avoided, improving the accuracy of motion parameter prediction. Furthermore, by determining the temporal information of the human body and objects separately, the electronic devices can recover the complete motion trajectory by utilizing temporal continuity when the object is occluded. This can accurately distinguish between stationary and moving objects, effectively avoiding trajectory distortion or loss caused by occlusion. Moreover, this method does not require complex multi-stage iterative modeling, further reducing computational resource consumption and improving the processing efficiency of human interaction recognition. It also does not rely on depth sensors or multi-camera systems, reducing the dependence on hardware devices and deployment costs, and improving the processing efficiency of human interaction recognition to meet real-time or near-real-time business requirements.

[0205] Based on any of the above embodiments, the method for determining the first human body feature and the first object feature in the above method will be described in detail.

[0206] Figure 3A schematic diagram illustrating the process of a feature extraction method provided in this application, as shown below. Figure 3 As shown, the method includes:

[0207] S301. Obtain the first mask corresponding to the object in each first image.

[0208] The first mask is used to indicate the visible pixel region of an object in the current first image. In this embodiment, the first mask can be automatically generated by an instance segmentation model (e.g., Mask R-CNN) or provided by user annotation.

[0209] The first mask is a binary image, where pixels with a value of 1 represent the visible part of the object, and pixels with a value of 0 represent the background or an occluded area. The purpose of the first mask is to guide the network to focus only on the visible area of ​​the object during feature extraction, avoiding noise interference from occluded areas.

[0210] S302. Obtain the pre-trained recognition model.

[0211] The recognition model is used to track and identify the interaction information between the human body and objects. The recognition model is trained based on the first loss function.

[0212] The first loss function can refer to the total loss function used for end-to-end training of the entire recognition model. This loss function enables the model to learn a nonlinear mapping from images to three-dimensional motion parameters by jointly optimizing multiple sub-objectives such as human reconstruction, object reconstruction, and contact constraints.

[0213] In some embodiments, the first loss function of the recognition model is obtained based on the following steps: determining a second loss function and a third loss function; determining an interaction loss function between the human body and the object; and determining the first loss function based on the second loss function, the third loss function, and the interaction loss function.

[0214] The second loss function is used to indicate the adjustment of relevant parameters of the human body, and can be denoted as: .

[0215] The third loss function is used to indicate the relevant parameters of the object being adjusted, and can be denoted as: .

[0216] The interaction loss function is used to constrain the contact relationship between the human body and objects, and can be denoted as: .

[0217] In some embodiments, the second loss function can be constructed by building a human-related loss function. Using L2 / L1 loss to constrain multi-dimensional human body parameters, it is robust to outliers. The relevant formula can be expressed as:

[0218]

[0219] in, The L2 loss represents the attitude parameters, constraining the difference between the predicted attitude and the true attitude; its coefficients... , The L2 loss represents the shape parameter, constraining the difference between the predicted shape and the true shape; its coefficient is... , The L2 loss, representing the vertices of the 3D human body mesh, directly constrains the geometric accuracy of the 3D reconstruction. Its coefficient... , The L2 loss represents the reprojection coordinates of 2D joints, which is used to supervise the accuracy of 3D human projection using 2D joints detected in the image. Its coefficient... The L1 loss, representing the three-dimensional trajectory of the human body, is robust to outliers, and its coefficient... This ensures the priority of trajectory estimation.

[0220] For the third loss function, an object-related loss function can be constructed. Using L2 / L1 loss to constrain object rotation and trajectory, balancing accuracy and robustness, the relevant formula can be expressed as:

[0221]

[0222] in, The L2 loss represents the object rotation parameters, constraining the difference between the predicted rotation and the true value; its coefficients... , The L1 loss represents the scale-invariant translation parameter, and its coefficient is... , The L1 loss represents the 3D trajectory of an object, and its coefficient is... , This represents the L1 loss of the downsampled object mesh vertices (typically 64 key vertices), controlling computational cost while preserving shape constraints. Its coefficients... .

[0223] For the interaction loss function, a contact loss function can be constructed. A distance-weighted loss is used to constrain the geometric consistency of human-object contact, avoiding penetration or excessive separation. Specifically, the shortest distance di from each vertex of the human mesh to the surface of the object mesh is calculated; the distance threshold k corresponding to the top 20% of the closest vertices is determined; and the contact weight wi of each vertex is calculated. The relevant formula can be expressed as:

[0224]

[0225] in, Indicates the contact weight of the character. This represents the shortest distance from the i-th human vertex to the surface of the object. This represents a small constant to prevent division by zero. This represents the distance threshold, set as the distance value to the nearest 20% of vertices. Indicates taking a positive value. To indicate the degree of nonlinearity of the control weights, let... This causes the weight to increase sharply as the distance decreases, assigning significant weight only to vertices that are actually in contact. Let the number of vertices in the SMPL-H model be denoted as . This loss ensures that the human body and the object remain geometrically aligned in the contact area, preventing penetration or excessive separation.

[0226] Thus, the electronic device performs a weighted sum of the three loss functions to obtain the first loss function, which can be expressed as follows:

[0227]

[0228] In some embodiments, the recognition model is trained on a single NVIDIA RTX 3090 GPU, and its core configuration includes a time window length of T=64 frames; a batch size of 8; and an initial learning rate of... The learning rate decayed by 0.1 every 30 epochs; the total number of training epochs was 100; the optimizer was Adam; data augmentations included random cropping, color jittering, and horizontal flipping. Multiple datasets were used for training; for example, the InterCap dataset took 8 hours (100 epochs) to train, and the BEHAVE dataset took 30 hours (100 epochs) to train. This end-to-end training enabled the network to automatically learn the implicit constraints between people, achieving efficient training and use while ensuring the accuracy of various metrics.

[0229] In summary, by employing pre-training, batch processing, and a single forward propagation design, efficient model prediction can be achieved, meeting the requirements of real-time applications. Specifically, for pre-training feature extraction, image-level modules (HMR2.0 and OANet) can be trained first, and their weights frozen as feature extractors, allowing subsequent temporal networks to use longer temporal windows without significantly increasing computational cost. For batch processing, a training configuration with a temporal window length of 64 and a batch size of 8 can be used, improving training and recognition efficiency while maintaining accuracy. For single forward propagation, the end-to-end design ensures that the entire tracking process requires only one forward propagation, avoiding repeated iterations of multi-stage optimization. Thus, through the design of the network's loss function and training strategy, combined with end-to-end joint optimization, high-precision and highly robust HOI tracking can be achieved.

[0230] S303. By using the first feature extraction module in the recognition model, feature extraction is performed on each human body image to obtain multiple first human body features.

[0231] The first feature extraction module can refer to a pre-trained human feature extraction network, specifically the HMR2.0 (HumanMesh Recovery 2.0) model. This model is pre-trained on a large-scale human dataset and has robust human geometric and pose feature extraction capabilities.

[0232] In some embodiments, the electronic device can input a human image cropped from a first image into a first feature extraction module, which adopts an HMR2.0 model architecture. This model encodes the input image using a convolutional neural network to extract high-level visual features. Subsequently, it uses an iterative regression mechanism to progressively optimize the prediction of human pose and shape. During this process, feature vectors from the intermediate layers of the model are extracted as human feature representations. Finally, the first human feature, i.e., a 1024-dimensional high-dimensional feature vector, is output. This feature vector contains semantic information such as the human body's posture, shape, and joint positions. Through the above process, the electronic device processes the human body image of each first image frame by frame to obtain the corresponding human body feature sequence.

[0233] S304. By using the second feature extraction module in the recognition model, feature extraction is performed on each interactive image and the corresponding first mask to obtain multiple first object features.

[0234] The second feature extraction module can refer to an occlusion-aware network (OANet) designed with ResNet-34 as the backbone network. That is, the occlusion-aware network OANet designed in this application can be used as a feature extractor. The cropped image containing human body and object (interactive image) and the object visibility mask (first mask) are input into OANet to obtain 1024-dimensional object features. .

[0235] In some embodiments, the electronic device can extract features from each interactive image and the corresponding first mask through the second feature extraction module in the recognition model to obtain multiple first object features based on the following implementation: determining the object feature map corresponding to the interactive image; predicting the object coordinate map, two-dimensional correspondence map, and second mask in the interactive image based on the object feature map; performing element-wise multiplication of the first mask with the stitched object coordinate map and two-dimensional correspondence map to obtain the features corresponding to the visible pixel area of ​​the object; processing the features corresponding to the visible pixel area of ​​the object through the convolutional layer and fully connected layer of the second feature extraction module to obtain the first object features, wherein the loss function of the second feature extraction module is constructed based on the object coordinate map and the second mask.

[0236] Among them, the object coordinate map is used to indicate the mapping relationship between the two-dimensional pixels of the interactive image and the three-dimensional coordinates of the object.

[0237] A two-dimensional correspondence diagram is used to indicate the coordinate information of each two-dimensional pixel.

[0238] The second mask is used to indicate the visible and invisible pixel areas of an object.

[0239] In some embodiments, when extracting the first object features, it is necessary to use an RGB image containing both the human body and the object, i.e., an interactive image. and the first mask Input the backbone network of the second feature extraction module.

[0240] The backbone network adopts the ResNet-34 architecture, which is pre-trained on the ImageNet dataset and possesses powerful visual feature representation capabilities. Through layer-by-layer processing via convolutional layers, pooling layers, and residual connections of ResNet-34, object feature maps are output. , where 512 is the number of channels and 12×12 is the spatial resolution of the feature map, which contains high-level semantic information of the image.

[0241] Subsequently, the electronic device predicts three intermediate outputs based on the object feature map, specifically the object coordinate map, the two-dimensional correspondence map, and the second mask. In this way, by designing the three intermediate outputs as supervision signals, the network learns occlusion features and 3D geometric information.

[0242] Among them, the object coordinate diagram This is used to indicate the mapping relationship between two-dimensional pixels in an interactive image and three-dimensional coordinates of an object. That is, for each pixel position in the image, the three-dimensional coordinates of the corresponding object surface point in the object's own coordinate system are predicted, and the correspondence between 2D pixels and 3D surface points is established. During training, a ground truth coordinate map is generated using known object poses and rendering techniques.

[0243] Two-dimensional correspondence diagram Used to indicate the coordinate information of each two-dimensional pixel, that is, the normalized two-dimensional coordinates of each pixel on the object template, to help the network learn a stable 2D-3D mapping relationship.

[0244] Second mask Used to indicate the complete outline of an object, including visible and invisible pixel areas (i.e. occluded parts), the network is forced to learn the shape prior of the object by predicting the complete mask. Even if part of the object is occluded, the shape of the occluded part can be predicted. The complete rendering mask of the object is used as the ground truth during training.

[0245] Next, the electronic device targets the first mask. Coordinate diagram of the stitched object Corresponding diagram in two dimensions Element-wise multiplication is performed. Specifically, the object coordinate map and its corresponding 2D map are first concatenated along the channel dimension; then, the first mask is multiplied element-wise by the concatenated feature map. The relevant formula can be expressed as:

[0246]

[0247] in, This represents element-wise multiplication. This indicates splicing. Since the area with a value of 1 in the first mask represents the visible pixels of the object, and the area with a value of 0 represents the occluded or background area, after element-wise multiplication, the features of the occluded area are set to zero, and only the features of the visible area of ​​the object are retained, effectively avoiding noise interference from the occluded area.

[0248] Then, the electronic device will display the feature map after masking. Local spatial features are extracted through convolutional layers. These features are then fused with the object feature map from the backbone network, resulting in final features that contain both the semantic information of the original image and the geometric correspondence information of the visible regions. Finally, a fully connected layer (linear transformation) maps the fused features to a 1024-dimensional first object feature. The relevant formula can be expressed as:

[0249]

[0250] Here, Conv represents the convolution operation, and Linear represents the linear transformation of the fully connected layer.

[0251] In some embodiments, during the temporal reconstruction of occlusion features, the object features extracted by OANet are input into the object Transformer. Multi-frame temporal information is used to reconstruct the motion of the occluded object. Visible frames provide clear position and pose information, while occluded frames obtain motion cues from adjacent frames through a temporal attention mechanism. RoPE helps the network understand the temporal pattern of occlusion and visibility. It can be seen that OANet retains object information in each frame, and even if the object is severely occluded or completely invisible, it can still stably reconstruct the complete motion trajectory through occlusion features and temporal dependencies.

[0252] In some embodiments, the second feature extraction module (OANet) is trained using a multi-task loss function to ensure the extraction of high-quality occlusion-aware features. This loss function includes: first, a coordinate map loss, which is the difference between the object coordinate map predicted using L2 loss constraints and the ground truth coordinate map rendered using known object poses; and second, a complete mask loss, which is the second mask predicted using cross-entropy loss constraints. The second feature extraction module considers three main factors: the difference between the predicted and actual object coordinates; the difference between the predicted and actual object coordinates; and the pose loss, which is achieved by recovering the object's pose from the predicted object coordinates using the Perspective-n-Point (PnP) algorithm, while constraining the difference between this pose and the object's true pose. By jointly optimizing these losses, the second feature extraction module learns the ability to accurately infer the object's geometric information and complete shape from the visible region.

[0253] The method, apparatus, device, and medium for recognizing human interaction based on a monocular camera provided in this application obtain a first mask corresponding to the object in each first image, obtain a pre-trained recognition model, extract features from each human image through a first feature extraction module in the recognition model to obtain first human features, and extract features from each interactive image and the corresponding first mask through a second feature extraction module to obtain first object features. The second feature extraction module predicts an object coordinate map, a two-dimensional correspondence map, and a second mask based on the object feature map corresponding to the interactive image, performs element-wise multiplication of the first mask with the stitched object coordinate map and the two-dimensional correspondence map to obtain visible area features, and obtains the first object features through a convolutional layer and a fully connected layer. In the above method, electronic devices can guide feature extraction through a first mask, enabling the network to learn the correspondence between two-dimensional and three-dimensional features only from the visible area of ​​the object, effectively avoiding noise interference in occluded areas. Furthermore, by predicting the second mask to learn the complete shape prior of the object, the network can still recover complete geometric information when the object is partially occluded. Moreover, through the parallel design of the first and second feature extraction modules, independent extraction and collaborative optimization of human and object features are achieved, avoiding the error propagation problem caused by serial processing in traditional methods. This method does not rely on depth sensors or multi-camera systems, reducing dependence on hardware devices and deployment costs, and enhancing robustness to occluded scenes. In addition, the recognition model is trained based on the first loss function constructed from the second loss function, the third loss function, and the interaction loss function, realizing joint constraints on human parameters, object parameters, and human interaction parameters, ensuring the training effect of the feature extraction module, and making the extracted features more closely match the real feature attributes of the human body and objects. Furthermore, this method is applicable to different interaction scenarios and object categories, significantly improving the robustness of recognition in occluded scenes and the convenience of system deployment while improving the accuracy of human interaction recognition.

[0254] Figure 4 A schematic diagram illustrating the process of another method for recognizing human interaction based on a monocular camera provided in this application. Please see [link / reference]. Figure 4 The method may include:

[0255] S401, Obtain input data.

[0256] The input data includes video sequences. (Multiple consecutive first images), object visible mask sequence (Multiple first masks) and a preset 3D object template, which defines the 3D mesh vertex coordinates of the object in a standard pose, for subsequent 3D object reconstruction.

[0257] S402. Perform data preprocessing on the input data to obtain human body images and interactive images.

[0258] The electronic device uses a target detection algorithm to perform human and object detection on each frame of the image, obtains human bounding boxes and object bounding boxes, and crops the input image of each frame based on the detected human and object bounding boxes to obtain human images and interactive images.

[0259] S403. Extract single-frame features in parallel to obtain the first human body feature and the first object feature.

[0260] Electronic devices will display human images The first feature extraction module (a pre-trained HMR2.0 model) is input to obtain the first human body feature. Simultaneously, the electronic device will display the interactive image. and the corresponding first mask Inputting the second feature extraction module (Occlusion Awareness Network OANet) yields the first object feature, and the relevant formula can be expressed as:

[0261]

[0262] S404. Perform feature fusion on the first human body feature and the first object feature to obtain the second human body feature and the second object feature.

[0263] The electronic device concatenates the vectors of the first human body feature and the first object feature corresponding to the same frame to obtain a combined feature vector that integrates human and object information. This combined feature vector is input into a first multilayer perceptron, and after linear transformation and nonlinear activation, it is mapped to a second human body feature. Simultaneously, this combined feature vector is input into a second multilayer perceptron and mapped to a second object feature. The relevant formula can be expressed as:

[0264]

[0265] S405. Perform temporal modeling on the second human body feature and the second object feature respectively to obtain the first temporal information and the second temporal information.

[0266] The electronic device arranges the second human features of multiple consecutive frames in chronological order to form a human feature sequence, where T is the temporal window length; this sequence is then input into the first temporal module (Human Transformer). This module adopts a Transformer architecture, firstly encoding the relative positions between frames through Rotation Position Encoding (RoPE), enabling the network to perceive the motion relationships between frames; then, it calculates the correlation weights between the features of each frame through an 8-head self-attention mechanism.

[0267] Furthermore, the second object features from multiple consecutive frames are arranged chronologically to form an object feature sequence; this sequence is then input into the second temporal module (Object Transformer). This module employs the same Transformer architecture as the first temporal module, but with a greater network depth to accommodate the high complexity of object motion and occlusion recovery requirements. Through the same RoPE encoding, multi-head self-attention, and gating mechanisms, it outputs an object-enhanced feature sequence incorporating temporal information, serving as the second temporal information. The relevant formula can be expressed as:

[0268]

[0269] S406. Perform motion parameter regression and trajectory calculation based on time series information to obtain human motion parameters and object motion parameters.

[0270] The electronic device can input the obtained first time-series information into the first regression module (HumanRegressor) to obtain human motion parameters, and input the second time-series information into the second regression module (Object Regressor). This module also adopts a multilayer perceptron architecture to map the time-series features into object motion parameters. The relevant formula can be expressed as:

[0271]

[0272] S407. Perform three-dimensional reconstruction based on human motion parameters and object motion parameters, and output the interaction information between the human body and the object.

[0273] The electronic device obtains the corresponding three-dimensional trajectory information based on the human motion parameters and the object motion parameters, performs three-dimensional reconstruction on it, and outputs the three-dimensional vertex information of the human body, the joint information and the three-dimensional vertex information of the object as the recognition result of human interaction, forming the three-dimensional interactive motion tracking result of human body and object.

[0274] In some embodiments, during performance testing, under the same hardware conditions, the performance of this application and the existing mainstream method VisTracker was compared on multiple datasets (BEHAVE dataset, InterCap dataset) for 1-minute videos. The results are shown in Table 1:

[0275] Table 1

[0276]

[0277] It can be seen that this application processes 1 minute of video in only 0.94 minutes, with each frame taking 0.015 seconds, which is 549 times and 573 times faster than VisTracker's 516.2 minutes and 8.6 seconds, respectively, meeting the requirements of real-time applications. In terms of accuracy, for the BEHAVE dataset, the human tracking v2v distance decreased from 8.74cm to 4.24cm, an improvement of 51.5%; object tracking decreased from 17.0cm to 14.67cm, an improvement of 13.7%. For the InterCap dataset, the human tracking v2v distance decreased from 9.83cm to 6.75cm, an improvement of 31.3%; object tracking decreased from 12.83cm to 10.51cm, an improvement of 18.1%. The Chamfer distance metric is significantly improved on both datasets, and the reconstructed 3D mesh closely matches the ground truth, achieving a good balance between accuracy and robustness. This dual improvement in efficiency and accuracy verifies the progressiveness and effectiveness of this solution.

[0278] In some embodiments, the effectiveness of each core module was verified by performing ablation experiments on the InterCap dataset, and the results are shown in Table 2:

[0279] Table 2

[0280]

[0281] It can be seen that removing OANet increases the object v2v error from 10.51cm to 19.7cm, demonstrating the core role of occlusion-aware feature extraction. Removing the contact loss has little impact on quantization metrics but reduces the detail reconstruction of human-object contact. Removing the occlusion-aware module increases the object tracking v2v distance on the InterCap dataset from 10.51cm to 12.32cm, proving the necessity of explicitly modeling occlusion states. The complete model achieves optimal results on all metrics, validating the rationality and synergy of the module design.

[0282] In summary, this invention achieves state-of-the-art performance on both the BEHAVE and InterCap standard HOI datasets, demonstrating strong generalization ability: it exhibits stable performance under various scenarios, including indoor and outdoor environments and different lighting conditions, and is effective for different object categories, such as rigid and deformable objects; the ablation experiment verifies the rationality of the design of each module.

[0283] Below, in conjunction with Figure 5 and Figure 6 This paper provides a detailed explanation of the tracking and recognition effects of the human interaction information in this application.

[0284] Figure 5 This is a schematic diagram illustrating the flow of the end-to-end HOI tracing method provided in this application. Figure 6 For a schematic diagram of the OANet and Transformer module structure provided in this application, please refer to [link / reference]. Figure 5 and Figure 6 As can be seen from the comparison results of multi-view 3D reconstruction, compared with the existing mainstream VisTracker method, this solution achieves accurate restoration of human posture, object posture and motion trajectory in typical interaction scenarios such as carrying bags and moving chairs. The reconstruction results from the front, side and back views are highly consistent, effectively avoiding the problems of object trajectory distortion, posture distortion and human interaction penetration that exist in VisTracker. In complex situations where objects are severely occluded by human bodies, this solution can still output stable and realistic 3D reconstruction results. While ensuring ultra-high reconstruction accuracy, it meets the real-time requirements of virtual reality, robotics and other scenarios, providing a high-performance and easy-to-deploy end-to-end solution for human interaction recognition in monocular scenes.

[0285] Below, in conjunction with Figure 7 The dual-branch architecture of this application will be described in detail.

[0286] Figure 7 For a schematic diagram of the dual-branch architecture provided in this application, please refer to [link / reference]. Figure 7 The dual-branch architecture includes four modules: feature extraction module, cross-fusion module, temporal modeling module, and motion regression module.

[0287] The feature extraction module is responsible for human body feature extraction and object feature extraction. Specifically, for human body feature extraction, a pre-trained HMR2.0 is used as the feature extractor. The cropped human body image is input to obtain 1024-dimensional single-frame human body features (first human body features). For object feature extraction, an occlusion perception network OANet is designed. The cropped image containing human body and object (interactive image) is input to obtain 1024-dimensional single-frame object features (first object features).

[0288] The cross-fusion module is constructed using a multi-layer MLP to perform cross-fusion on the first human feature. and the first object features Perform feature fusion to generate a second human feature Second object features By establishing adaptive relationships between characters at the feature level, implicit constraint relationships are learned, and error propagation caused by explicit dependencies is avoided.

[0289] The temporal modeling module is responsible for processing the second human body features. Second object features Each component is input into its respective Transformer module for temporal feature learning. Relative Position Encoding (RoPE) is used to implement a multi-head attention mechanism to capture the relative positional relationships between frames. The human Transformer has 6 network layers and outputs 512-dimensional temporal features. The object Transformer network has 12 layers (to accommodate higher complexity object features) and outputs 512-dimensional temporal features. .

[0290] The motion regression module can predict the motion parameters of the human body and objects based on the MLP architecture regressor. Both posture and rotation are represented by 6D rotation to ensure the continuity of SO(3) space. For human motion parameters, the trajectory is calculated by predicting human posture parameters, human shape parameters, and motion trajectory information, and by using weak perspective projection parameters.

[0291] For object motion parameters, the depth factor is directly calculated by predicting the object rotation parameters and motion trajectory information, avoiding the problem of bounding box instability. This architecture makes human and object motion tracking completely independent, establishing soft correlation only at the feature level, and completely avoiding direct dependence on trajectory estimation and error propagation.

[0292] Combination Figure 7It can be seen that the existing mainstream method VisTracker and the EiTracker proposed in this application have fundamental differences in technical architecture. Specifically, VisTracker adopts a multi-stage serial optimization architecture. Its core process is as follows: First, after inputting a monocular video sequence (t=0, ..., t=n), it is necessary to extract human pose and 2D key points in sequence. After the first round of optimization, iterative optimization is performed to complete the pose and transformation estimation. Then, through the distance field module, the 3D reconstruction of human body and object is realized. After introducing contact constraints, the occlusion network combined with the visibility score is used to process occlusion. Finally, after the second round of iterative optimization, the recognition results of human body and object are output.

[0293] The EiTracker proposed in this application adopts an end-to-end dual-branch parallel architecture. Its core process is as follows: First, after inputting a monocular video sequence (t=0, ..., t=n), it splits the image into independent human and object branches in parallel. The human branch extracts robust human features through a pre-trained human module, while the object branch explicitly models the occlusion state using an occlusion-aware object module, combining visible and full object masks to extract object features. Subsequently, an implicit interaction association between the human and object is established at the feature level through a cross module, and then fed into a dual-branch Transformer for differentiated temporal modeling. Finally, the corresponding motion parameters are output through human and object regressors, while also incorporating contact loss. The architecture constrains the geometric consistency of human interaction using loss; it achieves decoupled tracking of human body and object, completely blocks error propagation, and achieves accurate occlusion recovery by explicitly modeling occlusion using visible and complete masks, improving the single-frame prediction efficiency to 0.015s / frame.

[0294] Therefore, this solution adopts a parallel end-to-end architecture, which realizes independent tracking of human body and object through feature-level fusion and differentiated temporal modeling. Combined with the occlusion perception mechanism, it achieves significant improvements in accuracy, efficiency and occlusion robustness.

[0295] The method, apparatus, device, and medium for human interaction recognition based on a monocular camera provided in this application address three major technical problems of existing HOI tracking methods: First, the trajectories of the human body and objects interfere with each other, meaning that stationary objects may be misjudged as moving, and the independent motion characteristics of objects are masked by human movement, making it difficult to accurately recover the true motion trajectories of both; second, the motion recovery problem under severe occlusion, meaning that the complete motion cannot be fully recovered by making full use of some visible areas and temporal information, resulting in discontinuous and unstable motion reconstruction; and third, the problem of low processing efficiency, meaning that the multi-stage HOI tracking method based on optimization needs to go through multiple iteration stages, which cannot meet the real-time or near-real-time requirements of applications such as virtual reality and robotics.

[0296] To address the aforementioned technical challenges, this application proposes an end-to-end human interaction tracking method based on a monocular RGB camera. It employs a dual-branch architecture to simultaneously track both the human body and objects, designs an occlusion perception network to handle object occlusion, and combines end-to-end training with a loss function and an efficient prediction process to achieve accurate, stable, and efficient HOI tracking. Specifically, firstly, the dual-branch architecture enables independent tracking of the human body and objects, avoiding trajectory interference and error propagation, achieving excellent performance on mainstream datasets and significantly improving tracking accuracy. Secondly, OANet combined with a temporal network achieves stable motion recovery of occluded objects. This is achieved through visibility mask-guided geometric learning, predicting complete masks to understand the occlusion state, and using a temporal Transformer to recover the occlusion frame state using multi-frame information. Compared to traditional methods that filter low-visibility frames, this method retains object information from all frames, effectively addressing both occlusion and visibility states. This method achieves continuous and stable motion reconstruction in complex scenes with frequent switching, effectively handling severe occlusion. Thirdly, by proposing an end-to-end learning framework, it completely abandons the iterative process of multi-stage optimization, optimizing the overall objective through a unified loss function to avoid inconsistencies between stages. The network automatically learns implicit constraints on characters, eliminating the need for manually designed heuristic rules. Feature-level association learning is more robust than explicit geometric constraints, making it easy to integrate into existing video processing workflows and facilitating deployment and application. Therefore, this method achieves a qualitative leap in processing speed and can be directly applied to real-time or near-real-time scenarios. Fourthly, this technology can be widely applied in fields such as virtual reality and augmented reality interaction modeling, robot perception and operation analysis, game character animation generation, human-computer interaction behavior analysis, and video content understanding and retrieval, laying a solid foundation for the practical application of HOI tracking technology and possessing good generalization ability and practical value.

[0297] Figure 8 A schematic diagram of a human interaction recognition device based on a monocular camera provided in this application is shown below. Figure 8As shown, the human interaction recognition device 80 based on a monocular camera provided in this embodiment includes: an acquisition module 81, a first determination module 82, a fusion module 83, a second determination module 84, a third determination module 85, and a recognition module 86, wherein...

[0298] The acquisition module 81 is used to acquire multiple consecutive first images, the first images including the interaction scene between the human body and the object, and the first images are acquired based on a monocular camera;

[0299] The first determining module 82 is used to determine the first human features corresponding to the human body image of each first image, and the first object features corresponding to the human body and object interaction image.

[0300] The fusion module 83 is used to perform feature fusion on each first human body feature and the corresponding first object feature to obtain multiple second human body features and multiple second object features.

[0301] The second determining module 84 is used to determine the first temporal information between each second human body feature and the second temporal information between each second object feature;

[0302] The third determining module 85 is used to determine human motion parameters based on the first time sequence information and to determine object motion parameters based on the second time sequence information.

[0303] The recognition module 86 is used to identify the interaction information between the human body and the object in each first image based on the human body motion parameters and the object motion parameters.

[0304] The human interaction recognition device based on a monocular camera provided in this application embodiment can execute the technical solution shown in the above method embodiment. Its implementation principle and beneficial effects are similar, and will not be described again here.

[0305] In one possible implementation, the first determining module 82 is specifically used for:

[0306] Obtain the first mask corresponding to the object in each first image. The first mask is used to indicate the visible pixel area of ​​the object in the current first image.

[0307] Obtain a pre-trained recognition model, which is used to track and recognize the interaction information between the human body and objects. The recognition model is trained based on the first loss function.

[0308] By using the first feature extraction module in the recognition model, features are extracted from each human body image to obtain multiple first human body features;

[0309] By using the second feature extraction module in the recognition model, features are extracted from each interactive image and the corresponding first mask to obtain multiple first object features.

[0310] In one possible implementation, the first determining module 82 is specifically used for:

[0311] By using the second feature extraction module in the recognition model, features are extracted from each interactive image and its corresponding first mask to obtain multiple first object features, including:

[0312] Determine the object feature map corresponding to the interactive image;

[0313] Based on the object feature map, predict the object coordinate map, the two-dimensional correspondence map, and the second mask in the interactive image. The object coordinate map is used to indicate the mapping relationship between the two-dimensional pixels of the interactive image and the three-dimensional coordinates of the object. The two-dimensional correspondence map is used to indicate the coordinate information of each two-dimensional pixel. The second mask is used to indicate the visible and invisible pixel areas of the object.

[0314] The first mask is multiplied element-wise with the stitched object coordinate map and the two-dimensional corresponding map to obtain the features corresponding to the visible pixel area of ​​the object.

[0315] The convolutional and fully connected layers of the second feature extraction module process the features corresponding to the visible pixel regions of the object to obtain the first object features. The loss function of the second feature extraction module is constructed based on the object coordinate map and the second mask.

[0316] In one possible implementation, the second determining module 84 is specifically used for:

[0317] By recognizing the first temporal module of the model, multiple second human body features are processed temporally to obtain first temporal information, which is used to indicate the motion correlation between each second human body feature.

[0318] By recognizing the second temporal module of the model, multiple second object features are processed temporally to obtain second temporal information. The second temporal information is used to indicate the motion correlation between each second object feature. The number of network layers in the first temporal module is less than the number of network layers in the second temporal module.

[0319] In one possible implementation, the third determining module 85 is specifically used for:

[0320] The first regression module of the recognition model performs regression prediction on the first time series information to obtain human motion parameters. The human motion parameters include human posture parameters, human shape parameters and projection parameters. The human posture parameters are used to indicate the rotational posture of each joint of the human body, the human shape parameters are used to indicate the body shape characteristics of the human body, and the projection parameters are used to indicate the projection relationship of the human body on the image plane. The first regression module is used to map the human time series features into human motion parameters.

[0321] The second regression module of the identification model performs regression prediction on the second time series information to obtain the object motion parameters. The object motion parameters include object rotation parameters and translation parameters. The object rotation parameters are used to indicate the rotation state of the object in three-dimensional space, and the translation parameters are used to indicate the position offset of the object in three-dimensional space. The second regression module is used to map the object time series features to the object motion parameters.

[0322] In one possible implementation, the identification module 86 is specifically used for:

[0323] Acquire the camera parameters of the monocular camera, the first image information of each human body image, and the second image information of each interactive image;

[0324] Based on the projection parameters, camera parameters, and first image information, determine the human body's motion trajectory information;

[0325] Based on the preset 3D template of the object, translation parameters, camera parameters, and second image information, the motion trajectory information of the object is determined;

[0326] Based on human posture parameters, human shape parameters, and human motion trajectory information, the three-dimensional vertex information and joint information of the human body in each first image are reconstructed.

[0327] Based on the object's rotation parameters and motion trajectory information, determine the three-dimensional vertex information of the object in each first image;

[0328] The three-dimensional vertex information and joint information of the human body and the three-dimensional vertex information of the object are determined as the interaction information between the human body and the object.

[0329] In one possible implementation, the first loss function of the identification model is obtained based on the following steps:

[0330] Determine the second loss function and the third loss function. The second loss function is used to indicate the relevant parameters of the human body to be adjusted, and the third loss function is used to indicate the relevant parameters of the object to be adjusted.

[0331] Determine the interaction loss function between the human body and the object. The interaction loss function is used to constrain the contact relationship between the human body and the object.

[0332] The first loss function is determined based on the second loss function, the third loss function, and the interaction loss function.

[0333] Figure 9 A schematic diagram of the structure of the electronic device provided in this application. Figure 9 As shown, the electronic device 90 provided in this embodiment includes at least one processor 91 and a memory 92. Optionally, the electronic device 90 further includes a communication component 93. The processor 91, memory 92, and communication component 93 are connected via a bus.

[0334] In a specific implementation, at least one processor 91 executes computer execution instructions stored in memory 92, causing at least one processor 91 to perform the above-described method.

[0335] The specific implementation process of processor 91 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0336] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0337] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0338] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0339] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0340] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0341] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0342] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0343] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0344] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0345] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0346] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0347] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0348] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for recognizing human interactions based on a monocular camera, characterized in that, include: Acquire multiple consecutive first images, the first images including interactive scenes between human body and objects, the first images being acquired based on a monocular camera; Determine the first human body feature corresponding to the human body image of each first image, and the first object feature corresponding to the interaction image between the human body and the object; Feature fusion is performed on each first human body feature and the corresponding first object feature to obtain multiple second human body features and multiple second object features; Determine the first temporal information between the second human body features and the second temporal information between the second object features; Based on the first time series information, human motion parameters are determined, and based on the second time series information, object motion parameters are determined. Based on the human motion parameters and the object motion parameters, the interaction information between the human body and the object in each of the first images is identified.

2. The method according to claim 1, characterized in that, Determining the first human feature corresponding to the human image in each first image, and the first object feature corresponding to the interaction image between the human body and the object, includes: Obtain the first mask corresponding to the object in each of the first images, whereby the first mask is used to indicate the visible pixel area of ​​the object in the current first image; A pre-trained recognition model is obtained, which is used to track and recognize the interaction information between the human body and the object. The recognition model is trained based on a first loss function. The first feature extraction module in the recognition model extracts features from each human body image to obtain multiple first human body features. The second feature extraction module in the recognition model extracts features from each interactive image and its corresponding first mask to obtain multiple first object features.

3. The method according to claim 2, characterized in that, For any given interactive image, the second feature extraction module in the recognition model extracts features from each interactive image and its corresponding first mask to obtain multiple first object features, including: Determine the object feature map corresponding to the interactive image; Based on the object feature map, predict the object coordinate map, the two-dimensional correspondence map, and the second mask in the interactive image. The object coordinate map is used to indicate the mapping relationship between the two-dimensional pixels of the interactive image and the three-dimensional coordinates of the object. The two-dimensional correspondence map is used to indicate the coordinate information of each two-dimensional pixel. The second mask is used to indicate the visible pixel area and the invisible pixel area of ​​the object. The first mask is multiplied element-wise with the concatenated object coordinate map and the two-dimensional corresponding map to obtain the features corresponding to the visible pixel area of ​​the object. The convolutional and fully connected layers of the second feature extraction module process the features corresponding to the visible pixel regions of the object to obtain the first object features. The loss function of the second feature extraction module is constructed based on the object coordinate map and the second mask.

4. The method according to claim 3, characterized in that, Determining the first temporal information between the second human body features and the second temporal information between the second object features includes: The recognition model performs time-series processing on the plurality of second human body features to obtain the first time-series information, which is used to indicate the motion correlation between the various second human body features. The second temporal module of the recognition model performs temporal processing on the multiple second object features to obtain the second temporal information. The second temporal information is used to indicate the motion correlation between the various second object features. The number of network layers in the first temporal module is less than the number of network layers in the second temporal module.

5. The method according to any one of claims 2-4, characterized in that, Based on the first time-series information, human motion parameters are determined, and based on the second time-series information, object motion parameters are determined, including: The first regression module of the recognition model performs regression prediction on the first time series information to obtain the human motion parameters. The human motion parameters include human posture parameters, human shape parameters, and projection parameters. The human posture parameters are used to indicate the rotational posture of each joint of the human body. The human shape parameters are used to indicate the body shape characteristics of the human body. The projection parameters are used to indicate the projection relationship of the human body on the image plane. The first regression module is used to map the human time series features into human motion parameters. The second regression module of the recognition model performs regression prediction on the second time series information to obtain the object motion parameters. The object motion parameters include object rotation parameters and translation parameters. The object rotation parameters are used to indicate the rotation state of the object in three-dimensional space, and the translation parameters are used to indicate the position offset of the object in three-dimensional space. The second regression module is used to map the object time series features to the object motion parameters.

6. The method according to claim 5, characterized in that, Based on the human motion parameters and the object motion parameters, the interaction information between the human body and the object in each of the first images is identified, including: The camera parameters of the monocular camera, the first image information of each human body image, and the second image information of each interactive image are obtained. The motion trajectory information of the human body is determined based on the projection parameters, the camera parameters, and the first image information; Based on the preset 3D template of the object, the translation parameters, the camera parameters, and the second image information, the motion trajectory information of the object is determined; Based on the human body posture parameters, the human body shape parameters, and the human body motion trajectory information, the three-dimensional vertex information and joint information of the human body in each of the first images are reconstructed. Based on the object's rotation parameters and the object's motion trajectory information, the three-dimensional vertex information of the object in each of the first images is determined; The three-dimensional vertex information of the human body, the joint information, and the three-dimensional vertex information of the object are determined as the interaction information between the human body and the object.

7. The method according to claim 2, characterized in that, The first loss function of the recognition model is obtained based on the following steps: A second loss function and a third loss function are determined, wherein the second loss function is used to indicate the adjustment of relevant parameters of the human body, and the third loss function is used to indicate the adjustment of relevant parameters of the object; Determine the interaction loss function between the human body and the object, the interaction loss function being used to constrain the contact relationship between the human body and the object; The first loss function is determined based on the second loss function, the third loss function, and the interaction loss function.

8. A human interaction recognition device based on a monocular camera, characterized in that, include: The module comprises an acquisition module, a first determination module, a fusion module, a second determination module, a third determination module, and an identification module, wherein... The acquisition module is used to acquire multiple consecutive first images, the first images including interactive scenes between human bodies and objects, and the first images are acquired based on a monocular camera; The first determining module is used to determine the first human body feature corresponding to the human body image of each first image, and the first object feature corresponding to the interaction image between the human body and the object; The fusion module is used to perform feature fusion on each first human body feature and the corresponding first object feature to obtain multiple second human body features and multiple second object features. The second determining module is used to determine the first temporal information between the second human body features and the second temporal information between the second object features; The third determining module is used to determine human motion parameters based on the first time-series information and to determine object motion parameters based on the second time-series information. The recognition module is used to identify the interaction information between the human body and the object in each of the first images based on the human body motion parameters and the object motion parameters.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images