A training method, device, medium, and product for a subject motion representation extractor.

By training a subject motion representation extractor and combining the loss function of viewpoint video with sub-network reconstruction technology, the problem of inaccurate capture of agent ontology motion information is solved, achieving efficient motion information perception and driving strategy-related feature extraction, thus improving the accuracy of perception.

CN120708135BActive Publication Date: 2026-11-17LANGCHAO ELECTRONIC INFORMATION IND CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510855047.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-06-24
Publication Date
2026-11-17
Estimated Expiration
2045-06-24

AI Technical Summary

Technical Problem

Existing technologies cannot accurately capture the motion information of intelligent agents, resulting in inaccurate motion information perception, especially in first-person perspective videos where it is difficult to extract features related to driving strategies.

Method used

The subject motion representation extractor training method is adopted. By combining the loss between the actual next frame and the reconstructed frame of the first-view video and the loss between the predicted six degrees of freedom and the standard six degrees of freedom, photometric reconstruction is performed using the trained extrinsic detection subnetwork, intrinsic detection subnetwork and depth estimation subnetwork. A neural network capable of capturing the motion features of embodied intelligent agents is trained.

Benefits of technology

It improves the accuracy of motion information perception, can filter out noise information, focus on areas related to driving strategy, accurately capture the motion information of the intelligent agent, and improve the accuracy of driving decisions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120708135B_ABST
    Figure CN120708135B_ABST
Patent Text Reader

Abstract

This invention discloses a method, device, medium, and product for training a subject motion representation extractor, relating to the field of artificial intelligence technology. The method includes: training a subject motion representation extractor based on first-person perspective video, combining the loss between the actual next frame of the training frame and the reconstructed frame of the training frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame. This enables the subject motion representation extractor to learn whether various details within the field of vision affect driving behavior. The trained subject motion representation extractor can efficiently extract features from the first-person perspective video, obtain accurate and effective six degrees of freedom, filter noise information irrelevant to driving, and focus on areas directly related to driving strategies, accurately capturing the agent's motion information in the first-person perspective video, thereby improving the accuracy of motion information perception.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a training method, device, medium, and product for a subject motion representation extractor. Background Technology

[0002] Embodied intelligence is a cutting-edge field at the intersection of artificial intelligence and robotics. It emphasizes that intelligent agents achieve autonomous learning and evolution through dynamic interactions between their bodies and the environment. Its core lies in the deep integration of perception, action, and cognition. Embodied agents must fully understand the human intent in language commands, actively explore their surroundings, comprehensively perceive multimodal elements from both virtual and physical environments, and execute appropriate actions to complete complex tasks. Therefore, perception is crucial for the realization of embodied intelligence.

[0003] Video is an important carrier of multimodal information and one of the bases for motion information perception. In related technologies, methods such as Residual Network (ResNet) and Inflated 3D ConvNets (I3D) are used to extract features from video frames or segments. However, these methods are only applicable to third-person videos and cannot capture the motion information of the agent itself. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a training method, device, medium, and product for a subject motion representation extractor, which can accurately capture the motion information of an intelligent agent and improve the accuracy of motion information perception. The specific solution is as follows: In a first aspect, this application discloses a method for training a subject motion representation extractor, including: The training frames are input into the motion representation extractor of the subject to be trained, and the predicted six degrees of freedom are obtained based on the output; the training frames are video frames in the first-view video. The total loss is determined using a first loss function; the total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame; the reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom. The parameters of the subject motion representation extractor to be trained are adjusted according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion characteristics of the embodied intelligent agent, so as to use the six degrees of freedom predicted by the subject motion representation extractor to perceive the motion information of the embodied intelligent agent.

[0005] Secondly, this application discloses a method for perceiving motion information of an embodied intelligent agent, including: Acquire the target image frame currently captured by the embodied intelligent agent from a first-person perspective; The target image frame is input into the aforementioned subject motion representation extractor to obtain the six degrees of freedom between the target image frame and the next image frame predicted by the subject motion representation extractor. The motion information of the embodied intelligent agent is perceived based on the six degrees of freedom.

[0006] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned subject motion representation extractor training method or embodied intelligent agent motion information perception method.

[0007] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned subject motion representation extractor training method or embodied intelligent agent motion information perception method.

[0008] Fifthly, this application discloses a computer program product, including a computer program that, when executed by a processor, implements the aforementioned subject motion representation extractor training method or embodied intelligent agent motion information perception method.

[0009] In this application, training frames are input into a subject motion representation extractor to be trained, and predicted six degrees of freedom are obtained based on the output. The training frames are video frames in a first-view video. A first loss function is used to determine the total loss. The total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame. The reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom. The parameters of the subject motion representation extractor to be trained are adjusted according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion features of an embodied intelligent agent, so as to use the predicted six degrees of freedom of the subject motion representation extractor to perceive the motion information of the embodied intelligent agent.

[0010] The beneficial effects of this invention are as follows: Based on the first-person perspective video combined with the loss between the actual next frame of the training frame and the reconstructed frame of the training frame, as well as the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame, the subject motion representation extractor is trained, enabling the subject motion representation extractor to learn whether various detailed information within the field of vision affects driving behavior. The trained subject motion representation extractor can efficiently extract features from the first-person perspective video, obtain accurate and effective six degrees of freedom, filter noise information irrelevant to driving and focus on areas directly related to driving strategy, accurately capture the agent's motion information in the first-person perspective video, and thus improve the accuracy of motion information perception. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort. Figure 1 A flowchart of a subject motion representation extractor training method provided in this application; Figure 2 A schematic diagram illustrating photometric reconstruction training of a specific extrinsic parameter detection subnetwork, intrinsic parameter detection subnetwork, and depth estimation subnetwork provided for this application; Figure 3 This application provides a training schematic diagram of a specific motion representation extractor for a subject to be trained; Figure 4 This is a schematic diagram of an optimized subject motion characterization extractor provided for this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0014] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0015] The specific application environment architecture or specific hardware architecture on which the execution of the subject motion representation extractor training method depends is described here.

[0016] In related technologies, features are extracted from video frames or segments using methods such as residual networks and dilated 3D convolutional networks. However, these methods are only applicable to third-person videos and cannot capture the motion information of the agent itself. Furthermore, feature extraction models for vehicle videos in related technologies rely on pseudo-labels, but the accuracy of pseudo-labels generated by the model is poor, reducing the model's accuracy. To overcome the above technical problems, this application proposes a method for training an agent motion representation extractor, which can accurately capture the motion information of the agent and improve the accuracy of motion information perception.

[0017] This application discloses a method for training a subject motion representation extractor. See also... Figure 1 As shown, the method may include the following steps: Step S11: Input the training frames into the motion representation extractor of the subject to be trained, and obtain the predicted six degrees of freedom based on the output; the training frames are video frames in the first-view video.

[0018] Training frames are image frames taken from the first-person perspective of the embodied agent. In other words, first-person perspective video refers to video content captured by the embodied agent using its own image acquisition device, presented from its own point of view. The trained subject motion representation extractor can capture the subject motion characteristics of the embodied agent.

[0019] Understandably, in third-person perspective scenarios, the focus is on capturing detailed information about objects in the video. However, in embodied intelligent devices (such as robots and autonomous vehicles), like autonomous vehicles, the onboard cameras record the vehicle's own motion and the constantly evolving surrounding scene as the vehicle moves, thus forming the corresponding video. Video understanding requires inferring changes in motion and scene based on the video information. Due to the complexity and variability of video scenes, traditional methods struggle to extract effective video representations. For example, while there are many objects in a driving scene, such as moving vehicles and stationary buildings, these do not affect the current vehicle's motion state, whereas traffic lights and other objects play a decisive role in vehicle movement. The vehicle's trajectory is determined by the driving strategy; only with driving decision-making awareness can one focus more on visual cues crucial to driving operations. Therefore, this application proposes a method for training a subject motion representation extractor for first-person perspective videos. The trained subject motion representation extractor possesses driving decision-making awareness.

[0020] The output of the subject motion representation extractor is the predicted six degrees of freedom between the input image frame and the next image frame. The next image frame is the image frame following the input image frame. The trained subject motion representation extractor possesses driving decision-making awareness and can predict the agent's changes between the current image frame and the next image frame. Six degrees of freedom (6-DoF) refers to the six independent directions of motion a rigid body can perform in three-dimensional space, including three translational degrees of freedom (movement along the X, Y, and Z axes) and three rotational degrees of freedom (rotation about the X, Y, and Z axes). In other words, the agent's steering changes and positional shifts can be clearly defined through the six degrees of freedom.

[0021] The training frames are input into the subject motion representation extractor to be trained, including: reconstructing video frames based on the training frames using the subject motion representation extractor to be trained, a post-trained intrinsic parameter detection subnetwork, and a post-trained depth estimation subnetwork to obtain reconstructed frames; inputting the training frames and the actual next frame of the training frames into the post-trained extrinsic parameter detection subnetwork, and obtaining the standard six degrees of freedom corresponding to the training frames based on the output. That is, the post-trained extrinsic parameter detection subnetwork, the post-trained intrinsic parameter detection subnetwork, and the post-trained depth estimation subnetwork are used as prior knowledge to guide the subject motion representation extractor to learn how to capture and represent the subject motion features of the embodied intelligent agent.

[0022] Step S12: Determine the total loss using the first loss function; the total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame; the reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom.

[0023] The first loss function is constructed based on the loss between the actual next frame of the training frame input to the subject motion representation extractor and the reconstructed frame obtained after photometric reconstruction, as well as the loss between the six degrees of freedom extracted by the subject motion representation extractor based on the training frame and the six degrees of freedom extracted by the extrinsic parameter detection subnetwork based on the training frame.

[0024] Based on the loss between the actual next frame and the reconstructed frame of the training frame, training constraints are applied using frame-level loss. Based on the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame, training constraints are applied using six-degree-of-freedom level loss. By combining frame-level and six-degree-of-freedom level loss functions, the training effect is improved and the accuracy of the subject motion representation extractor is enhanced.

[0025] Specifically, the training steps of the subject motion representation extractor may include: performing photometric reconstruction based on a first-view video using the subject motion representation extractor to be trained, a post-trained intrinsic parameter detection subnetwork, and a post-trained depth estimation subnetwork; and training the subject motion representation extractor using a first loss function. The first loss function is constructed by combining the loss between the actual next frame input to the training frame of the subject motion representation extractor and the reconstructed frame obtained based on that training frame, as well as the loss between the six degrees of freedom extracted by the subject motion representation extractor based on the training frame and the six degrees of freedom extracted by the post-trained extrinsic parameter detection subnetwork based on that training frame. That is, the post-trained extrinsic parameter detection subnetwork, post-trained intrinsic parameter detection subnetwork, and post-trained depth estimation subnetwork serve as prior knowledge to guide the subject motion representation extractor in learning how to capture and represent the subject motion features of the embodied intelligent agent.

[0026] Step S13: Adjust the parameters of the subject motion representation extractor to be trained according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion features of the embodied intelligent agent, so as to use the six degrees of freedom predicted by the subject motion representation extractor to perceive the motion information of the embodied intelligent agent.

[0027] Among them, the subject motion representation extractor is trained using first-person perspective video combined with photometric reconstruction technology, and the subject motion representation extractor is a neural network model that can capture the motion characteristics of an embodied intelligent agent.

[0028] As can be seen from the above, in this embodiment, training frames are input into the subject motion representation extractor to be trained, and the predicted six degrees of freedom are obtained based on the output; the training frames are video frames in the first-view video; the total loss is determined using a first loss function; the total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame; the reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom; the parameters of the subject motion representation extractor to be trained are adjusted according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion features of the embodied intelligent agent, so as to use the six degrees of freedom predicted by the subject motion representation extractor to perceive the motion information of the embodied intelligent agent.

[0029] It is evident that by combining the loss between the actual next frame of the training frame and the reconstructed frame of the retrained frame based on the first-person perspective video, as well as the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame, the subject motion representation extractor is trained. This enables the subject motion representation extractor to learn whether various details within the field of vision affect driving behavior. The trained subject motion representation extractor can efficiently extract features from the first-person perspective video, obtain accurate and effective six degrees of freedom, filter out noise information irrelevant to driving and focus on areas directly related to driving strategy, accurately capture the agent's motion information in the first-person perspective video, and thus improve the accuracy of motion information perception.

[0030] Training the subject motion representation extractor requires a post-trained extrinsic parameter detection subnetwork, a post-trained intrinsic parameter detection subnetwork, and a post-trained depth estimation subnetwork. Therefore, the steps for generating these subnetworks before training the subject motion representation extractor can specifically include: S21: Generate a training set based on first-person perspective videos; Specifically, the process involves downloading videos captured by the agent from its own first-person perspective, resulting in multiple first-person view videos. These videos are then preprocessed to create a training set containing multiple video frames. The preprocessing includes frame rate adjustment and frame extraction. Specifically, these videos are uniformly processed, for example, adjusted to 30 frames per second, and frames are extracted at intervals of two frames to obtain the training set. Taking the embodied agent as a vehicle as an example, multiple unedited, high-definition first-person driving videos are downloaded from video websites, covering various scenarios that might be encountered during driving.

[0031] S22: Based on the training set, perform video frame reconstruction training on the extrinsic detection subnetwork, intrinsic detection subnetwork, and depth estimation subnetwork to obtain the trained extrinsic detection subnetwork, trained intrinsic detection subnetwork, and trained depth estimation subnetwork.

[0032] The training process for video frame reconstruction based on the training set includes: training the extrinsic parameter detection subnetwork, intrinsic parameter detection subnetwork, and depth estimation subnetwork on video frame reconstruction using every three consecutive training frames as training units; the extrinsic parameter detection subnetwork is used to determine the camera extrinsic parameter change data between two frames, the intrinsic parameter detection subnetwork is used to determine the camera intrinsic parameters, and the depth estimation subnetwork is used to determine the depth map corresponding to the image frame.

[0033] Specifically, a self-supervised photometric reconstruction technique can be employed. Scene reconstruction is achieved by standardizing the color consistency between video frames. This process incorporates three sub-network modules: an extrinsic parameter detection sub-network, an intrinsic parameter detection sub-network, and a depth estimation sub-network. These three modules work together to achieve accurate scene reconstruction. Specifically, the extrinsic parameter detection sub-network can be... Network and internal parameter detection subnetwork can provide Networks and deep detection subnetworks can provide The network consists of two subnetworks. PoseNet is a deep learning-based human pose estimation method, and DepthNet is a recurrent neural network architecture for monocular depth prediction. Both are industry-standard network models, but other feasible networks can also be used. The depth estimation subnetwork calculates the distance from each point in the current scene to the camera, thus converting the current 2D image into a 3D reality scene. The two parameter detection subnetworks focus on estimating the intrinsic and extrinsic parameters of the vehicle camera. The intrinsic parameters record the inherent properties of the vehicle camera, while the extrinsic parameters reveal the angle change and position shift of the camera between two consecutive frames. These can be used to estimate the vehicle's steering angle and speed. This embodiment uses 6-DoF (six degrees of freedom) representation to provide crucial spatial dynamic information for accurate scene reconstruction.

[0034] Photometric reconstruction from the current video frame to the next frame is achieved by coordinating extrinsic parameter estimation networks, intrinsic parameter estimation networks, and depth estimation subnetworks. This process does not rely on external supervision, but rather uses a large number of unsupervised videos to achieve reconstruction through the interaction between networks.

[0035] Specifically, using every three consecutive training frames as a training unit, video frame reconstruction training is performed on the extrinsic parameter detection subnetwork, intrinsic parameter detection subnetwork, and depth estimation subnetwork, including: S221: Input the t-th training frame and the (t-1)-th training frame into the extrinsic parameter detection subnetwork and the intrinsic parameter detection subnetwork, respectively, and input the t-th training frame into the depth estimation subnetwork; For example Figure 2 As shown, three adjacent frames: , , Training is performed in units of frame t-1, frame t, and frame t+1. , Input extrinsic parameters to detect subnetworks respectively and internal reference detection subnetwork The output of the extrinsic parameter detection subnetwork is obtained: ; This represents the change in camera extrinsic parameters from frame t-1 to frame t, which is the relative pose between the two frames; The output of the intrinsic parameter detection subnetwork is the predicted camera intrinsic parameters: ; This represents the camera intrinsic parameters extracted based on frame t-1. This represents the camera intrinsic parameters extracted based on frame t. Will Input the depth detection subnetwork to obtain the depth map of frame t. : .

[0036] S222: Based on the outputs of the extrinsic detection subnetwork, the intrinsic detection subnetwork and the depth estimation subnetwork, a first reconstructed frame is obtained by photometric reconstruction. The first reconstructed frame is the t-th reconstructed frame obtained based on the (t-1)-th training frame. according to To rebuild The reconstructed frame t, obtained from the (t-1)th training frame, is represented as: ; Where K represents the camera intrinsic parameters, This represents the change in camera extrinsic parameters from frame (t-1) to frame t. This represents the depth map of frame t. The proj() function maps the original pixel space to the reconstructed two-dimensional pixel space using bilinear interpolation. The pixel space is downsampled to form the first reconstructed frame. .

[0037] S223: Input the t-th training frame and the (t+1)-th training frame into the extrinsic parameter detection subnetwork and the intrinsic parameter detection subnetwork, respectively, and input the t-th training frame into the depth estimation subnetwork; Will , Input extrinsic parameters to detect subnetworks respectively and internal reference detection subnetwork Based on the output of the extrinsic detection subnetwork, we obtain: ; This represents the predicted change in camera extrinsic parameters from frame t+1 to frame t. The output of the intrinsic parameter detection subnetwork is the predicted camera intrinsic parameters: ; This represents the camera intrinsic parameters extracted based on frame t+1. This represents the camera intrinsic parameters extracted based on frame t. Similarly, Input the depth detection subnetwork to obtain the depth map of frame t. : .

[0038] S224: Based on the outputs of the extrinsic detection subnetwork, the intrinsic detection subnetwork and the depth estimation subnetwork, a second reconstructed frame is obtained by photometric reconstruction. The second reconstructed frame is the t-th reconstructed frame obtained based on the (t+1)-th training frame. according to To rebuild The reconstructed frame t obtained based on the (t+1)th training frame is represented as: ; Where K represents the camera intrinsic parameters, This represents the change in camera extrinsic parameters from frame (t+1) to frame (t). This represents the depth map of frame t. The proj() function maps the original pixel space to the reconstructed two-dimensional pixel space, and then uses bilinear interpolation to... Pixel space downsampling forms the second reconstructed frame. .

[0039] S225: The extrinsic parameter detection subnetwork, the intrinsic parameter detection subnetwork, and the depth estimation subnetwork are trained for photometric reconstruction using the second loss function; the second loss function is constructed based on the difference between the t-th training frame and the first reconstructed frame, and the difference between the t-th training frame and the second reconstructed frame.

[0040] In a specific implementation, the second loss function is constructed based on the structural similarity metric (SSIM) and L1 loss between the t-th training frame and the first reconstructed frame, the structural similarity metric and L1 loss between the t-th training frame and the second reconstructed frame, and the disparity smoothness loss of the t-th training frame. Specifically, the second loss function can be: ; in, and All of these are preset weight hyperparameters. For frame reconstruction loss, Indicates the loss of parallax smoothness; It consists of the structural similarity metric SSIM and the L1 loss term: ; in, For coefficients; ; in, Represents the gradient in the horizontal direction. Represents the gradient in the vertical direction. This represents the mean-normalized inverse depth map.

[0041] Structural Similarity (SSIM) between two frames captures the similarity of images or features in terms of structure, brightness, and contrast. By utilizing SSIM loss, more natural and visually plausible motion estimates can be generated, especially suitable for tasks that require maintaining texture and edge continuity. It provides constraints on motion or change that are more in line with human perception. In vehicle motion estimation, combining SSIM can improve robustness to complex scenes (such as shadows and dynamic objects). L1 loss provides pixel-level photometric consistency constraints and is robust to noise. By providing low-order pixel accuracy with L1 and high-order structural awareness with SSIM, combined with disparity smoothness loss, the motion field is forced to conform to physical laws, avoiding cluttered estimation results and improving the robustness and visual effect of the model.

[0042] The first stage of training yields three sub-networks: a post-trained extrinsic parameter detection sub-network, a post-trained intrinsic parameter detection sub-network, and a post-trained depth estimation sub-network. The post-trained extrinsic parameter detection sub-network is closest to the requirements, but it essentially captures the relative motion differences between two adjacent frames. In reality, what's needed is the difference information within a single frame; this difference represents how driving should be performed to form the ideal next frame. This means simulating the driving / action decisions that should be made when encountering a scene in a given observation frame—that is, the network needs to learn driving / action policy information. In the second stage, specifically for training the subject motion representation extractor, the previously trained post-trained extrinsic parameter detection sub-network, post-trained intrinsic parameter detection sub-network, and post-trained depth estimation sub-network are frozen. That is, the model parameters of these sub-networks remain unchanged during the second stage of training and they are no longer involved in training. The subject motion representation extractor to be trained can specifically be a pose detection network. This network is also used to extract six degrees of freedom in order to estimate camera extrinsic parameters.

[0043] In specific implementation, the training frames are input into the motion representation extractor of the subject to be trained, including: S31: Based on the training frames, video frames are reconstructed using the subject motion representation extractor to be trained, the intrinsic parameter detection subnetwork after training, and the depth estimation subnetwork after training to obtain reconstructed frames; The training frame can be any frame in the training set generated based on first-person view video.

[0044] S32: Input the training frame and the actual next frame of the training frame into the post-training extrinsic parameter detection subnetwork, and obtain the standard six degrees of freedom corresponding to the training frame based on the output; Specifically, the training process of the subject motion representation extractor includes: training the p-1th training frame (i.e., ... The inputs are respectively fed into the subject motion representation extractor to be trained, the intrinsic parameter detection subnetwork after training, and the depth estimation subnetwork after training; based on the outputs of the subject motion representation extractor to be trained, the intrinsic parameter detection subnetwork after training, and the depth estimation subnetwork after training, the p-th reconstructed frame (i.e., the p-th frame) is obtained through photometric reconstruction. That is, the p-th reconstructed frame is obtained based on the p-1-th training frame; the motion representation extractor of the subject to be trained is trained using the first loss function; wherein, during the training process, the network parameters of the post-training intrinsic parameter detection subnetwork and the post-training depth estimation subnetwork are frozen and remain unchanged. For example Figure 3 The diagram illustrates the training process of the motion representation extractor for the subject to be trained, involving only input... Reconstruction yields At the same time, the p-1th training frame and the p-th training frame ( Input the extrinsic parameter detection subnetwork after training, and obtain the standard six degrees of freedom of the p-1th training frame based on the output.

[0045] Specifically, the first loss function includes a first sub-loss function and a second sub-loss function; the first sub-loss function is constructed based on the structural similarity metric and L1 loss between the p-th training frame and the p-th reconstructed frame, as well as the disparity smoothness loss of the p-th training frame; that is, based on... and The structural similarity measure and L1 loss between them generate the first sub-loss function Lc; the second sub-loss function is constructed based on the predicted six degrees of freedom output by the subject motion representation extractor based on the p-1th training frame and the standard six degrees of freedom output by the extrinsic parameter detection subnetwork based on the p-1th and pth training frames.

[0046] The (p-1)th and pth training frames are input into the trained extrinsic parameter detection network, and its output is used as the ground label for training. Since the extrinsic parameter detection network processes motion information changes between two adjacent frames after training, optimization... When determining the parameters, the process actually involves learning to acquire the correct motion policy information by observing the (p-1)th training frame in order to generate the ideal p-th reconstructed frame. Therefore... It will definitely be able to perceive rich motion information, and after training, a vehicle motion representation extractor will be obtained.

[0047] During motion state estimation, only one input is used at the input terminal. And utilize the subject motion characterization extractor Extract the corresponding motion state information When calculating the loss, the frozen extrinsic parameters are used to detect the network. Calculated relative pose The second sub-loss function Lm is obtained: ; Finally, Lc+Lm is used as the first loss function to train the parameters of the subject motion representation extractor.

[0048] In practice, the training process of the subject motion representation extractor includes: S41: Input the training frames into the subject motion representation extractor to be trained, and obtain the predicted six degrees of freedom based on the output; the training frames are video frames in the first-view video. S42: Determine the total loss using the first loss function; the total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame; the reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom; S43: Adjust the parameters of the subject motion representation extractor to be trained according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion features of the embodied intelligent agent; S44: Obtain the initial post-training subject motion representation extractor, and optimize the initial post-training subject motion representation extractor through visual odometry estimation to obtain the final post-training subject motion representation extractor.

[0049] That is, after training the subject motion representation extractor using the first loss function, the subject motion representation extractor is further optimized. It is understandable that after the first two stages, an initially trained vehicle subject motion representation extractor is obtained, but since all training is unsupervised, only basic motion features can be extracted. Therefore, supervised training using a large amount of open labeled data continues, and the feature extractor is further refined using a visual odometry task. Visual odometry aims to estimate the changes of a moving subject captured by a camera between adjacent frames. For example, the publicly available KITTI odometry dataset, which is recorded at 10 frames per second by a vision system mounted on a vehicle, captures street images during driving; this data is commonly used for visual odometry. To maintain the use of monocular vision, only video data captured from the left-side camera can be used.

[0050] The pre-trained feature extractor is further optimized and tuned using publicly available labeled datasets. This enables the feature extractor to better adapt to diverse driving scenarios, improving its generalization ability and accuracy in practical applications.

[0051] Specifically, optimizing the initially trained subject motion representation extractor through visual odometry estimation includes: inputting the odometry dataset into the initially trained subject motion representation extractor, the output of which is the input of the encoder; obtaining the encoded features corresponding to each image frame based on the encoder output, inputting the encoded features into a multilayer perceptron to obtain motion difference information; and optimizing the initially trained subject motion representation extractor using a third loss function; the third loss function is constructed based on the difference in encoded features between two adjacent frames and the motion difference information between those two frames.

[0052] For example Figure 4As shown, a basic Transformer encoder structure is used to further encode the features extracted by the vehicle motion representation extractor. Then, the differences between two adjacent frames are estimated using the features from the multilayer perceptron. These differences can be represented using six degrees of freedom (DOF). Six DEF, as a specific form of camera extrinsic parameter, records the ability of a 3D object to autonomously move along three dimensions and rotate at three angles in space from a first-person perspective. It can be understood that the output of the vehicle motion representation extractor is a low-level motion representation, while the encoder output is a higher-level temporal motion representation. The motion difference information obtained by the multilayer perceptron enables the model to further learn how to accurately predict motion changes between adjacent frames from the features.

[0053] For example, at time t, the motion state matrix of the car for: ; in This represents a rotation matrix that describes the car's steering angle. Representing the translation vector, the motion difference information between time t-1 and time t. Represented as: ; For a video containing N consecutive frames, N-1 relative motion difference information points can be calculated. Inputting N video frames results in N features after encoding. Calculate the difference between adjacent features to obtain The third loss function, constructed based on the differences in encoded features between two adjacent frames and the motion difference information between those two frames, is as follows: ; Where CrossEntropy represents the cross-entropy loss and Encoder represents the encoding.

[0054] The subject motion representation extractor obtained through the above training method can focus on subject motion information and the area that affects driving strategy, so that the extracted features contain rich motion information, realize the capture and representation of vehicle subject motion features with generalization, and avoid the influence of details of other objects in the video on perception.

[0055] Furthermore, in related technologies, the feature extraction models for first-person vehicle videos rely on pseudo-labels, which suffer from poor accuracy in generating pseudo-labels. This application addresses this issue by using a self-supervised photometric reconstruction-based subject motion representation extractor. This allows for training a video feature extractor without relying on manually labeled data, through a self-supervised approach. In the first two stages, photometric reconstruction technology is used to construct a subject motion feature extraction model capable of accurately capturing regions that substantially influence driving strategies. Then, in the third stage, labeled data is used to meticulously adjust and optimize the model, ensuring it can learn appropriate driving decisions to be made in specific scenarios. Ultimately, a vehicle subject motion representation extractor that conforms to driving strategies is obtained.

[0056] This application discloses a specific method for perceiving motion information of an embodied intelligent agent, which may include the following steps: Step S51: Obtain the target image frame from the first perspective currently acquired by the embodied intelligent agent; In this embodiment, the target video frame acquired is the one currently captured by the embodied intelligent agent, and it is an image frame from the first-person perspective of the embodied intelligent agent. That is, first-person perspective video refers to video content captured by the embodied intelligent agent using its own image acquisition device, from its own primary perspective. Step S52: Input the target image frame into the aforementioned subject motion representation extractor to obtain the six degrees of freedom between the target image frame and the next image frame predicted by the subject motion representation extractor; In this embodiment, the target image frame is input to a pre-trained subject motion representation extractor. The output of the subject motion representation extractor is the predicted six degrees of freedom between the target image frame and the next image frame. The next image frame is the image frame following the target image frame. The trained subject motion representation extractor has driving decision awareness and can predict the changes of the agent between the current image frame and the next image frame.

[0057] Step S53: Perceive the motion information of the embodied intelligent agent based on the six degrees of freedom. Finally, based on the six degrees of freedom output by the subject motion representation extractor, motion information perception for the matrix agent is achieved. Motion information perception refers to understanding the dynamic changes of objects or scenes by analyzing motion features in videos. However, motion information perception of a matrix agent is a complex process. This process cannot only utilize six degrees of freedom but also combines multiple perception methods and multimodal data fusion technology to achieve more comprehensive and accurate environmental perception and motion control. This embodiment does not limit this, and any method in related technologies that can achieve motion perception is acceptable. The specific process of the above-mentioned subject motion representation extractor training method can be referred to the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0058] As can be seen from the above, in this embodiment, the target image frame of the first perspective currently collected by the embodied intelligent agent is obtained; the target image frame is input into the aforementioned subject motion representation extractor to obtain the six degrees of freedom between the target image frame and the next image frame predicted by the subject motion representation extractor; and the motion information perception of the embodied intelligent agent is performed based on the six degrees of freedom.

[0059] It is evident that by combining the loss between the actual next frame of the training frame and the reconstructed frame of the retrained frame based on the first-person perspective video, as well as the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame, the subject motion representation extractor is trained. This enables the subject motion representation extractor to learn whether various details within the field of vision affect driving behavior. The trained subject motion representation extractor can efficiently extract features from the first-person perspective video, obtain accurate and effective six degrees of freedom, filter out noise information irrelevant to driving and focus on areas directly related to driving strategy, accurately capture the agent's motion information in the first-person perspective video, and thus improve the accuracy of motion information perception.

[0060] Accordingly, this application also discloses a subject motion representation extractor training device, which includes: The input module is used to input training frames into the motion representation extractor of the subject to be trained, and obtain the predicted six degrees of freedom based on the output; the training frames are video frames in the first-view video. A loss determination module is used to determine the total loss using a first loss function; the total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame; the reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom; The parameter adjustment module is used to adjust the parameters of the subject motion representation extractor to be trained according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion characteristics of the embodied intelligent agent, so as to use the six degrees of freedom predicted by the subject motion representation extractor to perceive the motion information of the embodied intelligent agent.

[0061] As can be seen from the above, in this embodiment, training frames are input into the subject motion representation extractor to be trained, and the predicted six degrees of freedom are obtained based on the output; the training frames are video frames in the first-view video; the total loss is determined using a first loss function; the total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame; the reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom; the parameters of the subject motion representation extractor to be trained are adjusted according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion features of the embodied intelligent agent, so as to use the six degrees of freedom predicted by the subject motion representation extractor to perceive the motion information of the embodied intelligent agent. It is evident that by combining the loss between the actual next frame of the training frame and the reconstructed frame of the retrained frame based on the first-person perspective video, as well as the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame, the subject motion representation extractor is trained. This enables the subject motion representation extractor to learn whether various details within the field of vision affect driving behavior. The trained subject motion representation extractor can efficiently extract features from the first-person perspective video, obtain accurate and effective six degrees of freedom, filter out noise information irrelevant to driving and focus on areas directly related to driving strategy, accurately capture the agent's motion information in the first-person perspective video, and thus improve the accuracy of motion information perception.

[0062] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0063] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above-described subject motion representation extractor training methods or embodied intelligent agent motion information perception method embodiments.

[0064] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described subject motion representation extractor training methods or embodied intelligent agent motion information perception method embodiments when running.

[0065] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0066] The embodiments of this application also provide a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-described subject motion representation extractor training methods or embodied intelligent agent motion information perception method embodiments.

[0067] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-described subject motion representation extractor training methods or embodied intelligent agent motion information perception method embodiments.

[0068] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0069] The foregoing has provided a detailed description of a subject motion representation extractor training method or an embodied intelligent agent motion information perception method. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for training a subject motion representation extractor, characterized in that, include: The training frames are input into the motion representation extractor of the subject to be trained, and the predicted six degrees of freedom are obtained based on the output; the training frames are video frames in the first-view video. The total loss is determined using a first loss function; the total loss includes the loss between the actual next frame of the training frame and the reconstructed frame, and the loss between the predicted six degrees of freedom and the standard six degrees of freedom corresponding to the training frame; the reconstructed frame is the next frame of the training frame reconstructed based on the predicted six degrees of freedom; the standard six degrees of freedom is the true label for training the subject motion representation extractor. The parameters of the subject motion representation extractor to be trained are adjusted according to the total loss until the training conditions are met to obtain a subject motion representation extractor that can capture the motion characteristics of the embodied intelligent agent, so as to use the six degrees of freedom predicted by the subject motion representation extractor to perceive the motion information of the embodied intelligent agent. The process of inputting training frames into the motion representation extractor of the subject to be trained includes: Based on the training frames, video frames are reconstructed using a subject motion representation extractor to be trained, a post-trained intrinsic parameter detection subnetwork, and a post-trained depth estimation subnetwork to obtain the reconstructed frames. The training frame and the actual next frame of the training frame are input into the extrinsic parameter detection subnetwork after training, and the standard six degrees of freedom corresponding to the training frame are obtained according to the output. The first loss function includes a first sub-loss function and a second sub-loss function; The first sub-loss function is constructed based on the structural similarity metric and L1 loss between the p-th training frame and the p-th reconstructed frame, as well as the disparity smoothness loss of the p-th training frame; The second sub-loss function is constructed based on the cross-entropy loss between the predicted six degrees of freedom output by the subject motion representation extractor based on the (p-1)th training frame and the standard six degrees of freedom output by the post-trained extrinsic parameter detection sub-network based on the (p-1)th training frame and the pth training frame.

2. The subject motion representation extraction device training method according to claim 1, characterized in that, Based on the training frames, video frame reconstruction is performed using a subject motion representation extractor to be trained, a post-trained intrinsic parameter detection subnetwork, and a post-trained depth estimation subnetwork, including: The p-1th training frame is input into the subject motion representation extractor to be trained, the post-training intrinsic parameter detection subnetwork, and the post-training depth estimation subnetwork, respectively. Based on the outputs of the subject motion representation extractor to be trained, the post-trained intrinsic parameter detection subnetwork, and the post-trained depth estimation subnetwork, the p-th reconstructed frame is obtained through photometric reconstruction. During the training of the main motion representation extractor, the network parameters of the post-training intrinsic parameter detection subnetwork and the post-training depth estimation subnetwork are frozen and remain unchanged.

3. The subject motion representation extraction device training method according to claim 1, characterized in that, Before inputting the training frames into the motion representation extractor of the subject to be trained, the following steps are also included: Generate a training set based on first-person perspective videos; Based on the training set, video frame reconstruction training is performed on the extrinsic detection subnetwork, intrinsic detection subnetwork, and depth estimation subnetwork to obtain the trained extrinsic detection subnetwork, the trained intrinsic detection subnetwork, and the trained depth estimation subnetwork.

4. The subject motion representation extraction device training method according to claim 3, characterized in that, A training set is generated based on first-person perspective videos, including: The first-view video is preprocessed to obtain a training set containing multiple video frames; the preprocessing includes frame rate adjustment and frame extraction.

5. The subject motion representation extraction device training method according to claim 3, characterized in that, Based on the training set, video frame reconstruction training is performed on the extrinsic detection subnetwork, intrinsic detection subnetwork, and depth estimation subnetwork, including: The extrinsic parameter detection subnetwork, intrinsic parameter detection subnetwork, and depth estimation subnetwork are trained for video frame reconstruction using every three consecutive training frames as training units. The extrinsic parameter detection subnetwork is used to determine the camera extrinsic parameter changes between two frames, the intrinsic parameter detection subnetwork is used to determine the camera intrinsic parameters, and the depth estimation subnetwork is used to determine the depth map corresponding to the image frame.

6. The subject motion representation extraction device training method according to claim 5, characterized in that, Using three consecutive training frames as training units, the extrinsic parameter detection subnetwork, intrinsic parameter detection subnetwork, and depth estimation subnetwork are trained for video frame reconstruction, including: The t-th training frame and the (t-1)-th training frame are respectively input into the extrinsic parameter detection subnetwork and the intrinsic parameter detection subnetwork, and the t-th training frame is input into the depth estimation subnetwork; Based on the outputs of the extrinsic detection subnetwork, the intrinsic detection subnetwork, and the depth estimation subnetwork, a first reconstructed frame is obtained through photometric reconstruction. The first reconstructed frame is the t-th reconstructed frame obtained based on the (t-1)-th training frame. The t-th training frame and the (t+1)-th training frame are respectively input into the extrinsic parameter detection subnetwork and the intrinsic parameter detection subnetwork, and the t-th training frame is input into the depth estimation subnetwork; Based on the outputs of the extrinsic detection subnetwork, the intrinsic detection subnetwork, and the depth estimation subnetwork, a second reconstructed frame is obtained through photometric reconstruction. The second reconstructed frame is the t-th reconstructed frame obtained based on the (t+1)-th training frame. The extrinsic parameter detection subnetwork, the intrinsic parameter detection subnetwork, and the depth estimation subnetwork are trained for photometric reconstruction using a second loss function; the second loss function is constructed based on the difference between the t-th training frame and the first reconstructed frame, and the difference between the t-th training frame and the second reconstructed frame.

7. The subject motion representation extraction method training method according to claim 6, characterized in that, The second loss function is constructed based on the structural similarity metric and L1 loss between the t-th training frame and the first reconstructed frame, the structural similarity metric and L1 loss between the t-th training frame and the second reconstructed frame, and the disparity smoothness loss of the t-th training frame.

8. The subject motion representation extraction method training method according to any one of claims 1 to 7, characterized in that, After adjusting the parameters of the motion representation extractor of the subject to be trained based on the total loss, the method further includes: The initial training subject motion representation extractor is obtained, and the initial training subject motion representation extractor is optimized by visual odometry to obtain the final subject motion representation extractor.

9. The subject motion representation extraction device training method according to claim 8, characterized in that, The initial trained subject motion representation extractor is optimized using visual odometry estimation, including: The odometry dataset is input into the initial trained subject motion representation extractor, and the output of the initial trained subject motion representation extractor is the input of the encoder. The encoded features corresponding to each image frame are obtained from the output of the encoder, and the encoded features are input into the multilayer perceptron to obtain motion difference information. The subject motion representation extractor after initial training is optimized using a third loss function; the third loss function is constructed based on the difference in encoded features between two adjacent frames and the motion difference information between the two frames.

10. A method for perceiving motion information of an embodied intelligent agent, characterized in that, include: Acquire the target image frame currently captured by the embodied intelligent agent from a first-person perspective; The target image frame is input into the subject motion representation extractor to obtain the six degrees of freedom between the target image frame and the next image frame predicted by the subject motion representation extractor; the subject motion representation extractor is trained according to the subject motion representation extractor training method according to any one of claims 1 to 9. The motion information of the embodied intelligent agent is perceived based on the six degrees of freedom.

11. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the subject motion representation extractor training method as described in any one of claims 1 to 9, or the embodied intelligent agent motion information perception method as described in claim 10.

12. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the subject motion representation extractor training method as described in any one of claims 1 to 9, or the embodied intelligent agent motion information perception method as described in claim 10.

13. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the subject motion representation extractor training method as described in any one of claims 1 to 9, or the embodied intelligent agent motion information perception method as described in claim 10.