Camera parameter prediction method, model optimization method, equipment, medium and product

Abstract

The invention discloses a camera parameter prediction method, a model optimization method, equipment, a medium and a product. The method comprises the following steps: acquiring a first video; performing inter-frame attention operation on at least two video frames in the first video to obtain camera parameters respectively corresponding to the at least two video frames; wherein the inter-frame attention operation is used for fusing the at least two video frames based on a sequential relationship between the at least two video frames, and the camera parameters are used for determining a predicted motion trajectory of the motion object corresponding to the first video. When camera parameter prediction is carried out, time sequence modeling based on inter-frame attention operation is introduced, so that information among a plurality of video frames can be mutually supplemented, parameter prediction deviation caused by isolated calculation of a single-frame image is reduced, and the prediction accuracy of the camera parameters is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a camera parameter prediction method, model optimization method, device, medium and product. Background Technology

[0002] Camera parameters mainly include intrinsic and extrinsic parameters, which are used to represent the camera's attitude and position in three-dimensional space, i.e., camera pose.

[0003] In related technologies, camera parameters in a single image can be predicted using a visual model, such as training a convolutional neural network to learn the mapping from image to camera parameters; then, a specified image is input into the trained convolutional neural network, which outputs the camera parameters of the specified image.

[0004] However, the relevant technologies extract visual information from isolated images to predict camera parameters, resulting in low accuracy in predicting camera parameters. Summary of the Invention

[0005] This application provides a camera parameter prediction method, a model optimization method, a device, a medium, and a product, the technical solution of which is as follows: On the one hand, a camera parameter prediction method is provided, the method comprising: Get the first video; Perform inter-frame attention operation on at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames respectively; The inter-frame attention operation is used to fuse the at least two video frames based on the temporal relationship between them, and the camera parameters are used to determine the predicted motion trajectory of the moving object corresponding to the first video.

[0006] On the other hand, a model optimization method is provided, the method comprising: Configure an inter-frame attention layer in the prediction model; The inter-frame attention layer performs inter-frame attention operations on at least two video frames in the video to predict the camera parameters corresponding to the at least two video frames respectively. The inter-frame attention operation is used to fuse the at least two video frames based on the temporal relationship between them.

[0007] On the other hand, a camera parameter prediction device is provided, the device comprising: The acquisition module is used to acquire the first video. The first prediction module is used to perform inter-frame attention operation on at least two video frames in the first video to obtain camera parameters corresponding to the at least two video frames respectively. The inter-frame attention operation is used to fuse the at least two video frames based on the temporal relationship between them, and the camera parameters are used to determine the predicted motion trajectory of the moving object corresponding to the first video.

[0008] On the other hand, a model optimization apparatus is provided, the apparatus comprising: The configuration module is used to configure inter-frame attention layers in the prediction model; The second prediction module is used to perform inter-frame attention operations on at least two video frames in the video through the inter-frame attention layer in order to predict the camera parameters corresponding to the at least two video frames respectively. The inter-frame attention operation is used to fuse the at least two video frames based on the temporal relationship between them.

[0009] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described camera parameter prediction method or model optimization method.

[0010] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described camera parameter prediction method or model optimization method.

[0011] On the other hand, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, and a processor reading from the computer-readable storage medium and executing the computer program to implement the above-mentioned camera parameter prediction method or model optimization method.

[0012] The beneficial effects of the technical solutions provided in this application include at least the following: The camera parameter prediction method provided in this application obtains camera parameters corresponding to multiple video frames by performing inter-frame attention operations on multiple consecutive video frames in a first video. On the one hand, when performing camera parameter prediction, temporal modeling based on inter-frame attention operations is introduced, allowing information between multiple video frames to complement each other, reducing parameter prediction bias caused by isolated calculation of single-frame images, and improving the accuracy of camera parameter prediction. On the other hand, single-frame image analysis in related technologies can only obtain camera parameters at a certain instant and cannot perceive the correlation of camera parameters in the time dimension. This application uses inter-frame attention operations to upgrade camera parameter prediction from image analysis to video sequence analysis, accurately capturing camera motion in the video and laying a solid foundation for generating accurate motion trajectories.

[0013] The model optimization method provided in this application predicts camera parameters by configuring an inter-frame attention layer in the prediction model and using this layer to perform inter-frame attention operations on at least two video frames. By configuring an inter-frame attention layer for the model, the model can capture the temporal dependencies between video frames, thereby fully utilizing the complementarity of multi-frame information when predicting camera parameters. This reduces the fluctuations or errors in prediction parameters caused by isolated processing of single-frame images, and improves the accuracy and robustness of camera parameter prediction. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a schematic diagram of a camera parameter prediction method provided in an exemplary embodiment of this application; Figure 2 This is a schematic diagram of a computer system provided in an exemplary embodiment of this application; Figure 3 This is a flowchart of a camera parameter prediction method provided in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the acquisition of a first feature representation provided in an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the fusion of third-person perspective video to obtain frame feature representations, provided in an exemplary embodiment of this application. Figure 6 This is a flowchart of a camera parameter prediction method provided in another exemplary embodiment of this application; Figure 7 This is a schematic diagram illustrating the acquisition of a first feature representation provided in another exemplary embodiment of this application; Figure 8 This is a flowchart of a camera parameter prediction method provided in yet another exemplary embodiment of this application; Figure 9 This is a flowchart of a camera parameter prediction method provided in another exemplary embodiment of this application; Figure 10 This is a schematic diagram of the structure of a camera parameter prediction model provided in an exemplary embodiment of this application; Figure 11 This is a flowchart of a model optimization method provided in an exemplary embodiment of this application; Figure 12 This is a structural block diagram of a camera parameter prediction device provided in an exemplary embodiment of this application; Figure 13 This is a structural block diagram of a model optimization apparatus provided in an exemplary embodiment of this application; Figure 14 This is a structural block diagram of a computer device provided in an exemplary embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In this application, the terms "first" and "second" are used to distinguish between identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first" and "second", nor is there any limitation on the quantity or execution order.

[0018] Camera parameters mainly include intrinsic and extrinsic parameters, used to represent the camera's pose and position in three-dimensional space, i.e., camera pose. In related technologies, camera parameters in a single image can be predicted using a visual model, such as training a convolutional neural network to learn the mapping from images to camera parameters; then, a specified image is input into the trained convolutional neural network, which outputs the camera parameters for that image. However, these technologies only extract visual information from isolated images to predict camera parameters, resulting in low prediction accuracy.

[0019] Based on this, this application provides a camera parameter prediction method, please refer to... Figure 1 It illustrates a method for predicting camera parameters in autonomous driving videos, which includes the following process: like Figure 1 As shown, the autonomous driving video 101 is a video acquired from the first-person perspective of vehicle A. First, T video frames 102 are acquired from the autonomous driving video 101, where T is an integer greater than 1. Then, inter-frame attention operations are performed on the T video frames 102 to obtain the camera parameters corresponding to each of the T video frames 102, i.e., T camera parameters 102. The prediction process of the camera parameters is explained below using the i-th video frame as an example, where i is a positive integer, i ≤ T: When predicting the camera parameters of the i-th video frame, an inter-frame attention operation is performed on the i-th video frame and other video frames to fuse the i-th video frame and other video frames, such as obtaining a first feature representation of the i-th video frame, which includes features of the i-th video frame and other video frames. Then, the camera parameters of the i-th video frame are predicted based on the first feature representation of the i-th video frame.

[0020] By traversing T video frames 102 according to the above prediction process for the i-th video frame, T camera parameters 102 can be obtained.

[0021] In some embodiments, the predicted T camera parameters 102 can be used to determine the motion trajectory of vehicle A in the autonomous driving video 101. The T camera parameters 102 include camera extrinsic parameters corresponding to T video frames, such as "[R1|t1], ..., [R1|t1]". i |t i ]、[R T |t T ]”, where [R i |t i ] represents the camera extrinsic parameters corresponding to the i-th video frame. i This refers to the camera rotation matrix (3×3) in the i-th video frame, t i It refers to the camera translation vector (3×1) in the i-th video frame.

[0022] For example, the global camera coordinates in the world coordinate system are calculated based on the camera's extrinsic parameters. The global camera coordinates corresponding to T video frames are concatenated in timestamp order to obtain a position change sequence as the camera motion trajectory. Then, the motion trajectory of vehicle A is determined based on the camera motion trajectory, such as using the camera motion trajectory as the motion trajectory of vehicle A. In autonomous driving scenarios, the camera can be considered to be fixedly mounted on vehicle A (e.g., behind the windshield, on the roof, etc.), therefore, the camera motion trajectory and the motion trajectory of vehicle A have a fixed geometric relationship, and thus the camera motion trajectory can be equated to the motion trajectory of vehicle A.

[0023] In summary, the camera parameter prediction method provided in this application reduces the parameter prediction bias caused by isolated calculation of a single frame image by fusing T video frames based on temporal relationships through inter-frame attention operations, making camera parameter prediction based on the overall context of the video and improving the accuracy of the predicted camera parameters.

[0024] Furthermore, the camera parameter prediction method provided in this application can be applied to vehicle trajectory prediction scenarios. When a vehicle is in motion, since the camera is fixedly mounted on the vehicle, there is a fixed geometric relationship between the camera's motion trajectory and the vehicle's motion trajectory; therefore, camera motion can be used to simulate vehicle motion. Using the camera parameter prediction method provided in this application, inter-frame attention operations are employed to upgrade camera parameter prediction from image analysis to video sequence analysis. The predicted multiple camera parameters accurately capture the camera motion in the video, thereby improving the accuracy of the simulated vehicle trajectory.

[0025] Next, we will introduce the computer system provided in the application.

[0026] Figure 2 A schematic diagram of a computer system provided in an exemplary embodiment of this application is shown. This computer system can implement a system architecture for a camera parameter prediction method. The computer system includes a terminal 210 and a server 220.

[0027] In some embodiments, the terminal 210 may be a smartphone, tablet computer, laptop computer, desktop computer, smart home appliance, smart vehicle terminal, AR device, action camera, etc., but is not limited thereto. The terminal 210 has a client application for a first application installed. Exemplarily, the first application may be a traditional application, a cloud application, a mini-program or application module within a host application, or a web platform; this embodiment does not limit its scope. Exemplarily, the first application includes, but is not limited to: model training applications, model evaluation applications, AR applications, vehicle simulation applications, browser applications, instant messaging applications, short video applications, etc., and is not limited here.

[0028] Terminal 210 is connected to server 220 via wireless or wired network.

[0029] In some embodiments, server 220 provides background services for the client of the first application installed on the terminal. It is worth noting that server 220 can be a standalone physical server, a server cluster or distributed system consisting of at least two physical servers, a cloud server providing basic cloud computing services, or a node in a blockchain system.

[0030] The camera parameter prediction method provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 2Taking the computer system shown as an example, the camera parameter prediction method can be executed by terminal 210 (for example, the camera parameter prediction method is executed by the client of the first application installed and running in terminal 210), or by server 220, or by interaction and cooperation between terminal 210 and server 220. This application does not limit this.

[0031] The following example illustrates the camera parameter prediction method executed through the interaction and cooperation of terminal 210 and server 220: Figure 2 As shown, terminal 210 sends the first video to server 220. After receiving the first video, server 220 inputs at least two video frames of the first video into a camera parameter prediction model deployed in server 220. The camera parameter prediction model outputs the camera parameters corresponding to the at least two video frames (i.e., the camera parameters of the first video). The camera parameter prediction model includes an inter-frame attention layer. After inputting at least two video frames into the camera parameter prediction model, the inter-frame attention layer performs inter-frame attention operations on the at least two video frames to obtain the camera parameters of the first video.

[0032] In some embodiments, the camera parameter prediction model described above is deployed in terminal 210, meaning that terminal 210 independently completes the camera parameter prediction process. In some embodiments, the first video described above is stored in server 220, meaning that server 220 independently completes the camera parameter prediction process. This application does not limit this aspect.

[0033] In some embodiments, the camera parameter prediction method provided in this application can be used to evaluate a video generation model (for generating first-person perspective videos): like Figure 2As shown, technicians can trigger an evaluation operation on the video generation model in the model evaluation application of terminal 210. Terminal 210 responds to the evaluation operation by sending an evaluation request to server 220. After receiving the evaluation request, server 220 evaluates its deployed video generation model using an evaluation dataset. The evaluation dataset includes a reference video and the corresponding control signals. The reference video is a video captured from the first-person perspective of a moving object. For example, the reference video can be a video captured by a vehicle-mounted camera on the windshield of a real vehicle. The control signals are vehicle control signals, such as sensor signals (e.g., LiDAR, camera, millimeter-wave radar, etc.) and vehicle status signals (e.g., vehicle speed, steering angle, acceleration, etc.), which are not limited here. In some embodiments, the process of evaluating the video generation model includes: first, inputting the first video frame corresponding to the reference video and the control signals into the video generation model, and the video generation model generates a first video; then, calculating the reference motion trajectory corresponding to the reference video and the predicted motion trajectory corresponding to the first video. The process of determining the reference motion trajectory and the predicted motion trajectory includes: server 220 deploying a camera parameter prediction model, which is used to predict camera extrinsic parameters in the video. The reference video is input into the camera parameter prediction model, which outputs the camera extrinsic parameters corresponding to at least two video frames of the reference video. Then, based on the camera extrinsic parameters corresponding to at least two video frames of the reference video, the reference motion trajectory of the moving object is determined. Alternatively, the first video is input into the camera parameter prediction model, which outputs the camera extrinsic parameters corresponding to at least two video frames of the first video. Then, based on the camera extrinsic parameters corresponding to at least two video frames of the first video, the predicted motion trajectory of the moving object is determined.

[0034] Finally, based on the difference between the reference motion trajectory and the predicted motion trajectory, the model evaluation result is determined (e.g., the average of the differences in motion trajectories corresponding to multiple reference videos is used as the model evaluation result). In some embodiments, after obtaining the model evaluation result, the server 220 feeds the model evaluation result back to the terminal 210.

[0035] In some embodiments, if the model evaluation result does not meet the evaluation quality requirements (e.g., the average difference of motion trajectories corresponding to multiple reference videos is greater than a preset trajectory difference), the server 220 retrains the video generation model. In other embodiments, the server 220 retrains the video generation model in response to a training request sent by the terminal 210. For example, after a technician views the model evaluation result of the video generation model through the terminal 210, if the technician believes that the model evaluation result does not meet expectations, he / she can click the retraining control displayed on the terminal 210, and the terminal 210 will send a training request to the server 220 in response to the click operation.

[0036] In related technologies, video generation models typically evaluate video generation quality by comparing the similarity of a first video and a reference video in visual features or semantic information, which is a relatively limited evaluation method. The camera parameter prediction method provided in this application, however, predicts camera parameters to predict the motion trajectory of the object corresponding to the video. By comparing the difference in motion trajectories of the moving object corresponding to the first video and the reference video, it effectively evaluates the deviation between the predicted motion trajectory and the reference motion trajectory expected by the control signal. This overcomes the limitations of evaluation methods in related technologies and achieves accurate evaluation of the signal following performance of the video generation model.

[0037] In some embodiments, the camera parameter prediction method provided in this application can be used to train a video generation model: like Figure 2 As shown, terminal 210 has a model training application installed and running, which can be used to train the video generation model. The training process of the video generation model includes the following steps: Technicians input the training dataset into terminal 210, and terminal 210 sends the training dataset to server 220. The training dataset includes a reference video and the control signals corresponding to the reference video.

[0038] A video generation model to be trained is deployed in server 220. Server 220 trains the video generation model using a training dataset. For example, the first video frame corresponding to the reference video and the control signal are input into the video generation model, and the video generation model generates the first video; then, the reference motion trajectory corresponding to the reference video and the predicted motion trajectory corresponding to the first video are determined; based on the difference between the reference motion trajectory and the predicted motion trajectory, the motion trajectory loss is determined; finally, based on the motion trajectory loss, the video generation model is trained.

[0039] In some embodiments, when training the video generation model, the server 220 can feed back the training results to the terminal 210. For example, the server 220 can feed back the motion trajectory loss calculated in each training process to the terminal 210. The terminal 210 displays the motion trajectory loss in a visual manner on the terminal screen, such as presenting the trend of the motion trajectory loss with the number of training iterations as a loss curve. Technicians can observe the training progress and status of the video generation model through the loss curve.

[0040] The above examples of application scenarios are merely illustrative and are not intended to limit the scope of this application.

[0041] Next, the flow of the camera parameter prediction method provided in the embodiments of this application will be described.

[0042] Figure 3This is a flowchart of a camera parameter prediction method provided in an embodiment of this application. The method is executed by a computer device, which may be... Figure 2 The terminal 210 and / or server 220 are shown. The method includes the following steps 310 and 320.

[0043] Step 310: Obtain the first video.

[0044] The first video is the one for which camera parameter prediction is to be performed.

[0045] In some embodiments, the first video is a video acquired from the target perspective of a moving object. The target perspective includes, but is not limited to, at least one of a first-person perspective, a third-person perspective, etc.

[0046] In some embodiments, the first video is a video acquired from the first-person perspective of the moving object. That is, the camera perspective corresponding to the first video is the first-person perspective of the moving object. The first video can also be referred to as a first-person perspective video. Here, the first-person perspective refers to the observation perspective from the moving object itself. For example, if the moving object is a person, the video content of the first video can be regarded as the content observed by the human eye, or the content captured by a camera worn by the person. For example, if the moving object is a vehicle, the video content of the first video can be regarded as the content captured by an onboard camera mounted in the vehicle; if the first video is an autonomous driving video, in the field of autonomous driving, autonomous driving video is a kind of first-person perspective video, which is a video acquired from the first-person perspective of the vehicle. For example, the video captured by the camera on the windshield of the vehicle during autonomous driving is an autonomous driving video.

[0047] In some embodiments, a moving object refers to an object with the ability to move. For example, moving objects include, but are not limited to: vehicles (such as cars, motorcycles, bicycles, electric vehicles, trucks, buses, trains, ships, airplanes, drones, etc.), people (such as pedestrians, cyclists, athletes, etc.), machines (such as robots, etc.), and virtual objects (such as virtual characters and virtual vehicles operated by the user in an AR scene). This application mainly uses a vehicle as an example to illustrate the moving object.

[0048] In some embodiments, the method of acquiring the first video includes, but is not limited to, at least one of the following: (1) Videos automatically generated by a video generation model.

[0049] For example, the video generation model is a pre-trained machine learning model for generating autonomous driving videos, which can generate autonomous driving videos as the first video.

[0050] (2) Videos acquired through video acquisition equipment.

[0051] For example, video capture devices include, but are not limited to, vehicle-mounted cameras, AR devices, action cameras, etc. First-person perspective video can be captured by these video capture devices and used as the first video.

[0052] (3) Virtual scene video rendered by virtual engine.

[0053] In some embodiments, the virtual scene video is video captured from the first-person perspective of a moving object. The moving object includes at least one of real objects and virtual objects.

[0054] In some embodiments, the video frame in the first video includes moving objects. For example, if the first video is a video of a vehicle moving forward, captured by an in-vehicle camera mounted on the driver's cab of the vehicle (the moving object), then the first video includes the front of the vehicle, such as the windshield and dashboard. In other embodiments, the video frame in the first video does not include moving objects. For example, if the first video is a video of a vehicle moving forward, captured by an in-vehicle camera mounted on the windshield of the vehicle, then the first video typically only includes the environment in front of the vehicle and does not include the vehicle itself.

[0055] Step 320: Perform inter-frame attention operation on at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames respectively.

[0056] Inter-frame attention operations are used to fuse at least two video frames based on the temporal relationship between them. Inter-frame attention operations include, but are not limited to, at least one of self-attention and cross-attention operations. This application primarily uses self-attention as an example for illustration.

[0057] Temporal relationships refer to the temporal connections between at least two video frames in the first video. For example, temporal relationships are based on the continuous temporal order of video frames to ensure the coherence of dynamic information changes in the first video. Specifically, by utilizing temporal relationships, inter-frame attention operations, when fusing video frames, consider factors such as the temporal order, time intervals, motion continuity, and causal dependencies between frames, thereby more accurately predicting camera parameters and ultimately determining the predicted motion trajectory of moving objects.

[0058] In some embodiments, the timing relationship includes, but is not limited to, at least one of the following: (1) The temporal relationship between the two video frames in the time series.

[0059] The temporal sequence refers to the sequential arrangement of at least two video frames along the timeline of a first video in the order they are played. In some embodiments, the temporal sequence is used to determine the temporal dependency between at least two video frames in order to merge the at least two video frames.

[0060] (2) The time interval relationship between at least two video frames.

[0061] A time interval relationship refers to the timestamp difference between at least two video frames. In some embodiments, the time interval relationship is used to determine the rate of motion change between at least two video frames in order to fuse at least two video frames.

[0062] (3) The dynamic continuity relationship between at least two video frames.

[0063] Dynamic continuity relationships are used to indicate the continuous influence of the state information (including but not limited to position, posture, and motion trend) of a moving object in a preceding video frame on the state information of a moving object in a subsequent video frame across at least two video frames. In some embodiments, dynamic continuity relationships are used to capture the motion coherence of a moving object in a first video to fuse at least two video frames.

[0064] (4) Causal dependency between at least two video frames.

[0065] Causal dependencies are used to indicate the influence of preceding video frames on subsequent video frames. For example, in at least two video frames, the scene state (including but not limited to the motion state of moving objects and the background environment state) of the subsequent video frame changes based on the physical motion laws of the scene state of the preceding video frame. In some embodiments, causal dependencies are used to infer changes in camera parameters to fuse at least two video frames.

[0066] (5) The relationship between inter-frame feature similarity and time.

[0067] The relationship between inter-frame feature similarity and time refers to the trend of change in the similarity between the visual features (including but not limited to pixel values, feature points, and semantic labels) of at least two video frames over time. In some embodiments, at least two video frames are fused based on the relationship between inter-frame feature similarity and time.

[0068] The following explanation uses the i-th video frame as an example. In some embodiments, the inter-frame attention operation is used to fuse information from other video frames associated with the i-th video frame when predicting the camera parameters of the i-th video frame. These other video frames associated with the i-th video frame refer to: all video frames other than the i-th video frame in at least two video frames; or, a subset of video frames other than the i-th video frame in at least two video frames. In some embodiments, when predicting the camera parameters of the i-th video frame, the inter-frame attention operation is performed on the i-th video frame and all video frames in the first video frame sequence to obtain the camera parameters of the i-th video frame. The first video frame sequence includes all video frames other than the i-th video frame in at least two video frames. Alternatively, when predicting the camera parameters of the i-th video frame, the inter-frame attention operation is performed on all video frames in the i-th video frame and the second video frame sequence to obtain the camera parameters of the i-th video frame. The second video frame sequence includes a subset of video frames other than the i-th video frame in at least two video frames.

[0069] In some embodiments, the second video frame sequence refers to a sequence of video frames that are temporally associated with the i-th video frame among at least two video frames. If the timestamp difference between the k-th video frame and the i-th video frame is less than a preset timestamp difference, it indicates that the k-th video frame and the i-th video frame are temporally associated, and then the k-th video frame is added to the second video frame sequence.

[0070] Among them, camera parameters refer to the parameters of the camera corresponding to the video frame. In this application, the understanding of camera includes the following two: (1) If the first video is a video obtained by a real physical acquisition device, then the camera is also the physical acquisition device. For example, if the first video is an autonomous driving video of a vehicle, the camera is an in-vehicle camera; (2) If the first video is not physically acquired, but generated by a model (such as the video generation model involved in this application), then the camera refers to a projection model in a mathematical sense, which is used to provide observation perspective information in the video frame generated by the model.

[0071] In some embodiments, an inter-frame attention operation is performed on at least two video frames in the first video using a camera parameter prediction model to obtain camera parameters corresponding to the at least two video frames respectively. The camera parameter prediction model includes an inter-frame attention layer, which is used to perform the inter-frame attention operation.

[0072] The camera parameter prediction model is a pre-trained machine learning model. For example, the camera parameter prediction model is built on the VGGT (Visual Geometry Grounded Transformer) model, such as by configuring an inter-frame attention layer in the VGGT model to obtain the camera parameter prediction model.

[0073] In some embodiments, inter-frame attention and target attention operations are performed on at least two video frames in the first video to obtain camera parameters corresponding to the at least two video frames respectively. The target attention operation includes at least one of intra-frame attention and global attention operations. Step 320 above also includes at least one of the following: Case 1: Perform intra-frame attention operation on at least two video frames in the first video to obtain the second feature representations corresponding to the at least two video frames respectively; perform inter-frame attention operation based on the at least two second feature representations to obtain the camera parameters corresponding to the at least two video frames respectively.

[0074] Intra-frame attention operations are used to fuse feature representations within a single video frame. Illustrated, intra-frame attention is an attention mechanism that fuses and enhances features within a single video frame. Its core objective is to improve the effectiveness of feature representations in a single video frame by modeling the relationships between different feature elements within that frame, highlighting key information, and suppressing redundant information.

[0075] In some embodiments, the camera parameter prediction model further includes an intra-frame attention layer; the intra-frame attention layer is used to perform intra-frame attention operations. Intra-frame attention operations are performed on at least two video frames in the first video by the intra-frame attention layer to obtain second feature representations corresponding to at least two video frames respectively; based on the at least two second feature representations, inter-frame attention operations are performed by the inter-frame attention layer to obtain camera parameters corresponding to at least two video frames respectively.

[0076] For example, an intra-frame attention layer refers to a network layer used to perform intra-frame attention operations, which include, but are not limited to, at least one of self-attention operations and multi-head self-attention operations. This application mainly uses self-attention operations as an example for illustration. Intra-frame attention operations can dynamically adjust the importance of each region according to the scene by weighted fusion of feature representations within a single frame. For example, in complex traffic scenes, intra-frame attention can highlight the features of key regions such as pedestrians, obstacles, or traffic lights in front of the vehicle, suppress the interference of background noise, significantly improve the local discriminative power of features, and provide more accurate spatial information for subsequent camera extrinsic parameter prediction.

[0077] In some embodiments, a global attention operation is performed on at least two video frames in the first video to obtain a third feature representation corresponding to each of the at least two video frames; an intra-frame attention operation is performed on each of the at least two third feature representations to obtain a second feature representation corresponding to each of the at least two video frames.

[0078] Global attention operations are used to fuse feature representations in the i-th video frame, as well as to fuse feature representations of the i-th video frame with those of other video frames. For example, a global attention operation is an attention mechanism that performs feature fusion at the level of the entire video frame sequence. Its core objective is to break the limitations of single or local frames, allowing the features of each video frame to establish associations and fuse with the features of all frames in the sequence (including itself), thereby capturing the global contextual information of the video.

[0079] In some embodiments, the camera parameter prediction model further includes a global attention layer; the global attention layer is used to perform global attention operations. The global attention layer performs global attention operations on at least two video frames in the first video to obtain third feature representations corresponding to at least two video frames respectively; the intra-frame attention layer performs intra-frame attention operations on at least two third feature representations respectively to obtain second feature representations corresponding to at least two video frames respectively.

[0080] Schematic, a global attention layer refers to a network layer used to perform global attention operations, which include, but are not limited to, at least one of self-attention operations and cross-attention operations. This application mainly uses self-attention operations as an example for illustration.

[0081] Scenario 2: Perform a global attention operation on at least two video frames in the first video to obtain the third feature representations corresponding to the at least two video frames respectively; perform an inter-frame attention operation based on the at least two third feature representations to obtain the camera parameters corresponding to the at least two video frames respectively.

[0082] In some embodiments, a global attention operation is performed on at least two video frames in the first video through a global attention layer to obtain a third feature representation corresponding to each of the at least two video frames; based on the at least two third feature representations, an inter-frame attention operation is performed through an inter-frame attention layer to obtain camera parameters corresponding to each of the at least two video frames.

[0083] In some embodiments, intra-frame attention operations are performed on at least two video frames in the first video to obtain second feature representations corresponding to at least two video frames respectively; global attention operations are performed on at least two second feature representations to obtain third feature representations corresponding to at least two video frames respectively.

[0084] In some embodiments, an intra-frame attention operation is performed on at least two video frames in the first video through an intra-frame attention layer to obtain a second feature representation corresponding to each of the at least two video frames; and a global attention operation is performed on the at least two second feature representations through a global attention layer to obtain a third feature representation corresponding to each of the at least two video frames.

[0085] In some embodiments, camera parameters include at least one of camera intrinsic parameters and camera extrinsic parameters.

[0086] Camera intrinsic parameters include focal length, principal point, etc. For example, focal length represents the distance from the camera's optical center (lens center) to the image sensor (imaging plane), and typically has two components. and These correspond to the focal lengths along the x and y axes of the video frame, respectively; the principal point is the intersection of the optical axis (a line perpendicular to the imaging plane from the camera center) and the image plane, usually denoted as . That is, the pixel coordinates of the principal point in the video frame.

[0087] Camera extrinsic parameters include rotation matrices and translation vectors. For example, the rotation matrix describes the orientation of the camera coordinate system relative to the world coordinate system (i.e., how the camera rotates), and is typically a 3×3 orthogonal matrix, denoted as [matrix name missing]. The translation vector describes the position of the camera coordinate system origin in the world coordinate system. It is usually a three-dimensional vector, denoted as . .

[0088] In some embodiments, after predicting the camera parameters corresponding to at least two video frames, the camera parameters can be used in at least one of the following scenarios: (1) Camera parameters are used to determine the predicted motion trajectory of the moving object corresponding to the first video.

[0089] In some embodiments, the predicted motion trajectory of a moving object corresponding to a first video is determined based on camera extrinsic parameters corresponding to at least two video frames. For example, the camera motion trajectory corresponding to the first video is determined based on camera extrinsic parameters corresponding to at least two video frames; this camera motion trajectory is then used as the predicted motion trajectory of the moving object.

[0090] It should be noted that the number of moving objects corresponding to the first video (i.e., the moving objects whose trajectories are to be predicted) can be single or multiple. This application does not limit this. The following explanation mainly uses a single moving object as an example.

[0091] (2) Camera parameters are used to generate other perspective videos corresponding to the first video.

[0092] In some embodiments, where the first video refers to a first-person perspective video, a third-person perspective video corresponding to the first video can be generated based on the camera extrinsic parameters corresponding to at least two video frames in the first video. The third-person perspective refers to the viewpoint from which a moving object is observed, such as a camera's overhead view of the moving object. For example, the target video generation model is a pre-trained neural network model for generating third-person perspective videos. The camera extrinsic parameters corresponding to at least two video frames in the first video are input into the target video generation model, and the target video generation model reconstructs the third-person perspective video corresponding to the first video.

[0093] (3) Camera parameters are used to calibrate the parameters of the video acquisition equipment.

[0094] Video capture devices include, but are not limited to, vehicle-mounted cameras, action cameras, and AR devices.

[0095] In some embodiments, the first video is a video captured by a video acquisition device. The camera intrinsics corresponding to at least two video frames in the predicted first video are matched against the calibration camera intrinsics of the video acquisition device. If the difference between the predicted camera intrinsics and the calibration camera intrinsics is greater than a preset difference, it indicates that the video acquisition device needs parameter calibration. For example, in autonomous driving scenarios, the parameters of an onboard camera may drift due to temperature changes, mechanical vibration, or long-term use. The method provided in this application can predict camera parameters in real time from the video captured by the onboard camera, and match the predicted camera intrinsics with the calibration camera intrinsics of the onboard camera. If they differ, it indicates that the onboard camera needs parameter calibration or repair.

[0096] It should be noted that the examples of camera parameter application scenarios above are merely illustrative and this application does not limit their application. The following explanation primarily uses the example of using camera parameters to determine the predicted motion trajectory of a moving object corresponding to the first video.

[0097] In summary, the camera parameter prediction method provided in this application introduces temporal modeling based on inter-frame attention operations during camera parameter prediction. This allows information from multiple video frames to complement each other, reducing prediction bias caused by isolated calculations of single-frame images and improving the accuracy of camera parameter prediction. On the other hand, single-frame image analysis in related technologies can only obtain camera parameters at a single instant and cannot perceive the correlation of camera parameters over time. This application upgrades camera parameter prediction from image analysis to video sequence analysis by utilizing inter-frame attention operations, accurately capturing camera motion in the video and laying a solid foundation for generating accurate motion trajectories.

[0098] In some embodiments, camera parameters for at least two video frames are obtained by performing an inter-frame attention operation on the camera feature representations of at least two video frames. For example, the above... Figure 3 Step 320 in the embodiment further includes the following steps 321 and 322.

[0099] Step 321: Obtain the camera feature representations corresponding to at least two video frames.

[0100] At least two video frames can be represented as T video frames, where T is an integer greater than 1. The at least two video frames include the i-th video frame and the j-th video frame, where i and j are positive integers. i ≤ T, j ≤ T.

[0101] Camera feature representation is a feature representation associated with the camera pose corresponding to a video frame. Here, camera pose refers to the position and orientation of the camera corresponding to the video frame.

[0102] In some embodiments, step 321 further includes the following steps: Step 3211: Perform an embedding operation on at least two video frames in the first video to obtain the frame feature representations corresponding to the at least two video frames respectively.

[0103] In some embodiments, the frame feature representation includes a first image feature representation and a first camera feature representation. The first image feature representation is used to characterize the image content in the video frame, and the first camera feature representation is a feature representation associated with the camera pose corresponding to the video frame.

[0104] In some embodiments, an embedding operation is performed on at least two video frames in the first video to obtain first image feature representations corresponding to at least two video frames respectively; the first image feature representations corresponding to at least two video frames are then concatenated with a first camera feature representation to obtain frame feature representations corresponding to at least two video frames respectively. For example, concatenating the first image feature representation of each video frame with a first camera feature representation yields the frame feature representation of each video frame.

[0105] In some embodiments, the camera parameter prediction model further includes an embedding layer; the embedding layer is used to perform embedding operations. The embedding layer performs embedding operations on at least two video frames in the first video to obtain frame feature representations corresponding to the at least two video frames respectively; based on the at least two frame feature representations, camera feature representations corresponding to the at least two video frames are obtained. Exemplarily, the embedding layer refers to a network layer used to perform embedding operations, and the embedding layer includes, but is not limited to, at least one of convolutional layers, visual Transformer layers, pooling layers, etc.

[0106] In some embodiments, an embedding operation is performed on at least two video frames through the embedding layer of the camera parameter prediction model to obtain a first image feature representation corresponding to each of the at least two video frames; the first image feature representations corresponding to each of the at least two video frames are concatenated with the first camera feature representation to obtain a frame feature representation corresponding to each of the at least two video frames.

[0107] In some embodiments, at least two video frames are all the video frames of the first video frame, or at least two video frames are a subset of the first video frames. For example, if at least two video frames are a subset of the first video frames, then the at least two video frames are obtained after frame filtering of the first video. In some embodiments, frame filtering includes, but is not limited to, at least one of the following methods: (1) Extract T video frames from all video frames of the first video frame at a fixed sampling interval. For example, the first video includes 300 video frames. The fixed sampling interval is set to "1 frame is extracted every 3 frames", and finally 100 video frames are obtained.

[0108] (2) Extract T video frames from all video frames of the first video frame according to the dynamic sampling interval. For example, the first video includes 300 video frames. The motion activity of the moving objects corresponding to the first 150 video frames is greater than the preset activity level (such as the scene of a vehicle making a sharp turn). The sampling interval is set to "1 frame is extracted every 2 frames". The motion activity of the moving objects corresponding to the last 150 video frames is less than or equal to the preset activity level (such as the scene of a vehicle moving at a constant speed). The sampling interval is set to "1 frame is extracted every 4 frames".

[0109] (3) Calculate the pixel difference value between adjacent video frames. If the pixel difference value is less than the preset threshold, it means that the content of the two frames is almost the same (such as a vehicle waiting at a red light), and only the first video frame is retained; if the pixel difference value is greater than or equal to the preset threshold (such as a vehicle turning), then the two adjacent video frames are retained. Traverse all adjacent video frames of all video frames of the first video frame to finally obtain T video frames.

[0110] (4) Perform abnormal frame detection on all video frames of the first video frame to obtain abnormal video frames in all video frames of the first video frame, and remove abnormal video frames from all video frames of the first video frame to obtain T video frames. Among them, abnormal video frames include, but are not limited to, at least one of the following: blurry video frames (such as video frames with image clarity less than a preset clarity can be determined as blurry video frames) and occluded video frames (such as video frames with a field of view occlusion rate of vehicles in the video greater than a preset occlusion rate can be determined as occluded video frames).

[0111] In some embodiments, preprocessing operations are performed on at least two video frames, and embedding operations are performed on the preprocessed at least two video frames to obtain frame feature representations corresponding to at least two video frames respectively. In some embodiments, the preprocessing operations include, but are not limited to, noise suppression (such as removing or reducing grainy noise in nighttime vehicle videos), adaptive lighting adjustment (such as adjusting the brightness and contrast of video frames for backlight, low light, and other scenarios to highlight key elements such as lane lines and traffic signs), and region cropping. In some embodiments, for region cropping: video frames are cropped according to the motion scene of the moving object. For example, in the driver's view frame of an autonomous driving system, the upper 20% of the "roof + sky" area and the lower 10% of the dashboard edge area are cropped, leaving only the middle 70% of the core area of ​​"road surface + vehicles in front + lane lines"; in the view frame of a virtual character in a game, the edge 10% of the UI interface (such as health bars and skill icons) is cropped, leaving only the area of ​​"scene within the character's field of vision + character's hand movements".

[0112] In some embodiments, the time interval between all adjacent video frames in at least two video frames is the same, or there are adjacent video frames in at least two video frames with different time intervals, which is not limited here.

[0113] Step 3212: Based on at least two frame feature representations, obtain the camera feature representations corresponding to at least two video frames respectively.

[0114] In some embodiments, the method for obtaining camera feature representations corresponding to at least two video frames includes at least one of the following: Method 1 The first camera feature representations corresponding to at least two frame feature representations are used as the camera feature representations corresponding to at least two video frames.

[0115] Method 2 Perform target attention operation on the feature representations of at least two frames to obtain camera feature representations corresponding to at least two video frames respectively.

[0116] The target attention operation includes at least one of intra-frame attention operation and global attention operation.

[0117] Intra-frame attention operations are used to fuse feature representations from a single video frame.

[0118] The above Method 2 Including at least one of the following situations: (1) Obtain camera feature representation based on intra-frame attention operation.

[0119] In some embodiments, intra-frame attention operations are performed on at least two frame feature representations to obtain second feature representations corresponding to at least two video frames, the second feature representations including second image feature representations and second camera feature representations; the second camera feature representations corresponding to at least two second feature representations are used as camera feature representations corresponding to at least two video frames.

[0120] In some embodiments, intra-frame attention operations are performed on the feature representations of at least two frames through the intra-frame attention layer of the camera parameter prediction model to obtain the second feature representations corresponding to at least two video frames respectively.

[0121] (2) Obtain camera feature representation based on global attention operation.

[0122] In some embodiments, a global attention operation is performed on at least two frame feature representations to obtain a third feature representation corresponding to at least two video frames, wherein the third feature representation includes a third image feature representation and a third camera feature representation; the third camera feature representation corresponding to at least two third feature representations is used as the camera feature representation corresponding to at least two video frames.

[0123] In some embodiments, a global attention operation is performed on the feature representations of at least two frames through the global attention layer of the camera parameter prediction model to obtain the third feature representations corresponding to at least two video frames respectively.

[0124] (3) Obtain camera feature representation based on global attention operation + intra-frame attention operation.

[0125] In some embodiments, a global attention operation is performed on at least two frame feature representations to obtain a third feature representation corresponding to at least two video frames respectively; an intra-frame attention operation is performed on at least two third feature representations to obtain a second feature representation corresponding to at least two video frames respectively; and the second camera feature representations corresponding to at least two second feature representations are used as the camera feature representations corresponding to at least two video frames respectively.

[0126] In some embodiments, a global attention operation is performed on the feature representations of at least two frames through the global attention layer of the camera parameter prediction model to obtain third feature representations corresponding to at least two video frames respectively. An intra-frame attention operation is then performed on the at least two third feature representations through the intra-frame attention layer of the camera parameter prediction model to obtain second feature representations corresponding to at least two video frames respectively.

[0127] (4) Obtain camera feature representation based on intra-frame attention operation + global attention operation.

[0128] In some embodiments, intra-frame attention operations are performed on at least two frame feature representations to obtain second feature representations corresponding to at least two video frames respectively; global attention operations are performed on at least two second feature representations to obtain third feature representations corresponding to at least two video frames respectively; and the third camera feature representations corresponding to at least two third feature representations are used as camera feature representations corresponding to at least two video frames respectively.

[0129] In some embodiments, intra-frame attention operations are performed on at least two frame feature representations by the intra-frame attention layer of the camera parameter prediction model to obtain second feature representations corresponding to at least two video frames respectively. Global attention operations are then performed on at least two second feature representations by the global attention layer of the camera parameter prediction model to obtain third feature representations corresponding to at least two video frames respectively.

[0130] Step 322: Perform inter-frame attention operation on at least two camera feature representations to obtain camera parameters corresponding to at least two video frames respectively.

[0131] Among them, the inter-frame attention operation is used to fuse at least two camera feature representations based on the attention mechanism.

[0132] In this embodiment, inter-frame attention is a cross-frame attention mechanism fusion method for the first video. Its core is to integrate camera pose information from other frames into each video frame using an attention mechanism when processing camera feature representations for at least two video frames. Specifically, the core of the attention mechanism is to determine the correlation weights between the query vector and the key vector, and then use these correlation weights to weight and fuse the value vectors to obtain the fused feature representation. Camera feature representations are feature dimensions related to camera motion. Inter-frame attention is performed only on camera feature representations, avoiding invalid inter-frame attention operations (inter-frame attention operations performed on other feature representations unrelated to camera motion), thus improving computational efficiency and avoiding waste of computational resources.

[0133] In some embodiments, the inter-frame attention layer of the camera parameter prediction model performs inter-frame attention operations on the camera feature representations corresponding to at least two video frames respectively, to obtain the camera parameters corresponding to at least two video frames respectively.

[0134] In some embodiments, step 322 above further includes the following steps: Step 3221: Perform inter-frame attention operation on at least two camera feature representations to obtain the first feature representations corresponding to at least two video frames respectively.

[0135] In some embodiments, the first feature representation is the final feature representation used for camera parameter prediction. For example, camera parameter prediction is performed on the first feature representation of the i-th video frame to obtain the camera parameters of the i-th video frame.

[0136] For example, the inter-frame attention operation is used to pay attention to the information of other video frames associated with the i-th video frame when processing the i-th video frame, and to extract useful information from the other video frames to enhance the first feature representation of the i-th video frame based on their correlation and temporal relationship with the i-th video frame, so that the first feature representation of the i-th video frame can contain rich contextual information.

[0137] In some embodiments, the inter-frame attention layer of the camera parameter prediction model performs inter-frame attention operations on the camera feature representations corresponding to at least two video frames respectively, and outputs the first feature representations corresponding to at least two video frames respectively.

[0138] The following explanation uses the i-th video frame out of at least two video frames as an example to illustrate the scheme for obtaining the first feature representation.

[0139] Option 1 Perform inter-frame attention operation on the i-th video frame and all video frames in the first video.

[0140] Step 1. Based on the camera feature representation of the i-th video frame and the camera feature representation of the j-th video frame, obtain the inter-frame fusion feature representation between the j-th video frame and the i-th video frame.

[0141] Here, the j-th video frame is any video frame in the first video, that is, the value of j ranges from 1 to T. When processing the i-th video frame, all video frames in the first video are traversed to obtain the inter-frame fusion feature representation between each video frame and the i-th video frame, which is denoted as N inter-frame fusion feature representations.

[0142] In some embodiments, the specific process for determining the inter-frame fusion feature representation is as follows: (1) Determine the query vector corresponding to the i-th video frame based on the camera feature representation corresponding to the i-th video frame.

[0143] In some embodiments, the camera feature representation corresponding to the i-th video frame is used as the query vector corresponding to the i-th video frame. Alternatively, a first linear transformation is performed on the camera feature representation corresponding to the i-th video frame to obtain the query vector corresponding to the i-th video frame.

[0144] (2) Determine the key vector and value vector corresponding to the j-th video frame based on the camera feature representation corresponding to the j-th video frame.

[0145] In some embodiments, the camera feature representation corresponding to the j-th video frame is used as the key vector corresponding to the j-th video frame, and the camera feature representation corresponding to the j-th video frame is used as the value vector corresponding to the j-th video frame. Alternatively, a second linear transformation is performed on the camera feature representation corresponding to the j-th video frame to obtain the key vector corresponding to the j-th video frame; a third linear transformation is performed on the camera feature representation corresponding to the j-th video frame to obtain the value vector corresponding to the j-th video frame.

[0146] (3) Based on the attention mechanism, the query vector corresponding to the i-th video frame, the key vector corresponding to the j-th video frame, and the value vector corresponding to the j-th video frame are fused to obtain the inter-frame fusion feature representation between the j-th video frame and the i-th video frame.

[0147] In some embodiments, an attention weight matrix is ​​determined based on the query vector corresponding to the i-th video frame and the key vector corresponding to the j-th video frame; a weighted summation process is performed on the value vector corresponding to the j-th video frame according to the attention weight matrix to obtain the inter-frame fusion feature representation between the j-th video frame and the i-th video frame.

[0148] By employing a "query-key-value" attention mechanism, the i-th video frame is used as the query, and the other video frames are used as the key and value. The camera pose information from the other video frames is fused into the camera feature representation of the i-th video frame, enriching the camera pose information in the camera feature representation of the i-th video frame, thereby improving the accuracy of camera parameter prediction.

[0149] Step 2. If there are two or more video frames in the first video, fuse the inter-frame fusion feature representations between each video frame in the first video and the i-th video frame to obtain the first feature representation corresponding to the i-th video frame.

[0150] In some embodiments, the method for fusing to obtain the first feature representation corresponding to the i-th video frame includes, but is not limited to, the following methods: Method 1: Concatenate N inter-frame fusion feature representations to obtain the first feature representation corresponding to the i-th video frame.

[0151] For example, all the obtained inter-frame fusion feature representations are concatenated along the feature dimension to form a feature vector with a higher dimension. For instance, if each inter-frame fusion feature representation has a dimension d and there are N inter-frame fusion feature representations, then the dimension of the first feature representation after concatenation is T×d.

[0152] Method 2: Based on the fusion weight, perform weighted fusion processing on the inter-frame fusion feature representation between each video frame in the first video and the i-th video frame to obtain the first feature representation corresponding to the i-th video frame.

[0153] For example, a fusion weight is assigned to each video frame and then to the i-th video frame. The fusion feature representations of the N frames are then weighted and summed according to the fusion weights to obtain the first feature representation corresponding to the i-th video frame. Let w be the fusion weight between the i-th and j-th video frames. j Then w j The inter-frame fusion feature representation O used to weight the j-th video frame and the i-th video frame is, for example, the weighted feature is w. j ×O.

[0154] In some embodiments, a time decay coefficient is determined based on the time interval between the i-th video frame and the j-th video frame; and a fusion weight between the i-th video frame and the j-th video frame is determined based on the time decay coefficient.

[0155] The time decay coefficient is positively correlated with the time interval and negatively correlated with the fusion weight. Introducing the time decay coefficient results in lower fusion weights for frames with longer time intervals, effectively suppressing noise and preventing interference from distant, irrelevant frames, thereby improving the accuracy of predicted camera parameters. For example, in autonomous driving, the current pose of the vehicle (camera) is mainly affected by the motion state within the last few seconds (such as the speed and steering angle of the last 10 frames), while the influence of older frames (such as those from 10 seconds ago) can be ignored.

[0156] In the above scheme, in the camera feature representation of the i-th video frame, the camera pose information in all video frames of the first video is fused to obtain the first feature representation of the i-th video frame, which improves the globality of the camera pose information described by the first feature representation of the i-th video frame, thereby improving the prediction accuracy of camera parameters.

[0157] Option 2 Perform inter-frame attention operation on the i-th video frame and K video frames in the first video.

[0158] At least two video frames comprise K video frames. The K video frames are a sequence of video frames extracted from at least two video frames, with the i-th video frame as the center. K is an integer greater than 1; in some embodiments, K is a pre-defined fixed number of frames.

[0159] For example, when K is odd, if i < a (a is...) If a ≤ i ≤ Ta, then the K video frames are the sequence of video frames extracted from at least two video frames with the i-th video frame as the center, and the i-th video frame is the center of the video frame sequence; if Ta < i < T (T is the total number of at least two video frames), then the K video frames are the sequence of video frames from the T-K+1-th video frame to the T-th video frame.

[0160] For example, when K is odd, if i < b (b is...) If b ≤ i ≤ Tb, then the K video frames are the 1st to the Kth video frames; if b ≤ i ≤ Tb, then the K video frames are a sequence of video frames extracted from at least two video frames with the i-th video frame as the center, and one of the two center video frames in the video frame sequence is the i-th video frame; if Tb < i < T, then the K video frames are the T-K+1th to the T-th video frames.

[0161] Step 1. Based on the camera feature representation of the i-th video frame and the camera feature representation of the p-th video frame, obtain the inter-frame fusion feature representation between the p-th video frame and the i-th video frame.

[0162] P is a positive integer, p≤K. Here, the p-th video frame is any one of the K video frames, that is, the value of p ranges from 1 to K. When processing the i-th video frame, the K video frames are traversed to obtain the inter-frame fusion feature representation between each video frame and the i-th video frame, which is denoted as the K inter-frame fusion feature representations.

[0163] It should be noted that the method for obtaining the inter-frame fusion feature representation between the p-th video frame and the i-th video frame can refer to the above description of obtaining the inter-frame fusion feature representation between the j-th video frame and the i-th video frame, and will not be repeated here.

[0164] Step 2. Fuse the inter-frame fusion feature representations of each of the K video frames with the i-th video frame to obtain the first feature representation corresponding to the i-th video frame.

[0165] That is, the first feature representation corresponding to the i-th video frame is obtained by fusing K inter-frame fusion feature representations.

[0166] In some embodiments, the fusion method includes, but is not limited to: (1) concatenating K inter-frame fusion feature representations to obtain a first feature representation corresponding to the i-th video frame; (2) determining the fusion weights between the K video frames and the i-th video frame respectively, and performing weighted fusion processing on the K inter-frame fusion feature representations according to the fusion weights to obtain a first feature representation corresponding to the i-th video frame.

[0167] In some embodiments, a time decay coefficient is determined based on the time interval between the i-th video frame and the p-th video frame; a fusion weight is then determined based on the time decay coefficient. The time decay coefficient is positively correlated with the time interval and negatively correlated with the fusion weight.

[0168] The above scheme does not use all video frames, but instead uses a fixed-size K-frame sliding window to perform inter-frame attention operations with the current frame. This avoids the high time consumption of calculating all video frames and improves the efficiency of inter-frame attention operations. Especially for long videos, where computational complexity increases quadratically with the number of frames, limiting the scope of inter-frame attention operations to the K frames near the current frame can reduce computational complexity and significantly improve computational efficiency.

[0169] Step 3222: Determine the camera parameters corresponding to at least two video frames based on the first feature representations corresponding to at least two video frames respectively.

[0170] In some embodiments, camera parameter prediction is performed on the first feature representations corresponding to at least two video frames to determine the camera parameters corresponding to at least two video frames.

[0171] In some embodiments, the camera parameter prediction model further includes a parameter prediction layer. The parameter prediction layer is a network layer used to predict camera parameters. The parameter prediction layer performs camera parameter prediction on the first feature representations corresponding to at least two video frames, respectively, to obtain the camera parameters corresponding to the at least two video frames. For example, the parameter prediction layer performs a linear transformation on the first feature representation to obtain the camera parameters corresponding to the video frames. For example, the parameter prediction layer may consist of 4 self-attention layers + 1 linear layer, with the first feature representation input to the parameter prediction layer, and the parameter prediction layer outputting the camera parameters.

[0172] In some embodiments, the scheme for obtaining the representation of the first feature includes, but is not limited to, at least one of the following: (1) Obtain the first feature representation based on inter-frame attention operation.

[0173] For example, please refer to Figure 4 In the "(1-1) inter-frame" section, embedding operations are performed on the T video frames in the first video 401 to obtain T first image feature representations. These T first image feature representations are then concatenated with a first camera feature representation to obtain T frame feature representations. Inter-frame attention operations are then performed on the first camera feature representations in the T frame feature representations to obtain T first feature representations 402.

[0174] (2) Obtain the first feature representation based on intra-frame attention operation + inter-frame attention operation.

[0175] For example, please refer to Figure 4 After obtaining T frame feature representations by “(1-2) intra-frame + inter-frame”, intra-frame attention operations are performed on these T frame feature representations to obtain T second feature representations. Inter-frame attention operations are performed on the second camera feature representations in the T second feature representations to obtain T first feature representations 402.

[0176] In the above scheme, intra-frame attention is first performed on each frame feature individually (optimizing the local features of a single frame), and then inter-frame attention is performed on the optimized second feature (associating temporal relationships). Inter-frame association is then performed based on the optimized single-frame feature (second feature), which improves the spatial accuracy of camera parameters (such as the rotation matrix providing a more accurate description of the camera pose).

[0177] (3) Obtain the first feature representation based on global attention operation + inter-frame attention operation.

[0178] For example, please refer to Figure 4 In the "(1-3) global + inter-frame" approach, after obtaining T frame feature representations, a global attention operation is performed on these T frame feature representations to obtain T third feature representations. An inter-frame attention operation is then performed on the third camera feature representations among the T third feature representations to obtain T first feature representations 402.

[0179] In the above scheme, global attention is first performed on all frame features (establishing global sequence association), and then inter-frame attention is performed on the globally optimized third feature (enhancing temporal sequence). Performing inter-frame attention based on the globally optimized third feature can ensure that the temporal sequence changes between frames conform to the scene logic of the entire sequence (such as in a highway scene, the translation vector changes of camera parameters will follow the straightness of the road and there will be no unreasonable directional changes), thus improving the global consistency of camera parameters in the entire sequence.

[0180] (4) Obtain the first feature representation based on global attention operation + intra-frame attention operation + inter-frame attention operation.

[0181] For example, please refer to Figure 4 After obtaining T frame feature representations through "(1-4) global + intra-frame + inter-frame", a global attention operation is performed on these T frame feature representations to obtain T third feature representations. Intra-frame attention operations are then performed on each of the T third feature representations to obtain T second feature representations. Inter-frame attention operations are then performed on the second camera feature representations among the T second feature representations to obtain T first feature representations 402.

[0182] In the above scheme, global attention is first used to inject full-sequence context into each frame, then intra-frame attention is used to optimize local features of a single frame, and finally inter-frame attention is used to correlate temporal sequences. This embodiment first clarifies the role of each frame in the full sequence, and then optimizes key features of a single frame based on the global context, resulting in more semantically targeted features for a single frame.

[0183] (5) Obtain the first feature representation based on intra-frame attention operation + global attention operation + inter-frame attention operation.

[0184] For example, please refer to Figure 4After obtaining T frame feature representations from “(1-5) Intra-frame + Global + Inter-frame”, intra-frame attention operations are performed on each of these T frame feature representations to obtain T second feature representations. Global attention operations are performed on the T second feature representations to obtain T third feature representations. Inter-frame attention operations are performed on the third camera feature representations among the T third feature representations to obtain T first feature representations 402.

[0185] In the above scheme, local features of a single frame are first optimized through intra-frame attention, then global correlations of high-quality single frames are fused through global attention, and finally temporal correlations are enhanced through inter-frame attention. This embodiment first optimizes local features of a single frame through intra-frame attention to filter out single-frame noise, and then performs global fusion to avoid the global diffusion of noise features. The accuracy of global correlation is higher, and the temporal correlation of subsequent inter-frame attention modeling is also reliable. It is suitable for scenarios with low video quality (such as low-resolution video and night video).

[0186] In some embodiments, the first feature representation corresponding to at least two video frames is a feature representation obtained after performing L inter-frame attention operations. L is an integer greater than 1.

[0187] For example, please refer to Figure 4 In “(1-1) Inter-frame”, L inter-frame attention operations are performed on T frame feature representations to obtain T first feature representations 402. In “(1-2) Intra-frame + Inter-frame”, L “intra-frame attention operations and inter-frame attention operations” are performed alternately on T frame feature representations to obtain T first feature representations 402. In “(1-3) Global + Inter-frame”, L “global attention operations and inter-frame attention operations” are performed alternately on T frame feature representations to obtain T first feature representations 402. In “(1-4) Global + Intra-frame + Inter-frame”, L “global attention operations, intra-frame attention operations, and inter-frame attention operations” are performed alternately on T frame feature representations to obtain T first feature representations 402. In “(1-5) Intra-frame + Global + Inter-frame”, L “intra-frame attention operations, global attention operations, and inter-frame attention operations” are performed alternately on T frame feature representations to obtain T first feature representations 402.

[0188] A single attention operation may have local biases (such as missing a key region during a single intra-frame attention operation, or mistakenly associating an irrelevant frame during a single global attention operation); while L alternations can reduce biases through iterative correction and improve the accuracy of camera parameter prediction.

[0189] In some embodiments, the method for obtaining a first feature representation and predicting camera parameters includes, but is not limited to, at least one of the following: Method 1: After obtaining the first feature representation through any of the above schemes (1)-(5), perform camera parameter prediction on the first feature representations corresponding to at least two video frames respectively to obtain the camera parameters corresponding to at least two video frames respectively.

[0190] Method 2: After obtaining the first feature representation through any of the above schemes (1)-(5), perform intra-frame attention operation on the first feature representation corresponding to at least two video frames respectively to obtain the spatiotemporal feature representation corresponding to at least two video frames respectively; perform camera parameter prediction on the spatiotemporal feature representation corresponding to at least two video frames respectively to obtain the camera parameters corresponding to at least two video frames respectively. In the above schemes, the temporally associated first feature representation is obtained, and then intra-frame attention is performed on the first feature representation separately to finally obtain the spatiotemporally fused feature. The temporal feature (first feature representation) is optimized twice intra-frame to further highlight the local features directly related to the extrinsic parameters in a single frame (such as the correspondence between the 3D coordinates of a point in the frame and the camera translation vector), providing accurate spatiotemporal feature input for the subsequent parameter prediction layer and reducing the extrinsic parameter prediction error.

[0191] In some embodiments, intra-frame attention operations are performed on at least two first feature representations by the intra-frame attention layer of the camera parameter prediction model to obtain spatiotemporal feature representations corresponding to at least two video frames. Camera parameter prediction is then performed on the spatiotemporal feature representations corresponding to at least two video frames by the parameter prediction layer of the camera parameter prediction model to obtain camera parameters corresponding to at least two video frames.

[0192] For example, please refer to Figure 4 In the process of “(2) inter-frame + intra-frame”, T frame feature representations are obtained. Inter-frame attention is then performed on these T frame feature representations to obtain T first feature representations 402. Intra-frame attention is then performed on each of these T second feature representations 402 to obtain T spatiotemporal feature representations 403 corresponding to the first video 401. The process of “performing inter-frame attention on the T frame feature representations to obtain T first feature representations 402” can be implemented as follows: Figure 4 At least one of (1-1), (1-2), (1-3), (1-4), and (1-5) in the above.

[0193] In some embodiments, the spatiotemporal feature representations corresponding to at least two video frames are feature representations obtained after performing L inter-frame attention operations. L is an integer greater than 1. For example, please refer to... Figure 4 In “(2) Inter-frame + Intra-frame”, “Inter-frame attention operation and intra-frame attention operation” are performed alternately L times on T frame feature representations to obtain T spatiotemporal feature representations 403.

[0194] In some embodiments, the method for obtaining frame feature representation includes, but is not limited to, at least one of the following: (1) Obtain frame feature representation directly through the first video.

[0195] In some embodiments, an embedding operation is performed on at least two video frames in the first video to obtain first image feature representations corresponding to at least two video frames respectively; the first image feature representations corresponding to at least two video frames are concatenated with the first camera feature representation to obtain frame feature representations corresponding to at least two video frames respectively.

[0196] The first image feature representation includes at least one embedded feature representation. For example, the embedding operation converts the raw video frame data into a feature vector (token) containing semantic information of the video frame, i.e., an image token (embedded feature representation). For each video frame, after obtaining at least one image token, a camera token (first camera feature representation) is concatenated to obtain the frame feature representation of that video frame. The camera token is a dedicated token supplementing the image token of the video frame, used to carry feature information related to camera parameters. In the frame feature representation, the image token and the camera token have the same feature dimension, and the value of the camera token can be a randomly initialized value.

[0197] (2) Fuse the first video and the target view video to obtain frame feature representation.

[0198] The target perspective video is a video acquired from the target perspective of a moving object. In some embodiments, the target perspective includes, but is not limited to, at least one of a first-person perspective and a third-person perspective. For example, when the target perspective is implemented as a first-person perspective, the first-person perspective corresponding to the target video and the first video are different. For instance, the target video can be regarded as a video of the front of the vehicle taken by a camera mounted in the driver's cab, and the first video can be regarded as a video of the front of the vehicle taken by a camera mounted in the passenger's cab.

[0199] The following explanation uses the implementation of the target perspective as a third-person perspective as an example. In some embodiments, the third-person perspective includes, but is not limited to, at least one of the following: top-down view, front view, rear view, left view, right view, etc.

[0200] In some embodiments, a second video is acquired; the second video is a video acquired from the third-person perspective of a moving object. For example, the first video is an autonomous driving video acquired from the first-person perspective of vehicle A; the second video is an autonomous driving video acquired from the third-person perspective of vehicle A. The second video can be considered as an autonomous driving video acquired from the first-person perspective of vehicle B, where vehicle A can be implemented as the vehicle preceding vehicle B.

[0201] In this video, at least two video frames in the first video correspond one-to-one with at least two video frames in the second video. For example, the at least two video frames corresponding to the first video include the i-th video frame; the at least two video frames corresponding to the second video include the i'-th video frame, and the timestamp corresponding to the i-th video frame is the same as the timestamp corresponding to the i'-th video frame.

[0202] In some embodiments, the step of obtaining the frame feature representation further includes: Step 1 : Perform an embedding operation on at least two video frames in the second video to obtain the second frame feature representations corresponding to the at least two video frames in the second video; perform an embedding operation on at least two video frames in the first video to obtain the first frame feature representations corresponding to the at least two video frames in the first video.

[0203] The first frame feature representation includes a first target image feature representation and a first target camera feature representation. In some embodiments, an embedding operation is performed on the i-th video frame through an embedding layer to obtain the first target image feature representation corresponding to the i-th video frame; a feature concatenation operation is performed on the first target image feature representation corresponding to the i-th video frame and the first target camera feature representation corresponding to the i-th video frame to obtain the first frame feature representation corresponding to the i-th video frame.

[0204] In some embodiments, the second frame feature representation includes only the second target image feature representation. An embedding operation is performed on the i'th video frame through an embedding layer to obtain the second target image feature representation corresponding to the i'th video frame, which serves as the second frame feature representation. Alternatively, the second frame feature representation includes both the second target image feature representation and the second target camera feature representation. In some embodiments, an embedding operation is performed on the i'th video frame through an embedding layer to obtain the second target image feature representation corresponding to the i'th video frame; a feature concatenation operation is then performed on the second target image feature representation and the second target camera feature representation corresponding to the i'th video frame to obtain the second frame feature representation corresponding to the i'th video frame.

[0205] Step 2 : Fuse at least two first feature representations and at least two second feature representations to obtain frame feature representations corresponding to at least two video frames in the first video.

[0206] In some embodiments, the first frame feature representation corresponding to the i-th video frame and the second frame feature representation corresponding to the i'-th video frame are fused to obtain the frame feature representation corresponding to the i-th video frame. Following the processing method for the i-th video frame, after traversing at least two video frames, frame feature representations corresponding to at least two video frames are obtained respectively.

[0207] In some embodiments, the first frame feature representation corresponding to the i-th video frame and the second frame feature representation corresponding to the i'-th video frame are concatenated to obtain the frame feature representation corresponding to the i-th video frame. Alternatively, the first frame feature representation corresponding to the i'-th video frame and the second frame feature representation corresponding to the i'-th video frame are fused based on an attention mechanism to obtain the frame feature representation corresponding to the i-th video frame.

[0208] For example, if the second frame feature representation only includes the second target image feature representation, then the method for fusing to obtain the frame feature representation includes, but is not limited to, at least one of the following: a. The second target image feature representation is concatenated into each feature representation in the first frame feature representation (including the first target image feature representation and the first target camera feature representation) to obtain the frame feature representation.

[0209] b. Perform a global attention operation on the first frame feature representation and the second target image feature representation to obtain the frame feature representation. For example, in the global attention operation, the first frame feature representation and the second target image feature representation are treated as a whole, denoted as R feature representations, where R is an integer greater than 1; each feature representation in the first frame feature representation is fused with the R feature representations based on an attention mechanism to obtain the frame feature representation.

[0210] For example, if the second frame feature representation includes the second target image feature representation and the second target camera feature representation: Figure 5 A schematic diagram is shown for fusing third-person perspective video to obtain frame feature representations, such as... Figure 5 As shown, the first video 510 is an autonomous driving video captured from the first-person perspective of vehicle A (the moving object); the second video 520 is an autonomous driving video captured from the third-person perspective of vehicle A, and the second video 520 can be considered as an autonomous driving video captured from the first-person perspective of vehicle B. Vehicle A and the vehicle ahead of vehicle B are also shown.

[0211] For the T video frames 511 in the first video 510, a feature extraction operation is performed on each video frame to obtain T first frame feature representations 512. The first frame feature representations include the first target image feature representation ( Figure 5 (blank squares in the image) and the first target camera feature representation ( Figure 5 (A square filled with diagonal lines). For the T video frames 521 in the second video 520, a feature extraction operation is performed on each video frame to obtain T second frame feature representations 522. The second frame feature representations include the second target image feature representation ( Figure 5 (blank squares in the image) and second target camera feature representation ( Figure 5 (A square filled with diagonal lines).

[0212] The T first-frame feature representations 512 and the T second-frame feature representations 522 are fused together, wherein the first-frame feature representation corresponding to the i-th video frame and the second-frame feature representation corresponding to the i'-th video frame are fused together to obtain the frame feature representation corresponding to the i-th video frame. The method for fusing to obtain the frame feature representation includes, but is not limited to, at least one of the following: a. The second frame feature representation is concatenated into each feature representation in the first frame feature representation (including the first target image feature representation and the first target camera feature representation) to obtain the frame feature representation.

[0213] b. Perform a global attention operation on the first frame feature representation and the second frame feature representation to obtain the frame feature representation. For example, in the global attention operation, the first frame feature representation and the second frame feature representation are treated as a whole, denoted as S feature representations, where S is an integer greater than 1; each feature representation in the first frame feature representation is fused with the S feature representations based on an attention mechanism to obtain the frame feature representation.

[0214] c. Perform intra-frame attention operations on the first frame feature representation and the second frame feature representation respectively. Perform global attention operations on the first frame feature representation after the intra-frame attention operation and the second frame feature representation after the intra-frame attention operation to obtain the frame feature representation.

[0215] d. Perform global attention operations on the feature representations of the first frame and the feature representations of the second frame to obtain global frame feature representations. Perform intra-frame attention operations on the global frame feature representations to obtain frame feature representations.

[0216] It should be noted that the above example of fusing third-person perspective videos to obtain frame feature representations is only an illustrative example. Other target perspective videos corresponding to the first video can also be fused to obtain frame feature representations; or, at least two target perspective videos can be fused to obtain frame feature representations. For the specific fusion process, please refer to the above description of fusing third-person perspective videos to obtain frame feature representations, which will not be repeated here.

[0217] In the above scheme, the feature representations of first-person view video frames and corresponding timestamped third-person view video frames are integrated to obtain motion information of moving objects from multiple perspectives, enriching the feature dimensions, making camera parameter prediction more comprehensive and accurate, and improving prediction reliability. For example, a first-person view video (inside the car) may not show the complete movement of the car body, but a third-person view video can clearly show the overall displacement and attitude of the vehicle, making up for the blind spots of the first-person view.

[0218] In some embodiments, a first feature representation is obtained by performing global attention operations and intra-frame attention operations on video frames. For example, please refer to... Figure 6The diagram illustrates a flowchart of the camera parameter prediction method provided in this application. Figure 6 As shown, Figure 3 The embodiment can also be implemented as follows: steps 601 to 606.

[0219] Step 601: Obtain the first video.

[0220] Step 602: Perform an embedding operation on at least two video frames in the first video to obtain the frame feature representations corresponding to the at least two video frames respectively.

[0221] Step 603: Perform a global attention operation on the feature representations of at least two frames to obtain the third feature representations corresponding to at least two video frames respectively.

[0222] The global attention operation is used to perform global fusion processing on feature representations of at least two frames. The at least two frame feature representations include the i-th frame feature representation. The global attention operation is used to fuse the feature representations in the i-th frame feature representation, and to fuse the i-th frame feature representation with other frame feature representations.

[0223] Figure 7 A schematic diagram of obtaining a first feature representation is shown, such as... Figure 7 As shown, global attention operation, intra-frame attention operation and inter-frame attention operation are performed sequentially on the T frame feature representations 701 to obtain T first feature representations 704.

[0224] (a) Global attention operation like Figure 7 As shown, the camera feature representation 7011 (i.e., the first camera feature representation) in the i-th frame feature representation is used as an example for explanation. The global attention operation on the camera feature representation 7011 includes the following steps: Based on the attention mechanism, the T×4 feature representations in the T frame feature representations 701 are sequentially fused with the camera feature representation 7011 to obtain the T×4 globally fused feature representations corresponding to the camera feature representation 7011. The T×4 globally fused feature representations are then fused to obtain the camera feature representation 7021 (i.e., the third camera feature representation) in the i-th third feature representation. Here, the camera feature representation 7011 is used to determine the query vector in the attention mechanism, and the first target feature representation is used to determine the value vector and key vector in the attention mechanism. The first target feature representation sequentially takes the T×4 feature representations in the T frame feature representations 701.

[0225] Following the global attention operation method described above for the camera feature representation 7011, by traversing the T×4 feature representations in the T frame feature representations 701, we can obtain T third feature representations 702.

[0226] It should be noted that the "4" in "T×4" above refers to the number of feature representations corresponding to each video frame. Figure 7 Taking the example of four feature representations per video frame, the number of feature representations per video frame can be more or less depending on the requirements, such as... Figure 7 Each frame feature representation (or third feature representation, or second feature representation) includes three feature representations: one camera feature representation and two embedding feature representations; or, Figure 7 Each frame feature representation (or third feature representation, or second feature representation) includes 5 feature representations, which include 1 camera feature representation and 4 embedding feature representations.

[0227] Step 604: Perform intra-frame attention operations on at least two third feature representations respectively to obtain second feature representations corresponding to at least two video frames respectively.

[0228] Intra-frame attention operations are used to fuse feature representations from the third feature representation corresponding to a single video frame.

[0229] (ii) Intra-frame attention operation like Figure 7 As shown, the camera feature representation 7021 in the i-th third feature representation is used as an example for explanation. Based on the attention mechanism, the four feature representations in the i-th third feature representation are sequentially fused with the camera feature representation 7021 to obtain four intra-frame fused feature representations corresponding to the camera feature representation 7021. These four intra-frame fused feature representations are then fused to obtain the camera feature representation 7031 in the i-th second feature representation (i.e., the second camera feature representation). Here, the camera feature representation 7021 is used to determine the query vector in the attention mechanism, and the second target feature representation is used to determine the value vector and key vector in the attention mechanism. The second target feature representation sequentially takes values ​​from the four feature representations in the i-th third feature representation.

[0230] Following the intra-frame attention operation of the camera feature representation 7021 described above, by traversing the T×4 feature representations in the T third feature representations 702, we can obtain the T second feature representations 703.

[0231] Step 605: Perform inter-frame attention operation on at least two second feature representations to obtain first feature representations corresponding to at least two video frames respectively.

[0232] Inter-frame attention operations are used to fuse at least two second feature representations based on the temporal relationship between at least two video frames.

[0233] (III) Inter-frame attention operations like Figure 7As shown, the camera feature representation 7031 in the feature representation of the i-th frame is used as an example for explanation. Based on the attention mechanism, the T camera feature representations in the T second feature representations 703 are sequentially fused with the camera feature representation 7031 to obtain the T inter-frame fused feature representations corresponding to the camera feature representation 7031. These T inter-frame fused feature representations are then fused to obtain the i-th first feature representation, which is the camera feature representation 7041. Here, the camera feature representation 7031 is used to determine the query vector in the attention mechanism, and the third target feature representation is used to determine the value vector and key vector in the attention mechanism. The third target feature representation sequentially takes values ​​from the T camera feature representations in the T second feature representations 703.

[0234] Following the above-described inter-frame attention operation on camera feature representation 7031, by traversing the T camera feature representations in the T second feature representations 703, we can obtain T first feature representations 704.

[0235] Step 606: Determine the camera parameters corresponding to at least two video frames based on the first feature representations corresponding to at least two video frames respectively.

[0236] In summary, the camera parameter prediction method provided in this application obtains a first feature representation through a combination of global attention operation, intra-frame attention operation, and inter-frame attention operation. This first feature representation is used to perform subsequent camera parameter prediction. Specifically, the global attention operation captures the long-range dependencies of the video sequence, the intra-frame attention operation optimizes single-frame features, and the inter-frame attention operation captures the temporal correlations of the video sequence. The combination of these three operations makes the first feature representation more comprehensive and robust, thereby improving its quality and thus increasing the accuracy of camera parameter prediction.

[0237] In some embodiments, in abrupt motion (sudden change in motion) scenarios, first-person perspective videos may exhibit "jump regions" (such as target image regions) in certain video frames due to "large displacement / large change in perspective / sudden disappearance of the target" between consecutive frames. For example, if a vehicle (moving object) experiences a sharp pitch after passing a speed bump, the first video will show an overall "frame jump" displacement. If, during inter-frame attention operations, all image regions in all video frames of the first video are treated equally, the "jump regions" will be treated as normal motion regions for alignment, leading to significant camera parameter prediction errors. Therefore, this application uses a "jump weight matrix" to reduce the weight of jump regions to suppress the noise impact caused by jumps. For example, please refer to... Figure 8 The diagram illustrates a flowchart of the camera parameter prediction method provided in this application. Figure 8 As shown, Figure 3 The embodiment can also be implemented as follows: steps 801 to 806.

[0238] Step 801: Obtain the first video.

[0239] Step 802: Determine the target image region of the target video frame in at least two video frames of the first video.

[0240] The target image region meets preset transition requirements. For example, the preset transition requirements include, but are not limited to, at least one of the following: (1) The pixel value difference between the target image region of the target video frame and the corresponding region of the previous video frame is greater than the preset difference.

[0241] For example, adjacent video frames are video frame a and video frame b. The difference in pixel values ​​between video frame a and video frame b is calculated. If the difference in target pixel values ​​between region 1 in video frame a and the corresponding region 2 in video frame b is greater than a preset difference, video frame b is determined to be the target video frame, and region 2 in video frame b is the target image region (such as the drastic change in road surface pixels caused by a speed bump).

[0242] (2) The motion vector of the target image region in the target video frame meets the requirements of abnormal motion.

[0243] Among them, motion vector refers to the motion vector of a pixel. For example, the motion vector of pixel a is a vector that describes how pixel a moves between two consecutive frames (the current frame and the previous frame).

[0244] For example, motion vectors of pixels in each region of a video frame are calculated using a motion estimation model (such as a model based on optical flow algorithms). If the average value of the motion vectors of region 3 in video frame c is greater than the average value of the overall motion vectors of the target video frame, then video frame c is determined to be the target video frame, and region 3 in video frame c is the target image region. And / or, if the directional difference between the motion vector of region 4 in video frame d and the motion vectors of the pixels surrounding region 4 is greater than a preset directional difference, then video frame d is determined to be the target video frame, and region 4 in video frame d is the target image region.

[0245] (3) The target object in the target image region of the target video frame meets the object jump requirement.

[0246] The semantic segmentation model identifies target objects (such as vehicles and pedestrians) in video frames. If a target object appears in video frame e and disappears in video frame f, and video frames e and f are adjacent video frames, then video frame e is determined to be the target video frame, and the display area of ​​the moving object in video frame e is the target image area.

[0247] Step 803: Determine the transition weight matrix corresponding to the target video frame based on the target image region.

[0248] In the transition weight matrix, the weight of the target image region is lower than the weight of other image regions in the target video frame. Other image regions refer to all image regions in the target video frame other than the target image region.

[0249] For example, the jump weight matrix is ​​a matrix that matches the spatial dimension of the video frame (e.g., if the frame resolution is 384×216, then the jump weight matrix size is 384×216), and each element represents the weight value at the corresponding position. In some embodiments, the weight value of the target image region (jump region) is set to a lower value (e.g., 0.3) to weaken its feature influence; the weight values ​​of other image regions (non-jump regions) are set to higher values ​​(e.g., 0.8-1.0) to retain their normal influence.

[0250] Step 804: Based on the jump weight matrix corresponding to the target video frame, perform weighted processing on the frame feature representation corresponding to the target video frame to obtain the weighted frame feature representation.

[0251] In some embodiments, the transition weight matrix is ​​multiplied element-wise with the frame feature representation corresponding to the target video frame to obtain the weighted frame feature representation.

[0252] For example, the frame feature representation corresponding to the target video frame is multiplied by the transition weight matrix. In the frame feature representation, each feature representation corresponding to the transition region is multiplied by a low weight, and each feature representation corresponding to the non-transition region is multiplied by a high weight, thus obtaining a weighted frame feature representation. In the weighted frame feature representation, the feature values ​​of the transition regions become smaller, while the feature values ​​of the non-transition regions remain unchanged or increase.

[0253] Step 805: Perform inter-frame attention operation on the weighted frame feature representation and other frame feature representations in at least two frame feature representations to obtain the first feature representations corresponding to at least two video frames respectively.

[0254] Among them, other frame feature representations refer to frame feature representations other than the weighted frame feature representation among at least two frame feature representations.

[0255] It should be noted that the explanation of "performing inter-frame attention operation on the weighted frame feature representation and other frame feature representations among at least two frame feature representations to obtain the first feature representations corresponding to at least two video frames respectively" can be found in the specific explanation of "obtaining the first feature representations corresponding to at least two video frames respectively" in the above embodiments, and will not be repeated here.

[0256] For example, a global attention operation is performed on the weighted frame feature representation and other frame feature representations among at least two frame feature representations to obtain third feature representations corresponding to at least two video frames respectively. Intra-frame attention operations are then performed on each of the at least two third feature representations to obtain second feature representations corresponding to at least two video frames respectively. Inter-frame attention operations are then performed on the second camera feature representations corresponding to each of the at least two second feature representations to obtain first feature representations corresponding to at least two video frames respectively.

[0257] Step 806: Determine the camera parameters corresponding to at least two video frames based on the first feature representations corresponding to at least two video frames respectively.

[0258] In the above scheme, before performing inter-frame attention operations, abrupt regions in the video frames (such as suddenly appearing pedestrians or obstacles) are identified, and their weights are reduced before performing the inter-frame attention operations. By reducing the weight of abrupt regions, the prediction of camera parameters prioritizes continuous and stable features, thereby improving prediction accuracy. For example, in autonomous driving videos, abrupt changes may originate from extreme lighting changes (such as exiting a tunnel) or severe occlusion by other objects. Identifying and suppressing the influence of abrupt regions can prevent erroneous visual information from contaminating the camera parameter prediction results.

[0259] In some embodiments, the camera parameters are obtained by analyzing the first video using a camera parameter prediction model. For example, please refer to... Figure 9 The diagram illustrates a flowchart of the camera parameter prediction method provided in this application. Figure 9 As shown above, Figure 3 The embodiment can also be implemented as follows: steps 910 to 940. It should be noted that... Figure 9 Implementation examples can be carried out in Figure 3 Prior to the implementation example, it can also be achieved in Figure 3 Following the examples, it is also possible to... Figure 3 Each of the embodiments is implemented independently, and this application does not limit them.

[0260] Step 910: Obtain the first video.

[0261] The first video is the video to be analyzed by the camera parameter prediction model.

[0262] The camera parameter prediction model is a pre-trained machine learning model; the camera parameter prediction model includes an embedding layer, a global attention layer, an intra-frame attention layer, an inter-frame attention layer, and a parameter prediction layer.

[0263] Step 920: Input at least two video frames from the first video into the embedding layer to obtain the frame feature representations corresponding to the at least two video frames respectively.

[0264] The embedding layer is used to perform embedding operations on video frames.

[0265] In some embodiments, at least two video frames from a first video are input into an embedding layer, and a first image feature representation corresponding to each of the at least two video frames is output. A concatenation process is then performed on the first image feature representation and the first camera feature representation corresponding to each of the at least two video frames to obtain frame feature representations corresponding to each of the at least two video frames. Specifically, a first camera feature representation is concatenated with the first image feature representation corresponding to each video frame. The camera feature representation consists of randomly initialized parameters used to carry camera parameter prediction features.

[0266] In some embodiments, at least two video frames from the first video are input into the embedding layer, and the first image feature representations corresponding to the at least two video frames are output. The first image feature representations, first camera feature representations, and register feature representations corresponding to the at least two video frames are then concatenated to obtain frame feature representations corresponding to the at least two video frames. Specifically, the first image feature representation corresponding to each video frame is concatenated with the first camera feature representation and the register feature representation.

[0267] Here, register features are randomly initialized parameters used to carry world coordinate system prediction features. In some embodiments, the register feature representation spliced ​​from the first image feature representation corresponding to the first video frame is different from the register feature representation spliced ​​from the first image feature representation corresponding to other video frames. In the feature representation of the first frame, the camera parameter prediction model can identify it as the reference frame through its unique register feature representation, so that the world coordinate system of the predicted camera parameters is the camera coordinate system in the first video frame.

[0268] Figure 10 A schematic diagram of the structure of a camera parameter prediction model is shown. Figure 10 As shown, the first video includes T video frames (1000), and the first video is a video obtained from the first-person perspective of vehicle A.

[0269] T video frames (1000) are input into the embedding layer 1011 of the parameter prediction model 1010. The embedding layer 1011 performs an embedding operation on each video frame to obtain a first image feature representation for each video frame. A first camera feature representation (such as a camera token, i.e., a camera feature vector) is then concatenated with the first image feature representation of each video frame. Figure 10 The diagonally filled squares in the frame feature representation shown yield T frame feature representations (1021). Alternatively, for each video frame, the first image feature representation is concatenated with the first camera feature representation and the register feature representation (such as the register token, i.e., the register feature vector). Figure 10 (The white-filled squares in the frame feature representation shown) yield T frame feature representations 1021.

[0270] In some embodiments, the embedding layer includes, but is not limited to, the DINO (DIstillation with NO labels) model. Exemplarily, the specific process of the embedding operation includes: the embedding layer 1011 performs segmentation processing on the video frame to obtain at least two image blocks; in some embodiments, each image block has the same size; the embedding layer performs linear transformation processing on the image blocks to obtain the image block feature representations corresponding to the two image blocks as the first image feature representation (such as an image block token, i.e., an image block feature vector). Figure 10 (The white-filled squares in the frame feature representation shown) where the token is the smallest processing unit for subsequent attention operations.

[0271] Step 930: Input at least two frame feature representations into the inter-frame attention layer and output the first feature representations corresponding to at least two video frames respectively.

[0272] The inter-frame attention layer is used to perform inter-frame attention operations on feature representations of at least two frames.

[0273] In some embodiments, the process of outputting the first feature representations corresponding to at least two video frames further includes the following steps: Step 1 Input at least two frame feature representations into the global attention layer and output the third feature representations corresponding to at least two video frames respectively.

[0274] The global attention layer is used to perform global attention operations on feature representations of at least two frames.

[0275] like Figure 10 As shown, the global attention layer 1012 of the prediction model 1010 takes T frame feature representations 1021 as input parameters. The global attention layer 1012 performs a global attention operation on the T frame feature representations 1021 to obtain the third feature representation of each video frame, i.e., T third feature representations 1022. For example, the specific process of the global attention operation includes: concatenating all image block tokens and camera tokens in the T frame feature representations 1021 into a global sequence, denoted as X∈R. （T ×（K+1）×C） Where K is the number of image blocks per frame (or the number of image blocks + e, where e represents the number of register tokens, such as 4), and C is the feature dimension. The process of performing self-attention calculation on the global sequence X is as follows: (1) Perform a linear transformation on the global sequence X to obtain the query matrix, key matrix, and value matrix. The linear transformation can be expressed as: Q = X × W Q K = X × W KV=X×W V W Q W K and W V For three different weight matrices, W Q W K and W V These are the model parameters determined during the training phase of the parameter prediction model 1010. The Q matrix includes the Query vector corresponding to each token, the K matrix includes the Key vector corresponding to each token, and the V matrix includes the Value vector corresponding to each token.

[0276] (2) The formula for calculating the global attention feature matrix is ​​shown in Formula 1 below: Formula 1: Attention(Q, K, V) = Softmax( )·V Where Attention(Q, K, V) represents the global attention feature matrix. The attention score matrix, This means calculating the dot product between each query vector and each key vector; The dimensions of the query vector and the key vector. This indicates scaling.

[0277] Softmax() represents the Softmax function, used to convert the attention score matrix into an attention weight matrix. The Softmax function operates on each row of the attention score matrix, transforming all elements in a row (representing the relevance of a query vector to all key vectors) into a probability distribution, making all values ​​positive, and the sum of all values ​​in each row equal to 1.

[0278] "·V" indicates that the attention weight matrix output by the Softmax function is multiplied by the V matrix. The essence of this multiplication is a weighted summation. The element in the i-th row and k-th column of the global attention feature matrix is ​​equal to the dot product of the i-th row of the attention weight matrix and the k-th column of the V matrix. In other words, the element in the i-th row and k-th column of the global attention feature matrix represents the element obtained by weighted summation of the k-th elements of all rows in the V matrix.

[0279] In some embodiments, the attention feature matrix described above is used as an attention feature representation.

[0280] In other embodiments, the global attention layer 1012 includes at least two attention heads. Each attention head performs the self-attention calculations described in (1) and (2) above, obtaining attention feature matrices output by the at least two attention heads respectively. The attention feature matrices output by the at least two attention heads are concatenated and subjected to a linear transformation to obtain an attention feature representation. Each attention head corresponds to a different W. Q W K and W V .

[0281] (3) Obtain the updated representation of each token through residual connection and normalization. For example, residual connection means connecting the tokens in the calculated attention feature representation to the tokens in the global sequence X in a one-to-one correspondence, such as connecting the camera token corresponding to the i-th video frame in the attention feature representation to the camera token corresponding to the i-th video frame in the global sequence X. Normalization means performing normalization processing on each token obtained by residual connection to obtain the updated global sequence.

[0282] Finally, the updated global sequence is grouped into T third feature representations (1022) by frame. Each third feature representation contains the updated image block token and the updated camera token (or contains the updated image block token, the updated camera token, and the updated register token).

[0283] Step 2 : Input at least two third feature representations into the intra-frame attention layer and output the second feature representations corresponding to at least two video frames respectively.

[0284] The intra-frame attention layer is used to perform intra-frame attention operations on at least two third feature representations respectively.

[0285] like Figure 10 As shown, T third feature representations 1022 are input into the intra-frame attention layer 1013 of the parameter prediction model 1010. The intra-frame attention layer 1013 performs an intra-frame attention operation on each third feature representation to obtain the second feature representation of each video frame, i.e., T second feature representations 1023. For example, the specific process of the intra-frame attention operation includes: concatenating the image block token and camera token in each third feature representation into a token sequence, denoted as X. i ∈R （K+1）×C X i This refers to the token sequence corresponding to the i-th third feature. For X... i The specific process of performing self-attention computation can be referred to the above process of performing self-attention computation on the global sequence X, and will not be repeated here. For X... iAfter performing self-attention computation, an updated token sequence is obtained. The updated token sequence is used as the i-th second feature representation, and each second feature representation contains the updated image patch token and the updated camera token (or contains the updated image patch token, the updated camera token, and the updated register token).

[0286] Step 3 : Input at least two second feature representations into the inter-frame attention layer and output the first feature representations corresponding to at least two video frames respectively.

[0287] The inter-frame attention layer is used to perform inter-frame attention operations on at least two second feature representations.

[0288] like Figure 10 As shown, the inter-frame attention layer 1014 of the parameter prediction model 1010 inputs T second feature representations 1023 into the T camera feature representations of the T second feature representations 1023. The inter-frame attention layer 1014 performs an inter-frame attention operation on the T camera feature representations in the T second feature representations 1023 to obtain the first feature representation of each video frame, i.e., T first feature representations 1024. For example, the specific process of the inter-frame attention operation includes: extracting all camera tokens from the T second feature representations 1023 to form a camera token sequence, denoted as G∈R. T×C The specific process of performing self-attention computation on G can be referred to the process of performing self-attention computation on the global sequence X described above, and will not be repeated here. After performing self-attention computation on G, the updated camera token sequence can be obtained. The updated camera token sequence is used as T first feature representations (1024), and each first feature representation contains the updated camera token.

[0289] In some embodiments, the "global attention layer 1012 + intra-frame attention layer 1013 + inter-frame attention layer 1014" in the parameter prediction model 1010 constitute an attention sub-model. This attention sub-model can include L units in the parameter prediction model 1010, and these L attention sub-models are sequentially connected. Specifically, the global attention layer 1012 of the first attention sub-model is connected to the embedding layer 1011 to receive T frame feature representations 1021 output by the embedding layer 1011; the inter-frame attention layer 1014 of the last attention sub-model is connected to the parameter prediction layer 1015, and the T first feature representations 1024 output by the last attention sub-model are output to the parameter prediction layer 1015. The L attention sub-models are used to alternately perform "global attention operation + intra-frame attention operation + inter-frame attention operation" L times on the T frame feature representations 1021.

[0290] In the above scheme, the parameter prediction model comprises a multi-layered structure including an embedding layer, a global attention layer, an intra-frame attention layer, and an inter-frame attention layer. This multi-layered structure allows for the layered extraction of feature representations from video frames. For example, the embedding layer extracts basic visual features, the global attention layer captures long-term dependencies, the intra-frame attention layer focuses on key regions within a single frame, and the inter-frame attention layer models temporal relationships. The clear division of labor among the network layers avoids feature confusion and improves the efficiency and accuracy of trajectory prediction.

[0291] Step 940: Input at least two first feature representations into the parameter prediction layer and output camera parameters corresponding to at least two video frames respectively.

[0292] In some embodiments, at least two first feature representations are input to the parameter prediction layer, and the camera extrinsic parameters corresponding to at least two video frames are output.

[0293] like Figure 10 As shown, T first feature representations 1024 are input into the parameter prediction layer 1015 of the parameter prediction model 1010. The parameter prediction layer 1015 performs camera parameter prediction on each first feature representation to obtain the camera extrinsic parameters of each video frame, i.e., T camera extrinsic parameters 1025. In some embodiments, the parameter prediction layer 1015 can also obtain the camera intrinsic parameters of each video frame by performing camera parameter prediction on each first feature representation.

[0294] For example, the specific process of camera parameter prediction includes: the parameter prediction layer 1015 performs self-attention calculation on the first feature representation (the specific process of self-attention calculation can be referred to the above process of performing self-attention calculation on the global sequence X, which will not be repeated here) to obtain the processed first feature representation; the parameter prediction layer 1015 performs linear transformation processing on the processed first feature representation to obtain camera parameters, which can be implemented as a 9-dimensional vector for example.

[0295] The first four dimensions of a 9-dimensional vector are rotational quaternions. The rotation quaternion of the i-th video frame is expressed by the following formula: Formula 2:

[0296] in, It is the real part of the rotation quaternion. It is the imaginary part of the rotation quaternion. The rotation quaternion is obtained through... The cosine information of the rotation angle is encoded through... It encodes the direction information of the rotation axis and the sine information of the rotation angle, thus enabling the non-singular description of three-dimensional rotational transformations.

[0297] A rotation quaternion can be converted into a rotation matrix using the following formula: Formula 3:

[0298] The 5th to 7th dimensions of a 9-dimensional vector are translation vectors. For example, the 5th to 7th dimension of the i-th video frame is t. i The translation vector and rotation matrix constitute the camera's extrinsic parameters.

[0299] The 8th and 9th dimensions of the 9-dimensional vector represent camera intrinsic parameters (such as field of view). The 8th dimension describes the camera's horizontal field of view, and the 9th dimension describes the camera's vertical field of view.

[0300] In some embodiments, after obtaining the camera extrinsic parameters, the predicted motion trajectory of the moving object is determined based on the camera extrinsic parameters corresponding to at least two video frames.

[0301] like Figure 10 As shown, after obtaining T camera parameters (1025), the predicted motion trajectory of the moving object is calculated based on the camera extrinsic parameters. For example, the process of calculating the predicted motion trajectory includes: concatenating the translation vectors in the camera extrinsic parameters corresponding to at least two video frames in timestamp order to obtain a position change sequence; concatenating the rotation vectors in the camera extrinsic parameters corresponding to at least two video frames in timestamp order to obtain a pose change sequence; using the position change sequence or the position change sequence and pose change sequence as the camera motion trajectory; and determining the camera motion trajectory as the predicted motion trajectory of the moving object.

[0302] In summary, the camera parameter prediction method provided in this application uses a camera parameter prediction model with an inter-frame attention layer to predict camera extrinsic parameters end-to-end, directly outputting camera extrinsic parameters from video frames, reducing error accumulation in intermediate stages and improving trajectory prediction accuracy.

[0303] In some embodiments, the camera parameter prediction method provided in this application can be used for trajectory prediction. For example, the above... Figure 3 The step 320 shown is followed by the following step 330.

[0304] Step 330: Determine the predicted motion trajectory of the moving object based on the camera parameters corresponding to at least two video frames.

[0305] In some embodiments, the camera motion trajectory corresponding to the first video is determined based on the camera parameters corresponding to at least two video frames; the predicted motion trajectory of the moving object is obtained based on the camera motion trajectory. For example, if the moving object is a vehicle (such as a vehicle that is autonomously driving), the predicted motion trajectory of the moving object refers to the vehicle's motion trajectory.

[0306] The camera parameters include camera extrinsic parameters.

[0307] Camera extrinsic parameters are used to express the camera pose corresponding to a video frame. For example, camera extrinsic parameters include a rotation matrix and a translation vector. For instance, [Ri|ti] represents the camera extrinsic parameters corresponding to the i-th video frame. Ri refers to the camera rotation matrix (3×3) in the i-th video frame, where each column of Ri represents the direction of the three axes (X, Y, Z) of the camera coordinate system in the world coordinate system, used to represent the camera pose; ti refers to the camera translation vector (3×1) in the i-th video frame. In some embodiments, the translation vector of the i-th video frame refers to the offset of the camera position in the i-th video frame relative to the origin of the world coordinate system.

[0308] The camera coordinate system is a local coordinate system established with the camera position as the origin and the camera pose as the coordinate axis direction. As the camera pose changes, the camera coordinate system changes accordingly; therefore, different video frames may correspond to different camera coordinate systems. In the camera coordinate system, the camera position is represented by local camera coordinates. The world coordinate system (or reference coordinate system, which is a global coordinate system) corresponds to the video scene represented by the first video. In the first video, the world coordinate system is fixed. For example, the world coordinate system can be set to the camera coordinate system corresponding to the first video frame. In subsequent video frames, changes in camera pose do not change the world coordinate system. In the world coordinate system, the camera position is represented by global camera coordinates.

[0309] In some embodiments, the camera motion trajectory corresponding to the first video is determined based on camera extrinsic parameters corresponding to at least two video frames; the predicted motion trajectory of the moving object is obtained based on the camera motion trajectory. The camera motion trajectory includes at least one of a position change sequence and a pose change sequence. (1) Sequence of position changes By concatenating the translation vectors in the camera extrinsic parameters corresponding to at least two video frames in the order of timestamps, a sequence of position changes is obtained.

[0310] For example, the translation vector of the i-th video frame refers to the position offset of the camera in the i-th video frame relative to the origin of the world coordinate system; assuming the world coordinate system is set to the camera coordinate system corresponding to the 1st video frame, then the translation vector t in the camera extrinsic parameters of the i-th video frame is... i This represents the coordinate offset of the camera in the i-th video frame relative to the camera in the first video frame, in the world coordinate system. Since the camera position in the first frame is the origin (0,0,0) of the world coordinate system, t... i This is directly equal to the global camera coordinates of the camera in the world coordinate system for the i-th video frame. Concatenating the T global camera coordinates in timestamp order yields the position change sequence.

[0311] (2) Posture change sequence By concatenating the rotation vectors in the camera extrinsic parameters corresponding to at least two video frames in the order of timestamps, a pose change sequence is obtained.

[0312] In some embodiments, the camera motion trajectory is determined as the predicted motion trajectory of the moving object. Alternatively, a fixed position offset between the camera and the moving object is obtained; the camera motion trajectory is adjusted based on the fixed position offset to obtain the predicted motion trajectory of the moving object.

[0313] In some embodiments, the position change sequence is adjusted according to a fixed position offset to obtain an adjusted position change sequence; the adjusted position change sequence and the attitude change sequence are used as the predicted motion trajectory of the moving object. For example, taking the moving object as vehicle A, there is a fixed position offset between the camera position and the vehicle A position (e.g., the center of gravity position). By uniformly applying this fixed position offset to the calculated T global camera coordinates, the position change sequence of vehicle A (i.e., the adjusted position change sequence) can be obtained. If the fixed offset is a vector Δp, then the global camera coordinates of vehicle A in the world coordinate system in the i-th video frame = the global camera coordinates of the camera in the world coordinate system in the i-th video frame + Δp. By concatenating the T global camera coordinates of vehicle A in the world coordinate system according to the timestamp order, the position change sequence of vehicle A can be obtained.

[0314] In some embodiments, if there are multiple moving objects corresponding to the first object, the camera motion trajectory corresponding to the first video is determined based on the camera parameters corresponding to at least two video frames; and the predicted motion trajectories corresponding to at least two moving objects are obtained based on the camera motion trajectory. The predicted motion trajectories corresponding to the at least two moving objects may be the same or different.

[0315] The method for determining the predicted motion trajectories corresponding to at least two moving objects includes at least one of the following: Method 1 The camera motion trajectory corresponding to the first video is used as the predicted motion trajectory for at least two moving objects.

[0316] That is, at least two moving objects have the same predicted motion trajectory, which is the camera motion trajectory corresponding to the first video.

[0317] For example, while the vehicle is in motion, an onboard camera is fixed to the windshield inside the vehicle to capture a first video of the road ahead. The moving objects corresponding to the first video include the vehicle and the driver inside the vehicle. After predicting the camera's motion trajectory in the first video, the camera's motion trajectory is used as the predicted motion trajectory of the vehicle and the driver inside the vehicle.

[0318] Method 2: Obtain the fixed position offsets between at least two moving objects and the camera; adjust the camera motion trajectory based on the fixed position offsets to obtain the predicted motion trajectories corresponding to at least two moving objects.

[0319] In some embodiments, different fixed position offsets correspond to different moving objects. Therefore, by adjusting the same camera trajectory based on these different fixed position offsets, different predicted motion trajectories can be obtained. Taking the w-th (where w is a positive integer) moving object out of at least two moving objects as an example: obtain the w-th fixed position offset between the w-th moving object and the camera; adjust the camera trajectory based on this w-th fixed position offset to obtain the predicted motion trajectory corresponding to the w-th moving object. The method for adjusting the camera trajectory based on the w-th fixed position offset can be found above and will not be repeated here.

[0320] For example, the drone formation includes a master drone and at least one slave drone, which maintain a fixed formation. The master drone carries a camera to capture a first video of the surrounding environment, which is a first-person view video of the master drone. After obtaining the camera motion trajectory corresponding to the first video, this camera motion trajectory is used as the flight trajectory of the master drone. Then, the fixed position offset between the slave drone and the master drone in the drone formation is obtained, and the camera motion trajectory is adjusted according to this fixed position offset to obtain the flight trajectory of the slave drone.

[0321] In summary, the camera parameter prediction method provided in this application predicts camera extrinsic parameters for each video frame in a first-person perspective video of a moving object, and then generates the motion trajectory of the moving object based on the camera extrinsic parameters of each video frame. This avoids the problem of inaccurate trajectory prediction caused by the scarcity of matching key points in related technologies, thereby improving the accuracy of trajectory prediction. Especially in the field of autonomous driving, the motion state of a vehicle can usually be simplified and simulated using a motion camera model. The camera parameter prediction method proposed in this application is based on this principle, and indirectly achieves accurate assessment of the motion state of autonomous vehicles by accurately predicting the camera motion trajectory. This not only improves the accuracy of assessing the degree of follow-up of autonomous driving video commands, but also provides strong protection for the safety and reliability of autonomous driving technology, bringing significant beneficial effects to the development and application of autonomous driving technology.

[0322] The following section uses the camera parameter prediction method provided in this application as an example to illustrate the trajectory prediction scheme in each application scenario, including autonomous driving, video generation, and AR scenarios.

[0323] (a) Model Evaluation In some embodiments, the first video is a video generated by a video generation model. (The above is an example.) Figure 3 The embodiment can also be implemented as follows: steps 1 to 5.

[0324] Step 1: Obtain the reference motion trajectory of the moving object corresponding to the reference video.

[0325] The reference video is a video captured from the first-person perspective of the moving object.

[0326] In some embodiments, the reference motion trajectory is a motion trajectory acquired by a motion acquisition device installed on the moving object.

[0327] In other embodiments, camera parameters of a reference video are determined based on the camera parameter prediction method provided in any of the above embodiments; and a reference motion trajectory of a moving object is determined based on the camera parameters of the reference video. For example, at least two video frames from the reference video are input into an embedding layer to obtain frame feature representations corresponding to at least two video frames respectively; the at least two frame feature representations are input into a global attention layer to output third feature representations corresponding to at least two video frames respectively; the at least two third feature representations are input into an intra-frame attention layer to output second feature representations corresponding to at least two video frames respectively; the at least two second feature representations are input into an inter-frame attention layer to output first feature representations corresponding to at least two video frames respectively; the at least two first feature representations are input into a parameter prediction layer to obtain camera extrinsic parameters corresponding to at least two video frames respectively; a camera motion trajectory is determined based on the camera extrinsic parameters corresponding to at least two video frames respectively; and a reference motion trajectory of the moving object is determined based on the camera motion trajectory.

[0328] For the technical details of obtaining the reference motion trajectory, please refer to the above description of predicting motion trajectory, which will not be repeated here.

[0329] Step 2: Generate the first video using the video generation model.

[0330] The first video is a video taken from the first-person perspective of the moving object.

[0331] In some embodiments, the video generation model is used to generate autonomous driving videos; the moving object includes a first vehicle; and a reference video is used to demonstrate a scenario where the first vehicle drives automatically according to control signals.

[0332] In some embodiments, the control signals include, but are not limited to, at least one of the following: (1) Environmental perception sensor signals: such as lidar signals, camera signals, millimeter-wave radar signals, etc. (2) Vehicle status feedback signals: driving status (such as real-time vehicle speed, acceleration / deceleration, driving mileage), control status (such as current steering angle, throttle opening, brake pedal travel, gear position), etc. (3) Control command signals: motion control (such as accelerating to 40km / h), attitude control (turning left 25°), decision commands (red light ahead, decelerate and stop, etc.). In some embodiments, the control command signals come from the decision layer of the autonomous driving algorithm and are a set of instructions that guide the first vehicle (autonomous driving vehicle) to perform specific driving actions.

[0333] In some embodiments, a control signal and a first video frame are input to a video generation model, and a first video is output. The first video frame is used to indicate the first N video frames in the reference video, where N is a positive integer. Exemplarily, the video generation model can be implemented as at least one of a video diffusion model, a generative adversarial network, etc., without limitation herein.

[0334] In some embodiments, the reference video and the first video are two videos time-aligned, such that the timestamp of the i-th video frame in the reference video is the same as the timestamp of the i-th video frame in the first video.

[0335] In the above scheme, the video generation model uses the control signal as an input feature to ensure that the generated first video is the result of the control signal. Thus, by using the difference in motion trajectory between the first video and the reference video, the instruction (control signal) following performance of the video generation model can be evaluated.

[0336] Step 3: Predict the camera extrinsic parameters corresponding to at least two video frames in the first video.

[0337] Step 4: Determine the predicted motion trajectory of the moving object based on the camera extrinsic parameters corresponding to at least two video frames.

[0338] Step 5: Determine the model evaluation results based on the predicted motion trajectory and the reference motion trajectory.

[0339] The model evaluation results are used to indicate the similarity of moving object trajectories between the first video and the reference video.

[0340] In some embodiments, the difference between the predicted motion trajectory and the reference motion trajectory is determined, and the model evaluation result is determined based on the difference in motion trajectory. For example, if the evaluation dataset contains only a single reference video, the difference in motion trajectory corresponding to that single reference video is used as the model evaluation result. If the evaluation dataset contains at least two reference videos, the model evaluation result is determined based on the difference in motion trajectory corresponding to each of the at least two reference videos. This could be achieved by calculating the average (or weighted average) of the difference in motion trajectory corresponding to each of the at least two reference videos as the model evaluation result; or by using the maximum (or minimum, median, or mode) of the difference in motion trajectory corresponding to each of the at least two reference videos as the model evaluation result. This application does not limit the scope of this method.

[0341] In some embodiments, if the model evaluation results do not meet the evaluation quality requirements, the video generation model is retrained. For example, if the model evaluation results (such as the difference in motion trajectories corresponding to a single reference video or the average of the differences in motion trajectories corresponding to at least two reference videos) are greater than a preset trajectory difference, the video generation model is retrained. In some embodiments, if the model evaluation results meet the evaluation quality requirements, the video generation model can be deployed to an application, such as a target server, for user use.

[0342] In some embodiments, the function for calculating the difference in motion trajectories includes, but is not limited to, at least one of the mean squared error loss function, L1 loss function, L2 loss function, etc., which is not limited here.

[0343] In some embodiments, the reference trajectory includes M reference trajectory points, and the predicted trajectory includes first trajectory points corresponding to the M reference trajectory points respectively; M is an integer greater than 1. The trajectory difference is determined based on the coordinate difference between the q-th reference trajectory point and the q-th first trajectory point; the model evaluation result is determined based on the trajectory difference; q is a positive integer and q≤M.

[0344] In some embodiments, the difference in motion trajectories is determined by at least one of the following losses: (1) First trajectory loss For example, both the predicted trajectory and the reference trajectory include a sequence of position changes. Based on the coordinate difference between the coordinates of the q-th reference trajectory point and the coordinates of the q-th first trajectory point, the loss of the q-th trajectory point is determined (e.g., the loss calculated according to the L2 loss function). The losses of the M trajectory points are added together to obtain the first trajectory loss, or the average of the losses of the M trajectory points is calculated as the first trajectory loss.

[0345] In some embodiments, the first trajectory loss is taken as the difference in motion trajectories.

[0346] (2) Second trajectory loss For example, the qth reference trajectory point is the trajectory endpoint of the reference motion trajectory; the qth first trajectory point is the trajectory endpoint of the predicted motion trajectory; based on the coordinate difference between the coordinates of the qth reference trajectory point and the coordinates of the qth first trajectory point, the loss of the qth trajectory point (such as the loss calculated according to the L2 loss function) is determined as the second trajectory loss.

[0347] In some embodiments, the second trajectory loss is taken as the difference in motion trajectories.

[0348] In (1) and (2), regarding the scheme for generating the loss of the q-th trajectory point: In some embodiments, both the predicted motion trajectory and the reference motion trajectory include a position change sequence and an attitude change sequence. The q-th position loss is determined based on the coordinate difference (e.g., translation vector difference) between the coordinates corresponding to the q-th reference trajectory point and the coordinates of the q-th first trajectory point; the q-th attitude loss is determined based on the attitude difference (e.g., rotation matrix difference) between the attitude corresponding to the q-th reference trajectory point and the attitude of the q-th first trajectory point; the q-th trajectory point loss is determined based on the q-th position loss and the q-th attitude loss. In some embodiments, the position loss and attitude loss can be calculated using the L2 loss function. In some embodiments, the sum of the q-th position loss and the q-th attitude loss is calculated as the q-th trajectory point loss.

[0349] (3) First trajectory loss + Second trajectory loss For example, after calculating the first trajectory loss and the second trajectory loss according to (1) and (2) above, the sum of the first trajectory loss and the second trajectory loss is calculated as the motion trajectory difference; or the product of the first weight and the first trajectory loss is obtained as the weighted first trajectory loss, the product of the second weight and the second trajectory loss is obtained as the weighted second trajectory loss, and the sum of the weighted first trajectory loss and the weighted second trajectory loss is calculated as the motion trajectory difference, wherein the first weight and the second weight are learnable parameters in the model training process; or the first weight and the second weight are preset parameters.

[0350] (4) Error loss For example, the video generation model processes at least two reference videos to generate multiple videos (including the first video). For each video, it determines the relationship between its corresponding trajectory loss and a preset error threshold. If the trajectory loss is greater than the preset error threshold, it is marked as a failed sample; if the trajectory loss is less than or equal to the preset error threshold, it is marked as a valid sample. The proportion of failed samples to the total number of videos in the at least two reference videos is calculated, and this proportion is used as the error loss. The trajectory loss can be implemented as a first trajectory loss, a second trajectory loss, the sum of the first and second trajectory losses, or a weighted sum of the first and second trajectory losses, etc., and is not limited here.

[0351] In some embodiments, the error loss is taken as the difference in motion trajectory.

[0352] In some embodiments, different types of moving objects correspond to different preset error thresholds. For example, when a video generation model is used to generate pedestrian motion videos, the preset error threshold is set to a first error threshold (the pedestrian target is small and the tolerance is low); when a video generation model is used to generate autonomous vehicle videos, the preset error threshold is set to a second error threshold (the vehicle size is large and the tolerance is high), wherein the first error threshold is less than the second error threshold.

[0353] In the above scheme, the difference in motion trajectory is determined by the difference in coordinates of trajectory points, thereby determining the model evaluation result. The difference in motion trajectory calculated point by point can accurately reflect the trajectory deviation, thereby improving the accuracy of the model evaluation result.

[0354] In summary, the camera parameter prediction method provided in this application can predict motion trajectories even when the video generation model cannot directly generate video motion trajectories. By utilizing the difference between the actual trajectory and the predicted trajectory (i.e., the reference motion trajectory and the predicted motion trajectory), the strength of the instruction following ability of the video generation model is measured, thus effectively evaluating the instruction following degree of the autonomous driving video generated by the video generation model.

[0355] In other embodiments, based on the aforementioned differences in motion trajectories, a motion trajectory loss is determined between the predicted motion trajectory and the reference motion trajectory; the video generation model is then trained based on this motion trajectory loss. For example, the model parameters of the video generation model are updated based on the motion trajectory loss to train the video generation model.

[0356] In some embodiments, the difference in motion trajectory is used as the motion trajectory loss.

[0357] It should be noted that the above example only illustrates the use of motion trajectory loss to train a video generation model. In practice, motion trajectory loss can also be combined with other losses (such as visual consistency loss) to train the video generation model; this is not limited here. Specifically, regarding visual consistency loss: it is determined based on the image similarity between video frames of the reference video and video frames of the first video.

[0358] Compared to related technologies that rely on video frame similarity-based training methods, the training method proposed in this application can more intuitively and accurately measure the ability of the generated video to follow instructions (i.e., control signals). While video frame similarity-based training methods often focus on loss calculations at the image vision level, this application achieves direct quantification of instruction following ability by predicting and comparing the difference between the actual trajectory and the predicted trajectory, thus improving the model training effect.

[0359] The above evaluation scheme will be illustrated below using the field of autonomous driving as an example.

[0360] Signal input-based autonomous driving video generation is a cutting-edge technology in the field of autonomous driving, integrating sensor data with video generation techniques. For example, by using multimodal inputs, including the first frame or multiple frames of video and control signals, a video generation model is trained to predict future video frames. The quality of the generated video frames can be evaluated using the camera parameter prediction method provided in this application. This evaluation scheme can be referred to as the autonomous driving video command follow-up evaluation method, which includes the following process: (1) Obtain the test set of the video generation model.

[0361] The test set includes the predicted video generated by the video generation model (i.e., the first video mentioned above), and the real video corresponding to the predicted video (i.e., the reference video mentioned above).

[0362] For example, the real video is the vehicle autonomous driving video actually captured in the physical environment. Each real video corresponds to a control signal. The control signal and the first frame or multiple frames of the real video are input into the video generation model, and the video generation model will output a predicted video corresponding to the real video.

[0363] (2) Input the real video and the predicted video into the trained camera parameter prediction model respectively to obtain the camera parameters corresponding to the real video and the predicted video respectively.

[0364] (3) Calculate the vehicle’s real trajectory based on the camera extrinsic parameters corresponding to the real video; calculate the vehicle’s predicted trajectory based on the camera extrinsic parameters corresponding to the predicted video.

[0365] (4) Calculate the L2 norm between the vehicle's actual trajectory and the vehicle's predicted trajectory to evaluate the instruction following performance of the autonomous driving video.

[0366] The aforementioned evaluation method directly quantifies the command-following capability by predicting and comparing the difference between the actual trajectory and the predicted trajectory. By quantifying the evaluation indicators, the assessment of the command-following ability in autonomous driving videos becomes more objective and accurate, providing strong support for the development of autonomous driving technology.

[0367] The evaluation method proposed in this application is not only applicable to traditional autonomous driving video instruction follow-up evaluation scenarios, but can also be widely applied to multiple fields such as video generation model optimization and autonomous driving technology improvement. No limitations are specified here.

[0368] (ii) Generation of 3D motion video In some embodiments, the first video is video captured by an action camera worn by a moving object (such as an athlete). Figure 3 The embodiment can also be implemented as follows: steps 1 to 4.

[0369] Step 1: Capture the first video using an action camera worn by the moving object.

[0370] The first video is a video taken from the first-person perspective of the moving object.

[0371] For example, the moving object can be an athlete, and the action camera can be worn on the athlete's head, hands, legs, etc., without limitation.

[0372] Step 2: Predict the camera extrinsic parameters corresponding to at least two video frames in the first video.

[0373] Step 3: Determine the predicted motion trajectory of the moving object based on the camera extrinsic parameters corresponding to at least two video frames.

[0374] Step 4: Based on the predicted motion trajectory and the 3D model of the moving object, generate a 3D motion video corresponding to the moving object.

[0375] In some embodiments, the moving object can be an athlete, and the first video can be a video captured by the athlete during training or competition. The resulting 3D motion video can then be used for reviewing the athlete's training or competition.

[0376] For example, the 3D model of a moving object is typically a 3D model with skeletal rigging (such as a 3D model of an athlete, including the skull, spine, limbs, and other skeletal joints, as well as the skin / muscle mesh that wraps the bones). The process of generating a 3D motion video corresponding to the moving object includes the following steps: (1) Determine the root bone position sequence of the three-dimensional model based on the position data of each trajectory point (such as translation vector). The root bone position sequence determines the overall movement of the unit model in three-dimensional space (such as when an athlete runs forward, the root bone moves along the positive x-axis of the track); determine the joint rotation angle sequence of the three-dimensional model based on the posture data of each trajectory point (such as rotation matrix).

[0377] In some embodiments, the position data of adjacent (adjacent timestamp) trajectory points are interpolated to calculate the root bone position at the intermediate time, ensuring smooth model movement (such as a natural stride transition when an athlete is running); the posture data of adjacent (adjacent timestamp) trajectory points are interpolated to calculate the joint rotation angle at the intermediate time, ensuring smooth limb swing (such as no sudden pause when an athlete swings their arm).

[0378] (2) Render the three-dimensional model based on the root bone position sequence and joint rotation angle sequence to obtain the three-dimensional motion video corresponding to the moving object.

[0379] In summary, the solution provided in this application only requires the moving object to wear a motion camera to collect first-person perspective video to obtain the object's motion data. It does not require complex hardware deployment or scene modification (such as multiple optical sensors in a professional motion capture studio or multiple inertial sensors attached to the human body), which reduces the cost of acquiring object motion data and improves the efficiency of generating 3D motion videos.

[0380] (III) AR Scenarios In some embodiments, the first video is a real-world scene video captured by an AR (Augmented Reality) device worn by the moving object. (The above...) Figure 3 The embodiment can also be implemented as follows: steps 1 to 4.

[0381] Step 1: Collect the first video using the AR device worn by the moving object.

[0382] The first video is a video taken from the first-person perspective of the moving object. For example, the moving object can be a player, and the first video could be a video of the real environment captured after the player wears an AR device.

[0383] Step 2: Predict the camera extrinsic parameters corresponding to at least two video frames in the first video.

[0384] Step 3: Determine the predicted motion trajectory of the moving object based on the camera extrinsic parameters corresponding to at least two video frames.

[0385] Step 4: Update the virtual objects in the AR screen based on the predicted motion trajectory.

[0386] For example, the goal of updating virtual objects in the AR screen is to ensure that the virtual objects maintain physical consistency with the real environment as the moving object (the user wearing the AR device) moves. That is, the position, angle and size of the virtual objects in the AR screen will change naturally with the user's movement, turning and other actions, just like real objects, avoiding problems such as virtual objects floating, misaligning, or shaking.

[0387] In some embodiments, the display coordinates of a virtual object in the current AR screen are calculated in real time based on the predicted motion trajectory. The virtual object in the AR screen is then updated based on the calculated display coordinates.

[0388] For example, taking a virtual object implemented as a static virtual object as an example, such as a virtual road sign in an AR navigation scenario, the specific process of updating the virtual road sign in the AR screen includes: (1) Determine the virtual landmark anchor points.

[0389] In an AR scene, a virtual landmark is pre-attached to an anchor point in a real-world coordinate system. The position and pose of the virtual landmark corresponding to this anchor point remain fixed. In some embodiments, the real-world coordinate system can be implemented as the world coordinate system corresponding to the first video.

[0390] (2) Based on the relative relationship between the predicted motion trajectory and the virtual landmark anchor point, the distance between the player and the virtual landmark and the angle of the player facing the virtual landmark are obtained.

[0391] (3) Based on the distance between the player and the virtual road sign and the angle of the player facing the virtual road sign, calculate the display coordinates of the virtual road sign in the current AR screen (display coordinates in the screen pixel coordinate system).

[0392] (4) Update the virtual road signs in the AR screen based on the displayed coordinates.

[0393] In summary, the solution provided in this application determines the player's movement trajectory by collecting real-world video footage after the player wears the AR device. This trajectory is then used to update virtual objects in the AR screen, allowing the virtual objects to change naturally as the player moves. For example, in an AR navigation scenario, a virtual road sign is attached to a traffic light at a real intersection. As the player walks towards the intersection, the arrow size changes with distance, and the angle adjusts as the player turns, preventing navigation misalignment.

[0394] Next, the process of the model optimization method provided in the embodiments of this application will be described.

[0395] Figure 11 This is a flowchart of a model optimization method provided in an embodiment of this application. The method is executed by a computer device, which may be... Figure 2The terminal 210 and / or server 220 are shown. The method includes the following steps 1110 and 1120.

[0396] Step 1110: Configure the inter-frame attention layer in the prediction model.

[0397] In some embodiments, the prediction model includes, but is not limited to, at least one of image processing models, video processing models, etc., and the prediction model has the ability to predict camera parameters for a single image. For example, the image processing model can be implemented as a model with a camera parameter prediction module, which is used to predict the camera parameters of a single image, such as a VGGT model. The video processing model can be implemented as a model with a camera parameter prediction module, which is used to predict the camera parameters of a single video frame, such as a VGGT model implementing the camera parameter prediction module in the video processing model.

[0398] In some embodiments, configuring an inter-frame attention layer in the prediction model includes: modifying the attention module of the prediction model based on the inter-frame attention layer (as described above in the attention sub-model), wherein the attention module is used to extract feature representations for camera parameter prediction, and the modified attention module can enhance the prediction model's ability to capture the temporal dependencies between video frames.

[0399] For example, the attention module includes at least one of a global attention layer and an inter-frame attention layer, and the method for modifying the attention module of the prediction model based on the inter-frame attention layer includes at least one of the following: (1) Modify it to include only the inter-frame attention layer.

[0400] (2) Modify it to connect the global attention layer to the inter-frame attention layer, with the global attention layer preceding the inter-frame attention layer.

[0401] (3) Modify the intra-frame attention layer to connect the inter-frame attention layer, with the intra-frame attention layer preceding the inter-frame attention layer.

[0402] (4) Modify the global attention layer to connect the intra-frame attention layer, and the intra-frame attention layer to connect the inter-frame attention layer, wherein the global attention layer is before the intra-frame attention layer and the intra-frame attention layer is before the inter-frame attention layer.

[0403] (5) Modify the frame attention layer to connect the global attention layer, and the global attention layer to connect the inter-frame attention layer, wherein the frame attention layer is before the global attention layer and the global attention layer is before the inter-frame attention layer.

[0404] (6) Modify the inter-frame attention layer to connect the intra-frame attention layer, wherein the intra-frame attention layer is before the inter-frame attention layer. In this case, at least one of the intra-frame attention layer and the global attention layer can be connected before the inter-frame attention layer, and the specific connection method can be any one of (2)-(6).

[0405] In some embodiments, the prediction model includes at least two attention modules, which are sequentially connected to extract feature representations for camera parameter prediction. In this application, at least one of the at least two attention modules can be modified, and the modification methods for different attention modules can be the same or different. For example, each of the at least two attention modules is modified according to the method (4) described above.

[0406] In some embodiments, the prediction model configured with an inter-frame attention layer includes a parameter prediction layer, which is used to predict camera parameters based on the feature representations extracted by the modified attention module.

[0407] In some embodiments, a prediction model configured with an inter-frame attention layer is trained to obtain a camera parameter prediction model; the camera parameter prediction model is used to perform the camera parameter prediction method involved in any of the above embodiments.

[0408] The training of the prediction model includes a first training phase and a second training phase. In the first training phase, the parameters of some network layers in the prediction model, including the inter-frame attention layer, are updated. In the second training phase, the parameters of all network layers in the prediction model are updated.

[0409] In some embodiments, the network layers that perform parameter updates during the first training phase may contain only inter-frame attention layers, or they may contain inter-frame attention layers and parameter prediction layers, etc., without limitation here.

[0410] For example, in the first training phase, the parameters (such as weights) of all network layers in the prediction model except for the inter-frame attention layer and the parameter prediction layer are frozen, and only the parameters (such as weights) of the inter-frame attention layer and the parameter prediction layer are updated. After the prediction model has initially converged, the second training phase begins, where the parameters of all network layers in the prediction model are unfrozen, and the prediction model is fully trained, that is, the parameters of all network layers in the prediction model are updated.

[0411] In some embodiments, before training the prediction model, a training dataset is obtained. The training dataset includes sample videos labeled with reference camera parameters, which include at least one of reference camera extrinsic parameters and reference camera intrinsic parameters. The prediction model configured with an inter-frame attention layer is trained using the training dataset to obtain a camera parameter prediction model.

[0412] In some embodiments, the training dataset is preprocessed, and the prediction model configured with an inter-frame attention layer is trained using the preprocessed training dataset to obtain a camera parameter prediction model. Preprocessing involves converting sample videos in the training dataset into a format suitable for the prediction model input (e.g., batch tensor format).

[0413] In some embodiments, a distributed system is used to train a prediction model configured with an inter-frame attention layer to obtain a camera parameter prediction model. Exemplarily, the distributed system includes a management node and multiple computing nodes; the training scheme in the distributed system includes: the management node initializes global model parameters of the prediction model configured with the inter-frame attention layer and broadcasts these global model parameters to all computing nodes; the management node divides the large-scale training dataset into multiple data subsets corresponding to each computing node and assigns each data subset to a corresponding computing node; each computing node independently performs gradient calculation using its data subset and the currently stored global model parameters; all computing nodes upload their calculated gradients to the management node. The management node aggregates all gradients using gradient averaging or gradient summation to obtain the global gradient. The management node updates the global model parameters based on the global gradient. The management node then broadcasts the updated global model parameters to all computing nodes again.

[0414] The training process for the prediction model configured with an inter-frame attention layer is described below. This prediction model with an inter-frame attention layer can be implemented as the sample camera parameter prediction model described below. The sample inter-frame attention layer in the sample camera parameter prediction model is the untrained inter-frame attention layer configured in the prediction model. The training process includes the following steps: Step 1: Obtain the first sample video.

[0415] The first sample video is a video captured from the first-person perspective of a moving object. This first sample video is used to train the sample camera parameter prediction model; the sample camera parameter prediction model includes a sample embedding layer, a sample global attention layer, a sample intra-frame attention layer, a sample inter-frame attention layer, and a sample parameter prediction layer.

[0416] The first sample video is labeled with reference camera parameters. In some embodiments, the reference camera parameters include reference camera extrinsic parameters and reference camera intrinsic parameters. For example, if you want to train a camera parameter prediction model for predicting vehicle motion trajectories in autonomous driving videos, the first sample video can be obtained from an autonomous driving dataset containing reference camera parameters. After obtaining the first sample video, the first sample video is preprocessed to convert it into a batch tensor form suitable for network layer input. Preprocessing may include adjusting the video format of the first sample video to a preset format (uniform format), adjusting the frame rate of the first sample video, etc., which are not limited here.

[0417] Step 2: Input at least two video frames from the first sample video into the sample embedding layer to obtain the frame feature representations corresponding to the at least two video frames respectively.

[0418] In some embodiments, at least two video frames from the first sample video are input into the sample embedding layer to obtain the first image feature representations corresponding to the at least two video frames respectively; the first image feature representation of each video frame is concatenated with the first camera feature representation to obtain the frame feature representations corresponding to the at least two video frames respectively.

[0419] In some embodiments, the first image feature representation of each video frame is concatenated with the first camera feature representation and the register feature representation to obtain frame feature representations corresponding to at least two video frames respectively. Exemplarily, each frame feature representation includes an image token, a camera token, and a register token. The image token carries the visual and geometric association features of the video frame; the camera token carries the camera parameter prediction features of the video frame; and the register token carries the world coordinate system and frame alignment features. In the frame feature representation, the camera token and register token are randomly initialized parameters that are learnable.

[0420] In some embodiments, the register tokens of the first video frame of the first sample video are different from those of other video frames (i.e., all other video frames besides the first video frame). During training, the register token of the first video frame learns the features of the world reference frame, while the register tokens of other video frames learn the alignment information between their own frame and the features of the world reference frame, so that the output camera parameters use the camera coordinate system of the first video frame as the world reference frame. For example, for other video frames, their register tokens learn how to align their own embedded feature representation, camera feature representation, and world coordinate system; for the first video frame, its register feature representation learns how to serve as a reference frame and adapt to other video frames.

[0421] Step 3: Input at least two frame feature representations into the inter-frame attention layer of the sample, and output the first feature representations corresponding to at least two video frames respectively.

[0422] In some embodiments, the process of obtaining the first feature representation further includes the following steps: inputting at least two frame feature representations into a global attention layer of the samples, and outputting third feature representations corresponding to at least two video frames respectively; inputting at least two third feature representations into an intra-frame attention layer of the samples, and outputting second feature representations corresponding to at least two video frames respectively; inputting at least two second feature representations into an inter-frame attention layer of the samples, and outputting first feature representations corresponding to at least two video frames respectively.

[0423] Step 4: Input at least two first feature representations into the sample parameter prediction layer, and output the predicted camera parameters corresponding to at least two video frames respectively.

[0424] In some embodiments, the predicted camera parameters include predicted camera extrinsic parameters and predicted camera intrinsic parameters.

[0425] Step 5: Determine the training loss based on the predicted camera parameters and the reference camera parameters.

[0426] In some embodiments, the difference between the predicted camera parameters and the reference camera parameters is calculated based on a preset loss function to obtain the camera parameter loss; the training loss is determined based on the camera parameter loss.

[0427] In some embodiments, based on a preset loss function, the difference between the predicted camera extrinsic parameters and the reference camera extrinsic parameters is calculated, as well as the difference between the predicted camera intrinsic parameters and the reference camera intrinsic parameters, to obtain the camera parameter loss. The preset loss function includes at least one of the following: L1 loss function, L2 loss function, etc., but is not limited thereto.

[0428] Step 6: Train the camera parameter prediction model based on the training loss to obtain the camera parameter prediction model.

[0429] For example, the model parameters in the sample camera parameter prediction model are updated according to the training loss to obtain the camera parameter prediction model.

[0430] In some embodiments, the training phase of the sample camera parameter prediction model includes a first training phase and a second training phase. The first training phase corresponds to a first training dataset, and the second training phase corresponds to a second training dataset. The first training dataset and the second training dataset are the same, or they are different. Both the first training dataset and the second training dataset include sample data pairs (such as the first sample video and reference camera parameters mentioned above).

[0431] In some embodiments, during the first training phase, a first training loss is calculated using sample data from the first training dataset; the model parameters of the sample inter-frame attention layer and the sample parameter prediction layer in the sample camera parameter prediction model are updated using the first training loss to obtain the camera parameter prediction model for the first training phase.

[0432] In some embodiments, in response to the camera parameter prediction model of the first training stage meeting the first training requirement, the camera parameter prediction model of the first training stage is trained in a second training stage. Exemplarily, the first training requirement includes, but is not limited to: the number of model iterations in the first training stage reaches a first preset number, and the first training loss corresponding to the trained camera parameter prediction model of the first training stage is less than the first preset loss.

[0433] In some embodiments, during the second training phase, a second training loss is calculated using sample data from the second training dataset; the model parameters in the camera parameter prediction model of the first training phase are updated using the second training loss, resulting in a camera parameter prediction model for the second training phase, which is then used as the final trained camera parameter prediction model. In some embodiments, in response to the second training phase camera parameter prediction model meeting second training requirements, the camera parameter prediction model meeting the second training requirements is used as the final trained camera parameter prediction model. Exemplarily, the second training requirements include, but are not limited to: the number of model iterations in the second training phase reaching a second preset number, and the second training loss corresponding to the trained second training phase camera parameter prediction model being less than the second preset loss.

[0434] In some embodiments, the second training phase includes at least one sub-training phase, each sub-training phase corresponding to a sub-training dataset. The sub-training dataset is a portion of the data in the second training dataset, or the sub-training dataset is the second training dataset itself. For example, the model parameters that need to be updated differ for each sub-training phase. For instance, in the first sub-training phase, the model parameters of the "sample embedding layer + sample inter-frame attention layer + sample parameter prediction layer" are updated; in the second sub-training phase, the model parameters of the "sample embedding layer + sample global attention layer + sample inter-frame attention layer + sample parameter prediction layer" are updated; and in the third sub-training phase, the model parameters of the "sample embedding layer + sample global attention layer + sample intra-frame attention layer + sample inter-frame attention layer + sample parameter prediction layer" are updated, etc. This process is not repeated here. If the camera parameter prediction model trained in the current sub-training phase meets the training requirements, the training process for the next sub-training phase begins. The specific content of the training requirements is referred to the second training requirements described above, and will not be repeated here. Then, the camera parameter prediction model that meets the training requirements and is trained in the last sub-training phase is taken as the final trained camera parameter prediction model.

[0435] It should be noted that the specific processes of "calculating the first training loss using sample data from the first training dataset" and "calculating the second training loss using sample data from the second training dataset" can be referred to in steps 1-5, where the training loss is calculated using the first sample video and the reference camera parameters. These processes will not be repeated here.

[0436] In some embodiments, the training process described above can be performed in a distributed system. Exemplarily, the distributed system includes a management node and at least two compute nodes. Each compute node includes at least two GPUs, which carry out the computational process for model training. Exemplarily, each compute node loads a slice of the training dataset (such as a first or second training dataset), and each compute node loads the same model architecture and initializes the same model parameters. Each compute node independently calculates the local training loss based on the slice (refer to the loss calculation logic in steps 1-5) and backpropagates to obtain the local gradient. The management node aggregates the local gradients of all compute nodes, updates the global parameters (such as the parameters of the inter-frame attention layer + parameter prediction layer in the first training phase), and each compute node synchronizes its parameters before entering the next iteration. When the training requirements are met, the management node saves the updated global parameters and uses these global parameters and the model architecture as the final trained camera parameter prediction model.

[0437] It should be noted that details of the embodiments not covered in the above training process can be found in the above camera parameter prediction method embodiments, and will not be repeated here.

[0438] Step 1120: Perform inter-frame attention operations on at least two video frames in the video through an inter-frame attention layer to predict the camera parameters corresponding to the at least two video frames respectively.

[0439] Among them, the inter-frame attention operation is used to fuse at least two video frames based on the temporal relationship between at least two video frames.

[0440] In some embodiments, a video (such as the first video, reference video, etc. mentioned above) is input into a trained camera parameter prediction model. The camera parameter prediction model performs inter-frame attention operations on at least two video frames in the video through its inter-frame attention layer to predict the camera parameters corresponding to at least two video frames respectively. The specific prediction process can be referred to the above-described camera parameter prediction method embodiments, and will not be repeated here.

[0441] In some embodiments, an evaluation system is constructed based on a camera parameter prediction model, which is obtained by training a prediction model configured with an inter-frame attention layer.

[0442] In the evaluation system, camera parameter prediction is performed on the first video and the reference video using a camera parameter prediction model to obtain the camera parameters corresponding to the reference video and the first video, respectively. Based on the camera parameters corresponding to the reference video, a reference motion trajectory is obtained. Based on the camera parameters corresponding to the first video, a predicted motion trajectory is obtained. Based on the predicted motion trajectory and the reference motion trajectory, an evaluation result is determined, which indicates the similarity of the moving object trajectories in the first video and the reference video. For example, the L2 norm between the predicted motion trajectory and the reference motion trajectory is calculated as the evaluation result.

[0443] In summary, the model optimization method provided in this application predicts camera parameters by configuring an inter-frame attention layer in the prediction model and using this layer to perform inter-frame attention operations on at least two video frames. By configuring an inter-frame attention layer for the model, the model can capture the temporal dependencies between video frames, thereby fully utilizing the complementarity of multi-frame information when predicting camera parameters. This reduces the fluctuations or errors in prediction parameters caused by isolated processing of single-frame images, and improves the accuracy and robustness of camera parameter prediction.

[0444] The model optimization scheme provided in this application can be used for model evaluation. Taking the field of autonomous driving as an example, the model optimization scheme provided in this application can be used to build an evaluation system and perform the task of evaluating the degree of follow-up of autonomous driving video instructions. The scheme for building the evaluation system and performing the evaluation task is described below.

[0445] (a) Data acquisition and preprocessing.

[0446] We collect an autonomous driving dataset, where each video segment includes camera parameters (i.e., the camera parameters mentioned above). We preprocess each video segment in the autonomous driving dataset, converting it into a batch tensor format suitable for input to the model's network layers. This preprocessing includes format standardization and frame rate adjustment to facilitate subsequent network input.

[0447] (ii) VGGT model configuration.

[0448] The VGGT model is used to predict camera parameters for a single image. After configuring the VGGT model with an inter-frame attention layer, its ability to capture the temporal dependencies between video frames can be improved. The main goal of the configuration is to enable the model to better understand the temporal information in the video, thereby more accurately evaluating the degree of instruction follow-up in autonomous driving videos.

[0449] In the example chosen from the VGGT model, to enhance the model's sensitivity to temporal information, a temporal attention module (i.e., an inter-frame attention layer) is added after the global attention layer and intra-frame attention layer. This enhances the VGGT model's ability to capture temporal dependencies between video frames, thereby improving its ability to capture dynamic information. In the VGGT model, the global attention layer, along with the intra-frame and inter-frame attention layers, work together to process the input video and predict camera parameters for at least two video frames.

[0450] (III) Model training.

[0451] A distributed system is used to fine-tune the VGGT model configured with an inter-frame attention layer: In the early stages of training (i.e., the first training phase), the focus is on training the inter-frame attention layer and parameter prediction layer configured in the model, while the weights of the other network layers are temporarily frozen. After the model has initially converged, all weights of the model are gradually unfrozen for full training.

[0452] (iv) Evaluation system construction.

[0453] After the VGGT model with inter-frame attention layer is trained, an evaluation system is constructed. This system includes a camera parameter prediction model, which is the model trained on the VGGT model with the inter-frame attention layer. The evaluation system is used to evaluate the degree of follow-up of autonomous driving video commands by the video generation model. The specific details are as follows: (1) Obtain the test set of the video generation model.

[0454] The test set includes the predicted video generated by the video generation model (i.e., the first video mentioned above), and the real video corresponding to the predicted video (i.e., the reference video mentioned above).

[0455] The process of obtaining the predicted video is as follows: the control signal and the first frame or multiple frames of the real video are input into the video generation model, and the video generation model will output the predicted video corresponding to the real video.

[0456] (2) Input the real video and the predicted video into the trained camera parameter prediction model respectively to obtain the camera parameters corresponding to the real video and the predicted video respectively.

[0457] (3) Calculate the vehicle’s real trajectory based on the camera extrinsic parameters corresponding to the real video; calculate the vehicle’s predicted trajectory based on the camera extrinsic parameters corresponding to the predicted video.

[0458] (4) Calculate the L2 norm between the vehicle's actual trajectory and the vehicle's predicted trajectory to evaluate the instruction following performance of the autonomous driving video.

[0459] In the above scheme, an inter-frame attention mechanism is introduced into the VGGT model, significantly improving the model's ability to capture and process temporal information, enabling it to deeply mine the temporal correlations in video data. The modified VGGT model can accurately predict the camera's intrinsic and extrinsic parameters when processing dynamic video scenes, thus effectively predicting the vehicle's trajectory in high-speed dynamic scenes of autonomous driving. Furthermore, the modified VGGT model can predict trajectories even when the video generation model cannot directly generate them, broadening the application scenarios for evaluating the degree of video command following in autonomous driving and providing new ideas for the development of autonomous driving technology.

[0460] In the field of autonomous driving, the motion state of a vehicle can typically be simplified and simulated using motion camera models. The aforementioned technical solution is based on this principle, indirectly achieving an accurate assessment of the motion state of autonomous vehicles by precisely predicting the camera's trajectory. This not only improves the accuracy of assessing the degree of follow-up to autonomous driving video commands but also provides strong assurance for the safety and reliability of autonomous driving technology.

[0461] In summary, this application improves the VGGT model by introducing an inter-frame attention mechanism, thereby enhancing the model's performance in handling dynamic video scenes and enabling accurate prediction of the trajectory of autonomous vehicles. This has brought significant benefits to the development and application of autonomous driving technology.

[0462] For example, please refer to Figure 12 It shows a structural block diagram of a camera parameter prediction device, such as Figure 12 As shown, the device includes: Module 1210 is used to acquire the first video; The first prediction module 1220 is used to perform inter-frame attention operation on at least two video frames in the first video to obtain camera parameters corresponding to the at least two video frames respectively. The inter-frame attention operation is used to fuse the at least two video frames based on the temporal relationship between them, and the camera parameters are used to determine the predicted motion trajectory of the moving object corresponding to the first video.

[0463] In some embodiments, the first prediction module 1220 is configured to obtain camera feature representations corresponding to the at least two video frames respectively; the camera feature representations are feature representations associated with the camera pose corresponding to the video frames; and perform inter-frame attention operations on the at least two camera feature representations to obtain camera parameters corresponding to the at least two video frames respectively.

[0464] In some embodiments, the at least two video frames include an i-th video frame and a j-th video frame, where i and j are positive integers; the first prediction module 1220 is configured to obtain an inter-frame fusion feature representation between the j-th video frame and the i-th video frame based on the camera feature representation of the i-th video frame and the camera feature representation of the j-th video frame; the j-th video frame is any one of the video frames in the first video; when there are two or more video frames in the first video, the inter-frame fusion feature representations between each video frame in the first video and the i-th video frame are fused to obtain a first feature representation corresponding to the i-th video frame.

[0465] In some embodiments, the first prediction module 1220 is configured to determine a time decay coefficient based on the time interval between the i-th video frame and the j-th video frame; determine a fusion weight between the i-th video frame and the j-th video frame based on the time decay coefficient; and perform weighted fusion processing on the inter-frame fusion feature representation between each video frame in the first video and the i-th video frame according to the fusion weight, to obtain a first feature representation corresponding to the i-th video frame.

[0466] In some embodiments, the at least two video frames include K video frames, where K is an integer greater than 1; the K video frames are a sequence of video frames extracted from the at least two video frames with the i-th video frame as the center; the first prediction module 1220 is used to fuse the inter-frame fusion feature representations between each of the K video frames and the i-th video frame to obtain a first feature representation corresponding to the i-th video frame.

[0467] In some embodiments, the first prediction module 1220 is configured to perform an embedding operation on at least two video frames in the first video to obtain frame feature representations corresponding to the at least two video frames respectively; and obtain camera feature representations corresponding to the at least two video frames based on the at least two frame feature representations respectively.

[0468] In some embodiments, the first video is a video acquired from the first-person perspective of a moving object; the acquisition module 1210 is used to acquire a second video; the second video is a video acquired from the third-person perspective of a moving object; the first prediction module 1220 is used to perform an embedding operation on at least two video frames in the second video to obtain second frame feature representations corresponding to at least two video frames in the second video respectively; perform an embedding operation on at least two video frames in the first video to obtain first frame feature representations corresponding to at least two video frames in the first video respectively; and fuse at least two first feature representations and at least two second feature representations to obtain frame feature representations corresponding to at least two video frames in the first video respectively.

[0469] In some embodiments, the first prediction module 1220 is configured to perform intra-frame attention operations on at least two video frames in the first video to obtain second feature representations corresponding to the at least two video frames respectively; the intra-frame attention operations are used to fuse feature representations in a single video frame; and to perform inter-frame attention operations based on at least two second feature representations to obtain camera parameters corresponding to the at least two video frames respectively.

[0470] In some embodiments, the first prediction module 1220 is configured to perform a global attention operation on at least two video frames in the first video to obtain third feature representations corresponding to the at least two video frames respectively. The global attention operation is used to fuse the feature representations in the i-th video frame and fuse the feature representations of the i-th video frame with those of other video frames. The module is also configured to perform intra-frame attention operations on the at least two third feature representations to obtain second feature representations corresponding to the at least two video frames respectively.

[0471] In some embodiments, the first prediction module 1220 is configured to perform a global attention operation on at least two video frames in the first video to obtain a third feature representation corresponding to each of the at least two video frames; and perform an inter-frame attention operation based on the at least two third feature representations to obtain camera parameters corresponding to each of the at least two video frames.

[0472] In some embodiments, the first prediction module 1220 is configured to perform inter-frame attention operations on at least two video frames in the first video using a camera parameter prediction model to obtain camera parameters corresponding to the at least two video frames respectively; wherein, the camera parameter prediction model includes an inter-frame attention layer, and the inter-frame attention layer is used to perform the inter-frame attention operations.

[0473] In some embodiments, the camera parameter prediction model further includes an embedding layer; the embedding layer is used to perform an embedding operation; the first prediction module 1220 is used to perform an embedding operation on at least two video frames in the first video through the embedding layer to obtain frame feature representations corresponding to the at least two video frames respectively; and based on the at least two frame feature representations, obtain camera feature representations corresponding to the at least two video frames respectively.

[0474] In some embodiments, the camera parameter prediction model further includes an intra-frame attention layer; the intra-frame attention layer is used to perform intra-frame attention operations; the first prediction module 1220 is used to perform intra-frame attention operations on at least two video frames in the first video through the intra-frame attention layer to obtain second feature representations corresponding to the at least two video frames respectively; based on the at least two second feature representations, perform inter-frame attention operations through the inter-frame attention layer to obtain camera parameters corresponding to the at least two video frames respectively.

[0475] In some embodiments, the camera parameter prediction model further includes a global attention layer; the global attention layer is used to perform global attention operations; the first prediction module 1220 is used to perform global attention operations on at least two video frames in the first video through the global attention layer to obtain third feature representations corresponding to the at least two video frames respectively; and to perform intra-frame attention operations on the at least two third feature representations respectively through the intra-frame attention layer to obtain second feature representations corresponding to the at least two video frames respectively.

[0476] In some embodiments, the camera parameter prediction model further includes a global attention layer; the first prediction module 1220 is configured to perform a global attention operation on at least two video frames in the first video through the global attention layer to obtain a third feature representation corresponding to each of the at least two video frames; and based on the at least two third feature representations, perform an inter-frame attention operation through the inter-frame attention layer to obtain the camera parameters corresponding to each of the at least two video frames.

[0477] In some embodiments, the first prediction module 1220 is used to determine the camera motion trajectory corresponding to the first video based on the camera parameters corresponding to the at least two video frames respectively; and to obtain the predicted motion trajectory of the moving object based on the camera motion trajectory.

[0478] In some embodiments, the first video is a video generated by a video generation model; the first prediction module 1220 is used to obtain a reference motion trajectory of the moving object corresponding to the reference video; based on the predicted motion trajectory and the reference motion trajectory, a model evaluation result is determined, and the model evaluation result is used to indicate the similarity of the motion object trajectories of the first video and the reference video.

[0479] For example, please refer to Figure 13 It shows a structural block diagram of a model optimization device, such as Figure 13 As shown, the device includes: Configuration module 1310 is used to configure inter-frame attention layers in the prediction model; The second prediction module 1320 is used to perform inter-frame attention operations on at least two video frames in the video through the inter-frame attention layer in order to predict the camera parameters corresponding to the at least two video frames respectively. The inter-frame attention operation is used to fuse the at least two video frames based on the temporal relationship between them.

[0480] In some embodiments, the model optimization apparatus further includes: The training module is used to train the prediction model configured with the inter-frame attention layer to obtain a camera parameter prediction model; the training includes a first training phase and a second training phase; wherein, in the first training phase, the parameters of some network layers in the prediction model, including the inter-frame attention layer, are updated; and in the second training phase, the parameters of all network layers in the prediction model are updated.

[0481] In some embodiments, the model optimization apparatus further includes: A construction module is used to build an evaluation system based on a camera parameter prediction model, wherein the camera parameter prediction model is obtained by training a prediction model configured with the inter-frame attention layer; In the evaluation system, camera parameter prediction is performed on the first video and the reference video respectively using the camera parameter prediction model to obtain the camera parameters corresponding to the reference video and the first video respectively; a reference motion trajectory is obtained based on the camera parameters corresponding to the reference video; a predicted motion trajectory is obtained based on the camera parameters corresponding to the first video; and an evaluation result is determined based on the predicted motion trajectory and the reference motion trajectory. The evaluation result is used to indicate the similarity of the motion object trajectories in the first video and the reference video.

[0482] It should be noted that the specific limitations of one or more of the device embodiments provided above can be found in the limitations of the method above, and will not be repeated here. Each module of the above device can be implemented entirely or partially by software, hardware, or a combination thereof. Each module can be embedded in the processor of the computer device in hardware form or independent of the processor, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0483] This application also provides a computer device, which includes: a processor and a memory, wherein the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the camera parameter prediction method provided in the above-described method embodiments.

[0484] For example, Figure 14 This is a structural block diagram of a computer device 1400 provided in an exemplary embodiment of this application. In some embodiments, the computer device 1400 is a server 1400.

[0485] Typically, server 1400 includes a processor 1401 and memory 1402.

[0486] Processor 1401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1401 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1401 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0487] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1402 are used to store at least one instruction, which is executed by the processor 1401 to implement the camera parameter prediction method provided in the various method embodiments of this application.

[0488] In some embodiments, server 1400 may optionally include an input interface 1403 and an output interface 1404. Processor 1401, memory 1402, and input interfaces 1403 and 1404 can be connected via a bus or signal lines. Various peripheral devices can be connected to input interfaces 1403 and 1404 via a bus, signal lines, or a circuit board. Input interfaces 1403 and 1404 can be used to connect at least one input / output (I / O) related peripheral device to processor 1401 and memory 1402. In some embodiments, processor 1401, memory 1402, and input interfaces 1403 and 1404 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1401, memory 1402, and input interfaces 1403 and 1404 can be implemented on separate chips or circuit boards, and this application does not limit this.

[0489] Those skilled in the art will understand that Figure 14 The structure shown does not constitute a limitation on the computer device 1400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0490] In an exemplary embodiment, this application provides a chip including programmable logic circuits and / or program instructions, which, when run on a computer device, are used to implement the camera parameter prediction methods provided in the above-described method embodiments.

[0491] In an exemplary embodiment, this application provides a computer-readable storage medium storing a computer program that is loaded and executed by a processor to implement the camera parameter prediction method provided in the above-described method embodiments.

[0492] In an exemplary embodiment, this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the processor of the computer device to load and execute the camera parameter prediction method provided in the above-described method embodiments.

[0493] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0494] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0495] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0496] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A camera parameter prediction method, characterized by, The method comprises: acquiring a first video; performing an inter-frame attention operation on at least two video frames in the first video to obtain camera parameters corresponding to the at least two video frames respectively; wherein the inter-frame attention operation is used to fuse the at least two video frames based on a time sequence relationship between the at least two video frames, and the camera parameters are used to determine a predicted motion trajectory of a moving object corresponding to the first video.

2. The method of claim 1, wherein, The inter-frame attention operation on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames respectively comprises: acquiring camera feature representations corresponding to the at least two video frames respectively; the camera feature representations are feature representations associated with camera poses corresponding to the video frames; performing an inter-frame attention operation on at least two camera feature representations to obtain camera parameters corresponding to the at least two video frames respectively.

3. The method of claim 2, wherein, The at least two video frames comprise an i-th video frame and a j-th video frame, i and j being positive integers; The inter-frame attention operation on the at least two camera feature representations comprises: acquiring an inter-frame fusion feature representation between the j-th video frame and the i-th video frame according to the camera feature representation of the i-th video frame and the camera feature representation of the j-th video frame; the j-th video frame is any one of the video frames in the first video; In the case that there are two or more video frames in the first video, fusing the inter-frame fusion feature representations between each video frame in the first video and the i-th video frame respectively to obtain a first feature representation corresponding to the i-th video frame.

4. The method of claim 3, wherein, The method further comprises: determining a time decay coefficient based on a time interval between the i-th video frame and the j-th video frame; determining a fusion weight between the i-th video frame and the j-th video frame based on the time decay coefficient; The fusion of the inter-frame fusion feature representations between each video frame in the first video and the i-th video frame respectively to obtain the first feature representation corresponding to the i-th video frame comprises: performing a weighted fusion process on the inter-frame fusion feature representations between each video frame in the first video and the i-th video frame respectively according to the fusion weight to obtain the first feature representation corresponding to the i-th video frame.

5. The method of claim 2, wherein, The at least two video frames comprise K video frames, K being an integer greater than 1; the K video frames are a video frame sequence centered on the i-th video frame and intercepted in the at least two video frames; The inter-frame attention operation on the at least two camera feature representations comprises: fusing the inter-frame fusion feature representations between each video frame in the K video frames and the i-th video frame respectively to obtain the first feature representation corresponding to the i-th video frame.

6. The method of claim 2, wherein, The acquisition of the camera feature representations corresponding to the at least two video frames respectively comprises: performing an embedding operation on at least two video frames in the first video to obtain frame feature representations corresponding to the at least two video frames respectively; acquiring camera feature representations corresponding to the at least two video frames respectively based on at least two frame feature representations.

7. The method of claim 6, wherein, The first video is a video acquired in a first-person view of the motion object; The method further comprises: acquiring a second video; the second video is a video acquired in a third-person view of the motion object; performing embedding operation on at least two video frames in the second video to obtain second frame feature representations corresponding to the at least two video frames in the second video respectively; The embedding operation performed on the at least two video frames in the first video to obtain the frame feature representations corresponding to the at least two video frames respectively comprises: performing embedding operation on at least two video frames in the first video to obtain first frame feature representations corresponding to the at least two video frames in the first video respectively; fusing the at least two first feature representations and the at least two second feature representations to obtain the frame feature representations corresponding to the at least two video frames in the first video respectively.

8. The method according to any one of claims 1 to 7, characterized in that, The inter-frame attention operation performed on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames respectively comprises: performing intra-frame attention operation on the at least two video frames in the first video respectively to obtain second feature representations corresponding to the at least two video frames respectively; the intra-frame attention operation is used for fusing feature representations in a single video frame; performing inter-frame attention operation based on the at least two second feature representations to obtain the camera parameters corresponding to the at least two video frames respectively.

9. The method of claim 8, wherein, The intra-frame attention operation performed on the at least two video frames in the first video respectively to obtain the second feature representations corresponding to the at least two video frames respectively comprises: performing global attention operation on the at least two video frames in the first video to obtain third feature representations corresponding to the at least two video frames respectively; the global attention operation is used for fusing feature representations in an i-th video frame, and fusing feature representations of the i-th video frame and other video frames; performing intra-frame attention operation on the at least two third feature representations respectively to obtain the second feature representations corresponding to the at least two video frames respectively.

10. The method according to any one of claims 1 to 7, characterized in that, The inter-frame attention operation performed on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames respectively comprises: performing global attention operation on the at least two video frames in the first video to obtain third feature representations corresponding to the at least two video frames respectively; performing inter-frame attention operation based on the at least two third feature representations to obtain the camera parameters corresponding to the at least two video frames respectively.

11. The method according to any one of claims 1 to 7, characterized in that, The inter-frame attention operation performed on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames respectively comprises: performing inter-frame attention operation on the at least two video frames in the first video by a camera parameter prediction model to obtain the camera parameters corresponding to the at least two video frames respectively; The camera parameter prediction model comprises an inter-frame attention layer, and the inter-frame attention layer is used for performing the inter-frame attention operation.

12. The method of claim 11, wherein, The camera parameter prediction model further comprises an embedding layer; the embedding layer is used for performing embedding operation; Before the camera parameter prediction model performs the inter-attention operation on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames, the method further includes: performing embedding operation on the at least two video frames in the first video by the embedding layer to obtain frame feature representations corresponding to the at least two video frames, respectively; obtaining camera feature representations corresponding to the at least two video frames based on the at least two frame feature representations.

13. The method of claim 11, wherein, The camera parameter prediction model further includes an intra-attention layer, and the intra-attention layer is configured to perform an intra-attention operation. The camera parameter prediction model performs the inter-attention operation on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames, including: performing the intra-attention operation on the at least two video frames in the first video by the intra-attention layer to obtain second feature representations corresponding to the at least two video frames, respectively; performing the inter-attention operation on the at least two video frames in the first video by the inter-attention layer to obtain the camera parameters corresponding to the at least two video frames based on the at least two second feature representations.

14. The method of claim 13, wherein, The camera parameter prediction model further includes a global attention layer, and the global attention layer is configured to perform a global attention operation. The camera parameter prediction model performs the intra-attention operation on the at least two video frames in the first video by the intra-attention layer to obtain the second feature representations corresponding to the at least two video frames, including: performing the global attention operation on the at least two video frames in the first video by the global attention layer to obtain third feature representations corresponding to the at least two video frames, respectively; performing the intra-attention operation on the at least two third feature representations by the intra-attention layer to obtain the second feature representations corresponding to the at least two video frames, respectively.

15. The method of claim 11, wherein, The camera parameter prediction model further includes a global attention layer. The camera parameter prediction model performs the inter-attention operation on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames, including: performing the global attention operation on the at least two video frames in the first video by the global attention layer to obtain third feature representations corresponding to the at least two video frames, respectively; performing the inter-attention operation on the at least two video frames in the first video by the inter-attention layer to obtain the camera parameters corresponding to the at least two video frames based on the at least two third feature representations.

16. The method according to any one of claims 1 to 7, characterized in that, After the camera parameter prediction model performs the inter-attention operation on the at least two video frames in the first video to obtain the camera parameters corresponding to the at least two video frames, the method further includes: determining a camera motion trajectory corresponding to the first video based on the camera parameters corresponding to the at least two video frames, respectively; obtaining a predicted motion trajectory of the moving object according to the camera motion trajectory.

17. The method of any one of claims 1 to 7, wherein, The first video is a video generated by a video generation model. The method further includes: obtaining a reference motion trajectory of the moving object corresponding to a reference video; Based on the predicted motion trajectory and the reference motion trajectory, a model evaluation result is determined, the model evaluation result being used to indicate motion object trajectory similarity of the first video and the reference video.

18. A model optimization method, characterized by, The method comprises: configuring an inter-frame attention layer in a prediction model; performing an inter-frame attention operation on at least two video frames in a video through the inter-frame attention layer to predict camera parameters corresponding to the at least two video frames respectively; wherein the inter-frame attention operation is used to fuse the at least two video frames based on a time sequence relationship between the at least two video frames.

19. The method of claim 18, wherein, The method further comprises: training the prediction model configured with the inter-frame attention layer to obtain a camera parameter prediction model; the training comprises a first training stage and a second training stage; wherein, in the first training stage, parameter updating is performed on part of network layers in the prediction model including the inter-frame attention layer; in the second training stage, parameter updating is performed on all network layers in the prediction model.

20. The method of claim 18, wherein, The method further comprises: building an evaluation system based on a camera parameter prediction model, the camera parameter prediction model being obtained by training the prediction model configured with the inter-frame attention layer; in the evaluation system, camera parameter prediction is performed on a first video and a reference video respectively through the camera parameter prediction model to obtain camera parameters corresponding to the reference video and the first video respectively; a reference motion trajectory is obtained based on the camera parameters corresponding to the reference video; a predicted motion trajectory is obtained based on the camera parameters corresponding to the first video; an evaluation result is determined based on the predicted motion trajectory and the reference motion trajectory, the evaluation result being used to indicate motion object trajectory similarity of the first video and the reference video.

21. A camera parameter prediction apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire a first video; a first prediction module configured to perform an inter-frame attention operation on at least two video frames in the first video to obtain camera parameters corresponding to the at least two video frames respectively; wherein the inter-frame attention operation is used to fuse the at least two video frames based on a time sequence relationship between the at least two video frames, and the camera parameters are used to determine a predicted motion trajectory of a motion object corresponding to the first video.

22. A model optimization apparatus characterized by comprising: The device comprises: a configuration module configured to configure an inter-frame attention layer in a prediction model; a second prediction module configured to perform an inter-frame attention operation on at least two video frames in a video through the inter-frame attention layer to predict camera parameters corresponding to the at least two video frames respectively; wherein the inter-frame attention operation is used to fuse the at least two video frames based on a time sequence relationship between the at least two video frames.

23. A computer device, comprising: The computer device comprises a processor and a memory, the memory storing at least one program, the at least one program being loaded and executed by the processor to implement the camera parameter prediction method according to any one of claims 1 to 17, or the model optimization method according to any one of claims 18 to 20.

24. A computer-readable storage medium, characterized in that, The storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the camera parameter prediction method according to any one of claims 1 to 17, or the model optimization method according to any one of claims 18 to 20.

25. A computer program product, characterised in that, The computer program is executed by the processor to implement the camera parameter prediction method according to any one of claims 1 to 17, or the model optimization method according to any one of claims 18 to 20.

Images