A video data processing method and system based on digital twin scenario

Abstract

The present application provides a video data processing method and system based on a digital twin scene. Among them, spatiotemporal feature extraction is performed on a video frame sequence to determine a first spatiotemporal feature matrix; multiple video elements are segmented from the video frame sequence, and an element segmentation mask corresponding to each video element is generated; a second spatiotemporal feature matrix is generated; based on the second spatiotemporal feature matrix, the 2D motion trajectory corresponding to the dynamic element is obtained; the 2D motion trajectory of the dynamic element is mapped into 3D space to create or update a target virtual 3D scene model, and a predicted video frame sequence is generated through the target virtual 3D scene model; according to the comparison result of the predicted video frame sequence and the video frame sequence decomposed by the video data, the model parameters in the target virtual 3D scene model are adjusted. The technical solution provided by the present application can effectively improve the efficiency of video data processing.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of digital twin technology, and in particular to a video data processing method and system based on a digital twin scenario. Background Art

[0002] In the digital age, the "digital twin," as an emerging technology concept, is profoundly changing data processing and decision-making across industries, particularly in the field of video data analysis. A digital twin integrates digital representations of physical entities or systems with sensor data, historical records, real-time monitoring, predictive models, and advanced analytics to create a virtual replica that is synchronized with the physical entity. In the field of video data processing, this concept focuses on building highly accurate virtual environment models to enable deep insights into video content, predictive analysis, and optimized decision support.

[0003] Currently, video data processing solutions usually use AI analysis technology to automatically identify objects and activities in the picture, and create a simplified "dynamic storyboard" of time and space to help quickly understand video information.

[0004] However, the above solutions still have difficulty in accurately identifying and predicting in complex scenes, especially in scenes with a combination of static information (such as light) and the motion state of objects, making it difficult to effectively process video data. Summary of the Invention

[0005] The embodiments of the present application provide a video data processing method and system based on a digital twin scenario to solve the problem of poor video data processing efficiency in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a video data processing method based on a digital twin scenario, characterized by comprising:

[0007] Collecting video data from an actual environment, decomposing the video data into a video frame sequence, and performing spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes an object motion path, a subject behavior pattern, and scene information corresponding to the video frame, the object motion path, the subject behavior pattern, and the scene information corresponding to the video frame being determined by the spatiotemporal feature extraction;

[0008] Segmenting a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix, and generating an element segmentation mask corresponding to each of the video elements;

[0009] Fusing the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step, and each column contains the segmentation mask features and motion features of all objects in the time step;

[0010] Based on the second spatiotemporal feature matrix, locating and tracking dynamic elements in the video frame sequence in real time, and obtaining 2D motion trajectories corresponding to the dynamic elements, wherein the dynamic elements are video elements with dynamic features in the video elements;

[0011] Mapping the 2D motion trajectory of the dynamic element into a 3D space to create or update a target virtual 3D scene model, and inputting the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model;

[0012] The predicted video frame sequence is compared frame by frame with the video frame sequence decomposed from the video data, and based on the comparison result, the model parameters in the target virtual 3D scene model are adjusted, wherein the model parameters include at least camera parameters or rendering parameters.

[0013] Optionally, extracting spatiotemporal features from the video frame sequence to determine a first spatiotemporal feature matrix includes:

[0014] Identify all objects in the video frame sequence and continuously track the position of each object in different frames to obtain the object motion path corresponding to the video frame;

[0015] identifying action information of all subjects in the video frame sequence to determine a subject behavior pattern of each subject;

[0016] Identifying scene change information that changes over time in the video frame sequence and static information in the video frame sequence, and forming scene information based on the scene change information and the static information;

[0017] The static information includes at least: scene structure, object layout and lighting conditions;

[0018] The object motion paths, subject behavior patterns, and scene information corresponding to the plurality of video frames are converted into a preset numerical form to form a first spatiotemporal feature matrix.

[0019] Optionally, segmenting a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix and generating an element segmentation mask corresponding to each video element includes:

[0020] Inputting the first spatiotemporal feature matrix into a pre-established semantic segmentation model to predict each video frame in the first spatiotemporal feature matrix through the semantic segmentation model and outputting a segmentation result, wherein the segmentation result includes a plurality of video elements and an element label corresponding to each of the video elements, the element label being used to indicate whether the video element belongs to the foreground, background, or a specific object category in the video frame, the semantic segmentation model being trained based on a plurality of video frame samples with element labels;

[0021] Generate an element segmentation mask corresponding to each video element, wherein the element segmentation mask exists in the form of a binary image, and the binary image form includes black and white colors, wherein white represents the area where the video element is located, and black represents the background area outside the video element.

[0022] Optionally, fusing the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix includes:

[0023] Extracting object motion paths corresponding to a plurality of the video frames from the first spatiotemporal feature matrix, wherein the object motion path corresponding to each video frame is consistent with a time step of the corresponding element segmentation mask;

[0024] Converting each of the element segmentation masks into segmentation mask features, and converting the object motion path corresponding to each of the video frames into motion features;

[0025] For each of the video frames, concatenating the segmentation mask features of the corresponding object with the motion features to form a comprehensive feature vector;

[0026] According to the time step and the number of objects, the comprehensive feature vectors corresponding to each of the video frames are arranged in sequence to generate a second spatiotemporal feature matrix.

[0027] Optionally, the real-time positioning and tracking of dynamic elements in the video frame sequence based on the second spatiotemporal feature matrix, and obtaining 2D motion trajectories corresponding to the dynamic elements, wherein the dynamic elements are video elements with dynamic features in the video elements, include:

[0028] Based on the second spatiotemporal feature matrix, identifying an area in the video frame sequence that has a significant motion difference from a background area outside the video element, and determining the area as an area where a dynamic element is located;

[0029] In the area where the dynamic element is located, the category and bounding box of the dynamic element are identified, and for each dynamic element, the motion trajectory of the dynamic element is established between the video frame sequence based on the category and bounding box of the dynamic element to generate a 2D motion trajectory corresponding to the dynamic element.

[0030] Optionally, mapping the 2D motion trajectory of the dynamic element into a 3D space to create or update a target virtual 3D scene model, and inputting the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model includes:

[0031] Extracting 2D feature points on the 2D motion trajectory corresponding to the dynamic element, and matching the 2D feature points between different video frames to determine matching 2D feature point pairs;

[0032] Using the matched 2D feature point pairs, calculating corresponding points of the dynamic element in 3D space to generate a 3D motion trajectory of the dynamic element;

[0033] constructing an initial virtual 3D scene model based on the subject behavior pattern and the scene information in the first spatiotemporal feature matrix, and integrating the 3D motion trajectory of the dynamic element into the initial virtual 3D scene model to generate a target virtual 3D scene model, wherein the dynamic element can move at a correct temporal and spatial position in the virtual 3D scene model;

[0034] According to the target virtual 3D scene model and the acquired position information of the dynamic elements in each of the video frames, new video frames are rendered under preset camera parameters to form a predicted video frame sequence.

[0035] Optionally, comparing the predicted video frame sequence with the video frame sequence decomposed from the video data frame by frame, and adjusting model parameters in the target virtual 3D scene model based on the comparison result, includes:

[0036] A difference loss function is calculated between the predicted video frame sequence and the video frame sequence decomposed from the video data, wherein the difference loss function includes a model parameter set, and the model parameter set includes at least: camera parameters or rendering parameters.

[0037] Optionally, calculating a difference loss function between the predicted video frame sequence and the video frame sequence decomposed from the video data includes:

[0038] By formula group:

[0039] , calculating a difference loss function between the predicted video frame sequence and the video frame sequence decomposed by the video data;

[0040] in, Expressed as a pixel-level loss function, Represented as a timestamp The predicted video frame corresponding to the frame is represents a sequence of video frames decomposed into the video data, It is expressed as the structural similarity index, which is used to measure the structural similarity between the predicted video frame sequence and the video frame sequence decomposed by the video data. The value range is between -1 and 1, where 1 means exactly the same and 0 means no correlation. It is calculated using the SSIM algorithm; Expressed as a hyperparameter, it is used to balance the weight of SSIM loss and L1 norm loss; Expressed as L1 norm loss, it is the sum of absolute errors between the predicted video frame sequence and the video frame sequence decomposed by the video data;

[0041] in, Expressed as geometric consistency loss function, Represented as the number of sampling points in the real 3D scene model, Represented as the number of sampling points in the target virtual 3D scene model, Represented as a real 3D scene model, Represented as the target virtual 3D scene model, Represented as a set of computational points Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a calculation point set Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a hyperparameter to adjust the normal consistency loss Loss of geometric consistency The weight in Expressed as normal consistency loss, it is used to measure and the differences between;

[0042] in, Expressed as a temporal consistency loss function, Represented as a timestamp The predicted video frame corresponding to the frame is Represented as a timestamp The predicted video frame corresponding to the frame is Expressed as optical flow error, it is used to measure the consistency of pixel motion between two predicted video frames. Expressed as the penalty coefficient of optical flow error, Expressed as optical smoothness loss, it is used to control the smooth transition of video frames in timestamps. Expressed as a hyperparameter, it is used to balance the influence of the optical smoothness loss and the optical flow error on the temporal consistency loss;

[0043] in, Expressed as the difference loss function, Represented as a set of model parameters, Expressed as the weight of the pixel-level loss function, Expressed as the weight of the geometric consistency loss function, is represented as the weight of the temporal consistency loss function.

[0044] In a second aspect, an embodiment of the present application provides a video data processing system based on a digital twin scenario, including:

[0045] an acquisition and processing module, configured to acquire video data from an actual environment, decompose the video data into a video frame sequence, and perform spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes an object motion path, a subject behavior pattern, and scene information corresponding to the video frame, the object motion path, the subject behavior pattern, and scene information corresponding to the video frame being determined through the spatiotemporal feature extraction;

[0046] a segmentation module, configured to segment a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix, and generate an element segmentation mask corresponding to each of the video elements;

[0047] a fusion module, configured to fuse the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step, and each column contains the segmentation mask features and motion features of all objects in the time step;

[0048] a positioning module, configured to locate and track dynamic elements in the video frame sequence in real time based on the second spatiotemporal feature matrix, and obtain 2D motion trajectories corresponding to the dynamic elements, wherein the dynamic elements are video elements with dynamic features in the video elements;

[0049] a generation module, configured to map the 2D motion trajectory of the dynamic element into a 3D space to create or update a target virtual 3D scene model, and input the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model;

[0050] An adjustment module is used to compare the predicted video frame sequence with the video frame sequence decomposed by the video data frame by frame, and based on the comparison result, adjust the model parameters in the target virtual 3D scene model, wherein the model parameters include at least object motion parameters, camera parameters or rendering parameters.

[0051] In a third aspect, an embodiment of the present application provides a computing device comprising a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a video data processing method based on a digital twin scenario as described in the first aspect above.

[0052] In an embodiment of the present application, video data is collected from an actual environment, the video data is decomposed into a video frame sequence, and spatiotemporal feature extraction is performed on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes an object motion path, a subject behavior pattern, and scene information corresponding to the video frame, and the object motion path, the subject behavior pattern, and scene information corresponding to the video frame are determined by spatiotemporal feature extraction; based on the first spatiotemporal feature matrix, a plurality of video elements are segmented from the video frame sequence, and an element segmentation mask corresponding to each of the video elements is generated; the element segmentation mask is fused with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step. , each column contains the segmentation mask features and motion features of all objects in the time step; based on the second spatiotemporal feature matrix, the dynamic elements in the video frame sequence are located and tracked in real time, and the 2D motion trajectory corresponding to the dynamic elements is obtained, and the dynamic elements are video elements with dynamic features in the video elements; the 2D motion trajectory of the dynamic elements is mapped into 3D space to create or update the target virtual 3D scene model, and the first spatiotemporal feature matrix and the dynamic elements are input into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model; the predicted video frame sequence is compared frame by frame with the video frame sequence decomposed by the video data, and based on the comparison result, the model parameters in the target virtual 3D scene model are adjusted, and the model parameters include at least object motion parameters, camera parameters or rendering parameters.

[0053] The beneficial effects of this method are mainly reflected in the following aspects:

[0054] Improve analysis accuracy and efficiency: By performing precise spatiotemporal feature extraction on video data, we can not only capture static information in the video, but also gain an in-depth understanding of dynamic changes, such as object movement, behavior patterns, and scene changes, thereby greatly improving the accuracy and processing speed of data analysis.

[0055] Enhanced dynamic tracking and prediction capabilities: Real-time positioning and tracking of dynamic elements, generation of 2D motion trajectories and mapping to 3D space help build a more realistic and dynamically responsive virtual environment. This is particularly effective for predicting future states and simulating "what-if" scenarios, improving the foresight and practicality of decision support systems.

[0056] Optimizing digital twin applications: By continuously iteratively optimizing 3D scene models, we ensure the consistency and synchronization between virtual models and the physical world, promoting the widespread application of digital twin technology in various fields such as industrial maintenance, urban management, and security monitoring, and improving the efficiency of simulation, fault prediction, and resource scheduling.

[0057] In summary, this method not only promotes the advancement of video data analysis technology, but also shows significant positive effects in promoting smart city construction, optimizing production management, enhancing user experience and other aspects.

[0058] These and other aspects of the present application will become more readily apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 A flowchart of a video data processing method based on a digital twin scenario provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of the structure of a video data processing system based on a digital twin scenario provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0064] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to being different types.

[0065] The technical solution of this application can be applied to a variety of scenarios that require in-depth analysis and simulation of real-world dynamics, including but not limited to the following aspects:

[0066] Intelligent security monitoring: In a monitoring environment, it can automatically identify abnormal behavior, track the movement of specific people or vehicles, and improve security warnings and incident response speed.

[0067] Smart city management: For example, traffic flow analysis can optimize traffic light control strategies by analyzing vehicle and pedestrian flow patterns, reducing congestion and improving urban traffic efficiency.

[0068] Industrial production and maintenance: In digital factories, equipment operating status is monitored, maintenance needs are predicted, and bottlenecks in the production process are analyzed to achieve a dual improvement in production efficiency and safety.

[0069] Sports event analysis: Capture athletes' movements in real time, analyze game strategies, evaluate athlete performance, provide instant feedback to coaching teams, and enhance the audience experience.

[0070] Film and television special effects and animation production: By capturing the dynamic characteristics of the real world, highly realistic virtual scenes and character animations can be quickly generated, reducing production costs and improving creative efficiency.

[0071] Disaster emergency response: At the scene of a natural disaster or emergency, quickly identify the disaster situation and personnel distribution, assist in formulating rescue plans, and improve the accuracy and timeliness of rescue operations.

[0072] Retail and consumer behavior research: Analyze customer movement paths and dwelling areas in commercial spaces to optimize merchandise layout and promotional strategies, thereby enhancing customer experience and sales performance.

[0073] The common feature of these scenarios is that they all require in-depth analysis and processing of video data to extract key information and make predictions or simulations. The above method provides a full-process solution from extracting spatiotemporal features from video data, real-time tracking of dynamic elements to building and optimizing digital twin models.

[0074] The inventors found that current video data processing solutions usually use AI analysis technology to automatically identify objects and activities in the picture, and create a simplified "dynamic storyboard" of time and space to help quickly understand video information.

[0075] However, the above solutions still have difficulty in accurately identifying and predicting in complex scenes, especially in scenarios where static information (such as light) and the motion state of objects are combined. It is difficult to effectively process video data, such as extracting and processing dynamic elements.

[0076] In view of this, the present application provides a video data processing method based on a digital twin scenario, which includes: collecting video data from an actual environment, decomposing the video data into a video frame sequence, and performing spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes the object motion path, subject behavior pattern and scene information corresponding to the video frame, and the object motion path, subject behavior pattern and scene information corresponding to the video frame are determined by spatiotemporal feature extraction; based on the first spatiotemporal feature matrix, multiple video elements are segmented from the video frame sequence, and an element segmentation mask corresponding to each of the video elements is generated; the element segmentation mask is merged with the object motion paths corresponding to the multiple video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, and the second spatiotemporal feature matrix is generated. Each row of the feature matrix represents a time step, and each column contains the segmentation mask features and motion features of all objects within the time step; based on the second spatiotemporal feature matrix, the dynamic elements in the video frame sequence are located and tracked in real time, and the 2D motion trajectory corresponding to the dynamic elements is obtained, and the dynamic elements are video elements with dynamic features in the video elements; the 2D motion trajectory of the dynamic elements is mapped into 3D space to create or update a target virtual 3D scene model, and the first spatiotemporal feature matrix and the dynamic elements are input into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model; the predicted video frame sequence is compared frame by frame with the video frame sequence decomposed by the video data, and based on the comparison result, the model parameters in the target virtual 3D scene model are adjusted, and the model parameters include at least object motion parameters, camera parameters or rendering parameters.

[0077] By extracting precise spatiotemporal features from video data, this method not only captures static information but also provides a deep understanding of dynamic changes, such as object movement, behavioral patterns, and scene changes, significantly improving the accuracy and processing speed of data analysis. Furthermore, by locating and tracking dynamic elements in real time, generating 2D motion trajectories and mapping them into 3D space, this method helps build a more realistic and dynamically responsive virtual environment. This is particularly effective for predicting future states and simulating "what-if" scenarios, enhancing the foresight and practicality of decision support systems.

[0078] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0079] Figure 1 A flowchart of a video data processing method based on a digital twin scenario provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0080] 101. Collect video data from an actual environment, decompose the video data into a video frame sequence, and perform spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix;

[0081] In this step, each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes the object motion path, subject behavior pattern and scene information corresponding to the video frame. The object motion path, subject behavior pattern and scene information corresponding to the video frame are determined through spatiotemporal feature extraction.

[0082] Specifically, a continuous video data stream is first collected from the actual environment using a device such as a high-definition camera. Subsequently, video processing software or algorithms are used to decompose the continuous video stream into a sequence of images, i.e., a video frame sequence.

[0083] For spatiotemporal feature extraction, this process involves complex computer vision technology and deep learning models, aiming to extract meaningful information from each frame of the image. The object's motion path is obtained through optical flow estimation or tracking algorithms, analyzing the changes in pixel points over time to determine the object's direction and speed. Subject behavior pattern recognition uses an action recognition model to classify or predict behavior types by analyzing the posture and action sequence of the human body (or other subjects). Scene information extraction involves scene understanding, including identifying objects in the background, lighting conditions, and overall layout, which is usually accomplished through scene parsing models. These features together constitute the spatiotemporal feature matrix, where each row corresponds to the feature profile of a frame, and each column is refined to a specific feature dimension, such as motion, behavior, and scene details.

[0084] In the embodiment of the present application, it is assumed that a video processing module is being developed for an intelligent transportation system. Starting from a high-definition video stream captured by a surveillance camera installed at a city intersection, the video data is decomposed into a video frame sequence of 30 frames per second. A combined model of a convolutional neural network (CNN) and a recurrent neural network (RNN) is used to extract spatiotemporal features. CNN is responsible for extracting visual features such as object contours and textures from each frame, while RNN tracks the changes in these features over time to capture the motion path. For subject behavior pattern recognition, especially for pedestrians and vehicles, a pre-trained behavior recognition model can be used, such as an I3D network, which can predict the vehicle's driving direction, acceleration or braking behavior, and pedestrian crossing patterns based on video clips. Scene information is obtained through a scene classification model, such as using a ResNet architecture to classify the background of each video frame and distinguish elements such as streets, sidewalks, and buildings. In this way, after each frame of video data is processed, it is integrated into a spatiotemporal feature matrix. Each row records comprehensive information at a specific moment, while the columns are subdivided into specific descriptions of object movement (such as the right turn path of vehicle ID_001), behavior (pedestrian_002 waiting to cross the street), and scene (night, rainy days), providing a rich data foundation for subsequent analysis and decision-making.

[0085] 102. Segment a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix, and generate an element segmentation mask corresponding to each of the video elements;

[0086] In this step, the input video frame sequence is first analyzed using a deep learning model, particularly a convolutional neural network (CNN) or a recurrent neural network (RNN) based on time series analysis. These models can learn and extract spatiotemporal features from the video, forming a high-dimensional feature matrix, known as the first spatiotemporal feature matrix. This process involves multiple layers of convolution and pooling operations (for CNNs) or recursive processing of temporal information (for RNNs), aiming to capture both static visual features and dynamic changes in the video. These features are then used to identify and locate different video elements, such as objects, people, and actions. The generation of feature segmentation masks is accomplished through relatively complex post-processing algorithms, such as conditional random fields (CRFs) or graph-cut-based methods, which precisely define the boundaries of each video element based on the response strength in the feature matrix, thereby effectively segmenting the video content.

[0087] In this embodiment, a pre-trained 3D CNN model is used to process video frame sequences. This model can simultaneously consider information in both spatial and temporal dimensions. First, the video frames are resized and normalized, then fed into the 3D CNN for forward propagation, resulting in a first spatiotemporal feature matrix rich in spatiotemporal information.

[0088] Next, a Mask R-CNN model is applied to predict segmentation masks for each video element. This model not only outputs the location of the object in each video frame but also provides pixel-level segmentation masks. For example, in analyzing a sports video, the system can accurately segment different elements such as players, balls, and fields from a complex scene, and generate detailed segmentation masks for each type of video element, making subsequent video editing, analysis, or content understanding tasks more efficient and accurate.

[0089] 103. Fusing the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step, and each column contains the segmentation mask features and motion features of all objects in the time step;

[0090] In this step, the object's motion trajectory must first be parsed from the first spatiotemporal feature matrix. This typically involves analyzing the object's position changes between consecutive frames using a tracking algorithm (such as optical flow, Kalman filtering, or a deep learning tracking network) to determine the object's motion path. The object's motion features can include position changes, velocity, acceleration, and so on. These motion features are then combined with the previously generated segmentation masks for each video element. This combination can be achieved by directly concatenating the segmentation mask features with the motion feature vector, or by devising more complex fusion mechanisms, such as using an attention mechanism to emphasize features in key motion regions. This process results in a new feature representation—the second spatiotemporal feature matrix. This not only preserves the object's static appearance information at each time step but also incorporates a description of its dynamic behavior, providing a more comprehensive feature foundation for subsequent advanced analysis or recognition tasks.

[0091] In an embodiment of the present application, the DeepSORT algorithm is used to track objects in the video and extract their motion paths. DeepSORT combines appearance feature matching and motion model prediction to ensure the stability and accuracy of tracking. For objects in each time step, we extract their segmentation masks in the first spatiotemporal feature matrix and record their motion parameters. Then, a feature vector is constructed for each object, where the first half is the feature encoding of the segmentation mask and the second half is the motion feature of the object. These vectors are arranged in chronological order to form a second spatiotemporal feature matrix. For example, in a surveillance video analysis scenario, the system not only identifies objects such as pedestrians and vehicles, but also records their movement trajectories and speed changes. All of this information is integrated in the second spatiotemporal feature matrix, providing detailed data support for further analysis of pedestrian behavior patterns or vehicle flow management.

[0092] 104. Based on the second spatiotemporal feature matrix, locate and track dynamic elements in the video frame sequence in real time, and obtain 2D motion trajectories corresponding to the dynamic elements, where the dynamic elements are video elements with dynamic features in the video elements;

[0093] In this step, the goal is to identify and track dynamic elements in the video in real time from the constructed second spatiotemporal feature matrix, thereby obtaining the two-dimensional motion trajectories of these elements. This step primarily involves feature decoding, motion detection, and trajectory generation. First, the object segmentation masks and motion features encoded in the matrix are decoded using a machine learning or deep learning model to identify the presence and category of dynamic elements in each frame. Next, a tracker based on correlation filtering, a deep learning-based tracking network (such as Siamese Tracker and DeepSORT), or an optical flow estimation method is applied to calculate and update the object's motion trajectory in two dimensions based on changes in appearance and position between consecutive frames. In this way, the motion path of each dynamic element is gradually constructed, providing intuitive motion information for subsequent analysis such as behavior recognition and interaction detection.

[0094] In the embodiment of the present application, an improved deep learning tracking framework, namely the Tracking Transformer (TranSTR) model based on Transformer, is used to implement this process. The TranSTR model can effectively integrate time and space information, thereby improving the robustness of tracking in complex scenes. In the specific operation, the second spatiotemporal feature matrix is first parsed, the feature map of each time step is extracted, and the feature map is input into the TranSTR model. The model uses the self-attention mechanism to associate object features across time and space dimensions, so as to accurately locate and predict the position of each dynamic element in the next frame. For example, in the video analysis scenario of sports events, the system can track the movement of athletes and balls in real time, and accurately capture their 2D motion trajectories even in the case of rapid movement and frequent occlusion, which is of great value for analyzing game strategies, athlete performance, etc. By continuously iterating this process, the system continuously updates the motion trajectory of each dynamic element until the end of the video sequence.

[0095] 105. Map the 2D motion trajectory of the dynamic element into 3D space to create or update a target virtual 3D scene model, and input the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence using the target virtual 3D scene model;

[0096] In this step, the motion trajectories of dynamic elements extracted from the 2D video are converted into 3D space through a technique called "2D-to-3D lifting." This technique typically involves using depth estimation, multi-view geometry, or pre-established 3D models to infer the three-dimensional position and pose of objects in the video. Once the 2D trajectory is obtained, it is combined with the scene's depth information or mapped into 3D space using kinematic principles to form the object's 3D motion trajectory. These 3D trajectories are then used to create or update a virtual 3D scene model that includes the static background and dynamic representations of the dynamic elements in the video. The first spatiotemporal feature matrix and dynamic element information serve as input, helping the model understand the spatiotemporal context of the video and thus more accurately predict the next sequence of video frames. This step is crucial for generating realistic video predictions, augmented reality content, or video content editing.

[0097] In this embodiment, a method combining deep learning and geometric constraints is employed to achieve 2D-to-3D mapping. First, a monocular depth estimation network (such as Monodepth2) is used to estimate the depth value of each pixel in a video frame, providing depth cues for the 2D motion trajectory. Then, through algorithms such as Bundle Adjustment, the motion trajectory and depth information are combined to optimize the object's 3D trajectory. To construct the target virtual 3D scene model, we chose the Unity 3D engine because of its powerful physics simulation and rendering capabilities. The scene structure information obtained by decoding the first spatiotemporal feature matrix and the 3D position and posture of dynamic elements are imported into Unity to create the corresponding 3D objects and animations. Furthermore, using Unity's built-in time series prediction model or a custom recurrent neural network (RNN), the predicted video frame sequence is generated based on the current 3D scene state and the motion patterns of dynamic elements. For example, in a city streetscape reconstruction project, not only are static elements such as streets and buildings meticulously modeled, but dynamic behaviors such as pedestrian movement and vehicle traffic are also accurately reproduced in the virtual 3D scene through this process, enabling prediction of potential streetscape changes in the next few frames.

[0098] 106. Compare the predicted video frame sequence with the video frame sequence decomposed from the video data frame by frame, and adjust model parameters in the target virtual 3D scene model based on the comparison result, where the model parameters include at least object motion parameters, camera parameters, or rendering parameters.

[0099] In this step, it is first necessary to extract the corresponding video frame sequence from the predicted video frame sequence and the original video data. The predicted video frame sequence is usually rendered by simulating parameters such as lighting, material, and camera motion from the established target virtual 3D scene model. The video frame sequence decomposed by the video data is each frame image extracted directly from the actual recorded or captured video. In order to perform frame-by-frame comparison, image similarity measurement methods such as structural similarity index (SSIM), peak signal-to-noise ratio (PSNR) or deep learning-driven image difference measurement models can be used to evaluate the difference between the predicted frame and the actual frame.

[0100] After acquiring these frame sequences, they are preprocessed using image processing techniques, such as alignment, scaling, and color space conversion, to ensure consistency and accuracy in comparison. Then, through pixel-level or feature-level comparisons, the predicted frames are analyzed for deviations in content, color, texture, and other aspects between the actual frames. Based on these comparison results, it is possible to identify aspects of the target virtual 3D scene model that fail to accurately reflect the real scene, such as inaccurate object positions and motion trajectories, or improper camera viewing angles and exposure settings.

[0101] In this example, we are developing a deep learning-based 3D reconstruction system to recreate the virtual scene of an ancient building. First, we use a neural network to predict a video sequence of the building, including the changing light and shadows at different times of day and the movements of visitors. Next, we obtain a real video recording of the location during the same time period and decompose it into a sequence of single-frame images.

[0102] During implementation, we selected SSIM as the image quality metric and combined it with optical flow estimation to analyze the coherence of object motion. Comparison revealed significant deviations in lighting and shadow effects between some frames in the predicted sequence and the actual image, particularly the underrepresentation of the golden glow at dusk. Furthermore, the virtual tourists in the video had a slight misalignment in their movement paths compared to the actual image.

[0103] Based on these analysis results, the parameters of the target virtual 3D scene model were adjusted accordingly. For example, the color temperature of the light source was adjusted to make the dusk lighting softer and more warm. Furthermore, the behavioral logic and path planning of the crowd simulation algorithm were optimized to better match actual crowd flow patterns. Finally, the predicted sequence was re-rendered and compared again until a satisfactory visual fit was achieved. This process embodies the practice of iteratively optimizing model parameters to improve the realism of video synthesis.

[0104] Optionally, in this embodiment of the present application, a detailed analysis of the video frame sequence is first performed using deep learning and computer vision techniques, particularly convolutional neural networks (CNNs) and recurrent neural networks (RNNs). This covers key aspects of object recognition, motion path tracking, behavioral patterns, and scene understanding. Therefore, the process of "extracting spatiotemporal features from the video frame sequence to determine a first spatiotemporal feature matrix" in step 101 may include:

[0105] 1011. Identify all objects in the video frame sequence, and continuously track the position of each object in different frames to obtain a motion path of the object corresponding to the video frame;

[0106] In this step, an object detection model (such as YOLOD or FasterNet) is used to identify all objects in the video. Optical flow estimation or sequential frame matching techniques (such as the Kalman-Box filter) are then used to track the movement of each object between video frames to map its motion path. This requires the model to have real-time recognition and prediction capabilities to ensure the stability of object identity between consecutive frames.

[0107] 1012. Identify action information of all subjects in the video frame sequence to determine a subject behavior pattern of each subject;

[0108] In this step, behavior recognition algorithms (such as those based on 3D-CNN or LSTM) analyze the dynamics of individuals or groups in the video sequence to identify and classify their behaviors (walking, running, sitting, waving, etc.). This requires understanding the action sequence in the time series, extracting the characteristics of the behavior, and combining it with the temporal context for pattern recognition.

[0109] 1013. Identify scene change information that changes over time in the video frame sequence and static information in the video frame sequence, and compose scene information based on the scene change information and the static information;

[0110] In this step, the static information includes at least scene structure, object layout, and lighting conditions. Scene understanding uses scene segmentation (such as Mask R-CNN) to identify static structures (buildings, roads, furniture layout) and lighting conditions (shadows, color temperature). Simultaneously, scene changes are analyzed to capture dynamic elements over time (such as weather changes and the appearance and disappearance of people), combining them with static elements to form a comprehensive scene depiction.

[0111] 1014. Convert the object motion paths, subject behavior patterns, and scene information corresponding to the plurality of video frames into a preset numerical form to form a first spatiotemporal feature matrix.

[0112] In this step, the above information is encoded as numerical values. For example, object motion paths can be represented as velocity and acceleration vectors, behavior patterns as category labels or behavior probabilities, scene changes and lighting intensity, etc., and organized into a matrix. Each row corresponds to a video frame, and the columns represent different feature dimensions (object motion, behavior, scene, etc.).

[0113] In the embodiment of the present application, it is assumed that this process is applied in an intelligent monitoring system. In the monitoring video stream, YOLODv3 is first used to identify pedestrians, vehicles and other objects, and the pedestrian trajectory is tracked by the DeepSORT algorithm, and its position changes in different frames are recorded. At the same time, 3D-CNN is used to identify pedestrian behavior patterns such as "walking", "running", and "standing", and scene segmentation is used to identify the "road", "trees", "buildings" and "daytime" lighting conditions in the monitoring area. Finally, this information is encoded into a feature matrix, with each frame corresponding to a row. The features include the trajectory change speed of object A, the behavior label "walking", the scene label "road" and the light intensity. This matrix provides a basis for subsequent analysis and prediction, such as abnormal behavior detection and crowd flow pattern analysis.

[0114] Optionally, in the embodiment of the present application, the process of “segmenting a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix and generating an element segmentation mask corresponding to each of the video elements” in step 102 may include:

[0115] 1021. Input the first spatiotemporal feature matrix into a pre-established semantic segmentation model, so that the semantic segmentation model predicts each video frame in the first spatiotemporal feature matrix and outputs a segmentation result, wherein the segmentation result includes multiple video elements and an element label corresponding to each video element, wherein the element label is used to indicate whether the video element belongs to the foreground, background, or a specific object category in the video frame, and the semantic segmentation model is trained based on multiple video frame samples with element labels.

[0116] In this step, first, the composition of the video frame is analyzed using the semantic segmentation technology in deep learning by taking the first spatiotemporal feature matrix as input. The semantic segmentation model is a deep neural network architecture, such as U-Net, DeepLabV3+ or Mask R-CNN, which is designed for pixel-level classification, that is, assigning a category label to each pixel in the image. During the training process, this type of model will learn how to distinguish between foreground and background, and identify specific object categories in the video frame, such as people, vehicles, buildings, etc. The training dataset contains a large number of labeled video frame samples, and each pixel on each sample image has a corresponding feature label to guide the model to learn the features of each category. In the inference stage, the model predicts the semantic attribution of each part in each video frame based on the rich contextual information provided by the first spatiotemporal feature matrix, and outputs high-precision segmentation results.

[0117] 1022. Generate an element segmentation mask corresponding to each of the video elements, wherein the element segmentation mask exists in the form of a binary image, and the binary image form includes black and white colors, wherein white represents the area where the video element is located, and black represents the background area outside the video element.

[0118] In this step, when generating feature segmentation masks, a binary mask is created for each predicted video feature based on the output of the semantic segmentation model. This process essentially binarizes the segmentation results. Pixels predicted by the model to be video features are assigned a white color (a value of 1), indicating that this portion of the image belongs to a specific video feature. Pixels predicted to be background or other non-feature areas are assigned a black color (a value of 0). This way, each video feature is assigned a distinct binary image mask, which visually displays the feature's spatial distribution within the original video frame, facilitating subsequent video editing, analysis, or content enhancement.

[0119] In an embodiment of the present application, a sports event broadcasting application scenario is considered. From a video of a football game, we have obtained a first spatiotemporal feature matrix containing multiple spatiotemporal features such as players, balls, and field lines. Next, the DeepLabV3+ semantic segmentation model that has been pre-trained on a large number of sports video samples is used to input the feature matrix for processing. The model performs fine segmentation on each frame, distinguishes different elements such as players (with uniforms in different team colors), footballs, spectator seats, grass, etc., and assigns the correct label to each element. Subsequently, based on the output of the model, a series of binary images are automatically generated as feature segmentation masks. For example, a mask specifically highlights the positions of all players, the white area represents the players, and the black area is other backgrounds. These masks can not only help realize real-time game data analysis, such as player running trajectories and ball possession area statistics, but also facilitate advanced editing tasks such as adding special effects or advertising implants during post-production.

[0120] Optionally, in the embodiment of the present application, the process of “fusing the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix” in step 103 may include:

[0121] 1031. Extracting object motion paths corresponding to a plurality of the video frames from the first spatiotemporal feature matrix, wherein the object motion path corresponding to each video frame is consistent with a time step of the corresponding element segmentation mask;

[0122] In this step, we first analyze the first spatiotemporal feature matrix to extract the time-varying motion paths of each object in the video frame sequence. These motion paths reflect the dynamic position changes of the objects in the scene and ensure their temporal consistency with the feature segmentation mask. That is, the object motion information at each time step corresponds to the mask at that moment.

[0123] 1032. Convert each of the element segmentation masks into a segmentation mask feature, and convert the object motion path corresponding to each of the video frames into a motion feature;

[0124] In this step, the mask information is converted into a numerical feature representation. This typically involves encoding the mask's structural information into a one-dimensional or higher-dimensional vector, reflecting the spatial distribution of the object. Meanwhile, the object's motion features are extracted, including its velocity, acceleration, or more complex trajectory characteristics, to describe its dynamic behavior.

[0125] 1033. For each of the video frames, concatenate the segmentation mask features and the motion features of the corresponding object to form a comprehensive feature vector;

[0126] In this step, a comprehensive feature vector is constructed for each object in each video frame. This step integrates the static position (through segmentation mask features) and dynamic behavior (through motion features) of the object to form a comprehensive description of the object in space and time.

[0127] 1034. Arrange the comprehensive feature vectors corresponding to each of the video frames in sequence according to the time step and the number of objects to generate a second spatiotemporal feature matrix.

[0128] In this step, the comprehensive feature vectors of all video frames are arranged in chronological order and object order to construct a new spatiotemporal feature representation—the second spatiotemporal feature matrix. This matrix not only contains the current state of the object but also embeds its evolution over time, providing a richer information foundation for subsequent advanced analysis or processing. The dimensions of this second spatiotemporal feature matrix are: number of time steps x (number of objects x (segmentation mask feature dimension + motion feature dimension)).

[0129] In this embodiment, assume that a clip from an action movie is being processed, containing a protagonist and several supporting characters pursuing him. In step 1031, through previous analysis, we have obtained the precise motion trajectories of the protagonist and each supporting character in consecutive frames. These trajectories record the characters' movements within the scene as a sequence of coordinates. Simultaneously, in step 1032, for each frame, the element segmentation masks of the protagonist and each supporting character are converted into feature vectors representing their position and shape within the frame. Their motion paths are also converted into motion feature vectors describing their speed and direction within the scene. In step 1033, taking a particular frame as an example, the protagonist's segmentation mask features (assuming 256 dimensions) are concatenated with the motion features (assuming 6 dimensions) to form a 322-dimensional combined feature vector. A similar process is performed for each supporting character. Finally, in step 1034, the combined feature vectors of all characters in all frames of the entire movie are arranged in chronological order to form a large second spatiotemporal feature matrix with dimensions (total number of frames x (number of characters x (segmentation mask feature dimensions + motion feature dimensions))). Such a matrix can help further analyze the interaction patterns between characters, action strategy analysis, or automatic generation of visual effects.

[0130] Optionally, in the embodiment of the present application, the process of "locating and tracking the dynamic elements in the video frame sequence in real time based on the second spatiotemporal feature matrix, and obtaining the 2D motion trajectory corresponding to the dynamic elements" in step 104 may include:

[0131] 1041. Based on the second spatiotemporal feature matrix, identify a region in the video frame sequence that has a significant motion difference from a background region other than the video element, and determine the region as a location of a dynamic element.

[0132] In this step, the high-dimensional information of the second spatiotemporal feature matrix is first used to identify significantly different regions by comparing the changes in each pixel or region over time with those in the background. These regions typically correspond to dynamic elements in the video, such as moving people or vehicles. Techniques such as optical flow estimation, background subtraction, or motion segmentation can effectively distinguish dynamic elements from the static background.

[0133] 1042. In the area where the dynamic element is located, identify the category and bounding box of the dynamic element, and for each dynamic element, establish a motion trajectory of the dynamic element between the video frame sequence based on the category and bounding box of the dynamic element to generate a 2D motion trajectory corresponding to the dynamic element.

[0134] In this step, for each identified dynamic region, object detection algorithms, such as YOLO (You Only Look Once) and SSD (Single Shot MultiBox Detector), are applied to accurately locate the category of the dynamic feature (such as a person or vehicle) and its real-time bounding box position. By matching these bounding boxes across frames and considering the consistency of the object's appearance, position, and velocity, the motion trajectory of the dynamic feature between consecutive frames is constructed, resulting in a 2D motion trajectory. This process requires the support of efficient object tracking algorithms, such as Kalman filters and deep learning Siamese tracking networks, to ensure tracking accuracy and stability.

[0135] In an embodiment of the present application, a surveillance video analysis scenario is considered in which pedestrians in a specific area need to be monitored and tracked in real time. In step 104.1, by analyzing the second spatiotemporal feature matrix, the system quickly identifies the areas where pedestrians are moving in the picture in sharp contrast to the static background. These areas are regarded as dynamic element areas. In step 104.2, advanced deep learning target detection models, such as Faster R-CNN, are used to accurately classify and locate the bounding boxes of pedestrians in these areas. Once a pedestrian is detected for the first time, the system immediately starts the SORT (Simple Online and Realtime Tracking) algorithm based on deep learning, which combines appearance features and motion models to continuously predict and adjust the bounding box of the pedestrian, and can keep tracking even in the case of partial occlusion or lighting changes. In this way, the system generates a continuous 2D motion trajectory for each pedestrian, and these trajectory data can be used in applications such as further behavior analysis, crowd density estimation, or abnormal event detection.

[0136] Optionally, in the embodiment of the present application, the process of "mapping the 2D motion trajectory of the dynamic element into 3D space to create or update a target virtual 3D scene model, and inputting the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model" in step 105 may include:

[0137] 1051. Extract 2D feature points on the 2D motion trajectory corresponding to the dynamic element, and match the 2D feature points between different video frames to determine matching 2D feature point pairs.

[0138] In this step, key feature points, such as contour inflection points or points with significant texture changes, are carefully selected from the 2D motion trajectory of the dynamic element in step 1051. These feature points are then matched between adjacent video frames to establish cross-frame 2D feature associations. This process involves feature detection algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF), which effectively identify and match invariant feature points, even under varying viewpoints or scales.

[0139] 1052. Using the matched 2D feature point pairs, calculate corresponding points of the dynamic element in 3D space to generate a 3D motion trajectory of the dynamic element;

[0140] Next, in step 1052, these matched 2D feature point pairs are used to infer their true coordinates in 3D space through stereo vision or multi-view geometry principles, such as fundamental matrix or essential matrix calculations, to reconstruct the 3D motion trajectory of the dynamic element. This step is crucial for understanding the true motion path of the dynamic element and lays the foundation for subsequent 3D modeling.

[0141] 1053. Based on the subject behavior pattern and the scene information in the first spatiotemporal feature matrix, construct an initial virtual 3D scene model, and integrate the 3D motion trajectory of the dynamic element into the initial virtual 3D scene model to generate a target virtual 3D scene model, wherein the dynamic element can move at a correct time and space position in the virtual 3D scene model.

[0142] In step 1053, a preliminary virtual 3D scene model is constructed based on the subject behavior patterns extracted from the first spatiotemporal feature matrix and the static structure of the scene. This model not only includes the scene's geometric layout but also incorporates the expected patterns of subject activity, such as pedestrian paths or vehicle routes. The 3D motion trajectories of dynamic elements are then embedded into this initial model, ensuring that each dynamic element moves in the correct spatiotemporal location according to its actual behavior pattern, thereby completing the target virtual 3D scene model.

[0143] 1054. Render new video frames under preset camera parameters according to the target virtual 3D scene model and the acquired position information of the dynamic elements in each video frame to form a predicted video frame sequence.

[0144] Finally, in step 1054, new video frames are generated using real-time rendering technology (e.g., using a game engine like Unity or Unreal Engine) by utilizing the target virtual 3D scene model and the specific positions of each frame of the dynamic elements, combined with preset camera angles and lighting conditions. These new frames are then connected to form a predicted video sequence, which not only reflects the content of the original video but also predicts the possible future motion of the dynamic elements. This sequence is suitable for a variety of application scenarios, including video synthesis, predictive analysis, and virtual reality.

[0145] In this embodiment, let's assume we want to create a predictive video for a preview of an action movie. First, from existing action clips, we use the SIFT algorithm to extract 2D feature points for key movements, such as arm swings and kicks, from the protagonist's fighting moves. Matching pairs are then found through feature matching across consecutive frames. In step 1052, binocular vision technology is used to calculate the protagonist's motion trajectory in three-dimensional space based on these matching point pairs. In step 1053, based on the script's scene layout (e.g., an ancient battlefield setting) and the protagonist's behavioral patterns (attacks, evasions), a preliminary 3D scene model is constructed using the Unity game engine. The calculated 3D motion trajectory is then integrated into the model, allowing the protagonist to move naturally through the scene. Finally, in step 1054, while the virtual camera continues to follow the protagonist's movements, a series of new video frames are rendered according to the specified camera language (e.g., tracking shots, overhead shots). This creates a preview sequence of actions that were not actually filmed in the film, but are consistent in style. This helps directors and producers preview and adjust their shooting plans in advance.

[0146] Optionally, in the embodiment of the present application, the process of “comparing the predicted video frame sequence with the video frame sequence decomposed from the video data frame by frame, and adjusting the model parameters in the target virtual 3D scene model based on the comparison result” in step 106 may include:

[0147] A difference loss function is calculated between the predicted video frame sequence and the video frame sequence decomposed from the video data, wherein the difference loss function includes a model parameter set, and the model parameter set includes at least: camera parameters or rendering parameters.

[0148] The step of calculating a difference loss function between the predicted video frame sequence and the video frame sequence decomposed from the video data includes:

[0149] By formula group:

[0150] , calculating a difference loss function between the predicted video frame sequence and the video frame sequence decomposed by the video data;

[0151] in, Expressed as a pixel-level loss function, Represented as a timestamp The predicted video frame corresponding to the frame is represents a sequence of video frames decomposed into the video data, It is expressed as the structural similarity index, which is used to measure the structural similarity between the predicted video frame sequence and the video frame sequence decomposed by the video data. The value range is between -1 and 1, where 1 means exactly the same and 0 means no correlation. It is calculated using the SSIM algorithm; Expressed as a hyperparameter, it is used to balance the weight of SSIM loss and L1 norm loss; Expressed as L1 norm loss, it is the sum of absolute errors between the predicted video frame sequence and the video frame sequence decomposed by the video data;

[0152] in, Expressed as geometric consistency loss function, Represented as the number of sampling points in the real 3D scene model, Represented as the number of sampling points in the target virtual 3D scene model, Represented as a real 3D scene model, Represented as the target virtual 3D scene model, Represented as a calculation point set Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a calculation point set Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a hyperparameter to adjust the normal consistency loss Loss of geometric consistency The weight in Expressed as normal consistency loss, it is used to measure and the differences between;

[0153] in, Expressed as a temporal consistency loss function, Represented as a timestamp The predicted video frame corresponding to the frame is Represented as a timestamp The predicted video frame corresponding to the frame is Expressed as optical flow error, it is used to measure the consistency of pixel motion between two predicted video frames. Expressed as the penalty coefficient of optical flow error, Expressed as optical smoothness loss, it is used to control the smooth transition of video frames in timestamps. Expressed as a hyperparameter, it is used to balance the influence of the optical smoothness loss and the optical flow error on the temporal consistency loss;

[0154] in, Expressed as the difference loss function, Represented as a set of model parameters, Expressed as the weight of the pixel-level loss function, Expressed as the weight of the geometric consistency loss function, is represented as the weight of the temporal consistency loss function.

[0155] In this step, the calculation of the difference loss function comprehensively considers the matching degree of the predicted video frame sequence and the original video frame sequence in multiple dimensions, aiming to minimize this difference by optimizing the model parameters, thereby improving the quality of the predicted video. Specifically, The structural similarity index (SSIM) and L1 norm are used to measure the similarity between the predicted frame and the original frame at the pixel level, emphasizing the maintenance of the visual quality and structural consistency of the image content; Focusing on the geometric accuracy of the 3D scene model, the distance and normal consistency between point clouds are used to evaluate the closeness between the predicted model and the real scene; The loss term ensures the temporal smoothness of the video sequence, promoting coherent dynamic performance by analyzing the consistency of optical flow and optical smoothness between adjacent frames. The weights of these loss terms (such as α, β, and γ) need to be adjusted according to the specific application scenario to achieve the best reconstruction effect.

[0156] In the embodiment of the present application, it is assumed that a replay prediction system for a sports event is to be optimized, and the following example with numerical values is specifically implemented:

[0157] First, set the initial hyperparameters to: , and the weights of the pixel-level loss function, the geometric consistency loss function, and the temporal consistency loss function are respectively , in order to balance the impact of each loss term on the final model adjustment.

[0158] Next, for each predicted frame, we calculate For example, for a frame, if , L1 norm loss is 500, point set and The average sum of squared nearest neighbor distances between them is 100, and the normal consistency loss , optical flow error 20, optical smoothness loss If is 0.02, then the sub-loss functions (pixel-level loss function, geometric consistency loss function, and temporal consistency loss function) of the frame are calculated as follows:

[0159] =(1-0.9)+0.01 500=10+5=15;

[0160] =100+100+0.001 0.01=200.0001;

[0161] =0.1 20+0.005 0.02=2+0.001=2.001;

[0162] Therefore, the difference loss function L for this frame is:

[0163] =1.0 15+0.5 200.0001+0.3 2.001≈15+100+0.6≈115.6;

[0164] By continuously iterating this process, the loss function is calculated for each frame in the entire video sequence, and the camera parameters and rendering parameters are adjusted based on the calculation results through backpropagation until the total loss L drops to an acceptable range, ensuring that the predicted video frame sequence is visually highly consistent with the original video sequence and dynamically smooth.

[0165] Furthermore, in the process of adjusting model parameters based on the difference loss function, an optimization algorithm such as gradient descent or its variants (such as Adam, RMSprop, etc.) is usually used. The following are the basic steps, taking gradient descent as an example:

[0166] First, we need to calculate the difference loss function About model parameters The gradient of The gradient points in the direction of the fastest increase in the loss function at the current parameter value, with each component corresponding to the partial derivative of the corresponding parameter in the parameter vector. This step is usually performed automatically by deep learning frameworks such as TensorFlow or PyTorch, and is calculated using the backpropagation algorithm.

[0167] After obtaining the gradient, according to the principle of gradient descent, we need to adjust the parameters in the direction of the negative gradient in order to reduce the value of the loss function. The general form of the update rule is:

[0168] ;

[0169] in, The learning rate determines the step size of parameter updates. Choosing an appropriate learning rate is crucial. A learning rate that is too large can lead to unstable training and may miss the optimal solution; a learning rate that is too small can lead to slow convergence.

[0170] The above process is repeated multiple times on the entire dataset (batch or mini-batch), with each iteration being called an epoch. At the end of each epoch, all samples (or a subset of samples) are traversed and the parameters are updated accordingly. As the number of iterations increases, the model parameters are gradually adjusted to the value that minimizes the discrepancy loss function 𝐿, resulting in increasingly better performance on the training data.

[0171] For example, a simplified example can be provided to illustrate how this formula can be applied in practice. Suppose we have a very simple linear regression model that we are trying to fit to a set of data points to predict the output The model can be in the form of ,in are the camera parameters, are rendering parameters.

[0172] Loss Function is the mean squared error (MSE).

[0173] Therefore, the following simplifications can be made:

[0174] Current model parameters: ;

[0175] Learning rate( ): 0.01;

[0176] Assume that for a particular input-output pair , calculated about and The gradients (partial derivatives) are: ;

[0177] According to the gradient descent formula, the parameters can be updated as follows:

[0178] for ;

[0179] for ;

[0180] Therefore, after one round of gradient descent update, the model parameters are adjusted to and .

[0181] It should be noted that the gradient calculation in actual applications involves averaging the entire training dataset (or its mini-batch), and the model and loss function may be more complicated, but the basic principles are the same.

[0182] Figure 2 A schematic diagram of the structure of a video data processing system based on a digital twin scenario provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the system includes:

[0183] An acquisition and processing module 21 is configured to acquire video data from an actual environment, decompose the video data into a video frame sequence, and perform spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes an object motion path, subject behavior pattern, and scene information corresponding to the video frame, the object motion path, subject behavior pattern, and scene information corresponding to the video frame being determined through spatiotemporal feature extraction;

[0184] a segmentation module 22, configured to segment a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix, and generate an element segmentation mask corresponding to each of the video elements;

[0185] a fusion module 23, configured to fuse the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step, and each column contains the segmentation mask features and motion features of all objects in the time step;

[0186] a positioning module 24 configured to locate and track dynamic elements in the video frame sequence in real time based on the second spatiotemporal feature matrix, and obtain 2D motion trajectories corresponding to the dynamic elements, wherein the dynamic elements are video elements with dynamic features in the video elements;

[0187] a generation module 25 for mapping the 2D motion trajectory of the dynamic element into a 3D space to create or update a target virtual 3D scene model, and inputting the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model;

[0188] The adjustment module 26 is used to compare the predicted video frame sequence with the video frame sequence decomposed by the video data frame by frame, and adjust the model parameters in the target virtual 3D scene model based on the comparison result, wherein the model parameters include at least object motion parameters, camera parameters or rendering parameters.

[0189] Optionally, in an embodiment of the present application, the acquisition and processing module 21 is specifically used to identify the action information of all subjects in the video frame sequence to determine the subject behavior pattern of each subject; identify scene change information that changes with time in the video frame sequence, and static information in the video frame sequence, and form scene information based on the scene change information and the static information; wherein the static information includes at least: scene structure, object layout and lighting conditions; convert the object motion paths, subject behavior patterns and scene information corresponding to multiple video frames into a preset numerical form to form a first spatiotemporal feature matrix.

[0190] Optionally, in an embodiment of the present application, the segmentation module 22 is specifically used to input the first spatiotemporal feature matrix into a pre-established semantic segmentation model, so as to predict each video frame in the first spatiotemporal feature matrix through the semantic segmentation model, and output a segmentation result, wherein the segmentation result includes multiple video elements and an element label corresponding to each of the video elements, the element label is used to indicate that the video element belongs to the foreground, background, or a specific object category in the video frame, and the semantic segmentation model is trained based on multiple video frame samples with element labels; an element segmentation mask corresponding to each of the video elements is generated, and the element segmentation mask exists in the form of a binary image, and the binary image form includes black and white colors, wherein white represents the area where the video element is located, and black represents the background area outside the video element.

[0191] Optionally, in an embodiment of the present application, the fusion module 23 is specifically used to extract object motion paths corresponding to multiple video frames from the first spatiotemporal feature matrix, and the object motion path corresponding to each video frame is consistent with the time step of the corresponding element segmentation mask; each element segmentation mask is converted into a segmentation mask feature, and the object motion path corresponding to each video frame is converted into a motion feature; for each video frame, the segmentation mask feature of the corresponding object is spliced with the motion feature to form a comprehensive feature vector; according to the time step and the number of objects, the comprehensive feature vectors corresponding to each video frame are arranged in sequence to generate a second spatiotemporal feature matrix, and the dimensions of the second spatiotemporal feature matrix include: number of time steps x (number of objects x (segmentation mask feature dimension + motion feature dimension)).

[0192] Optionally, in an embodiment of the present application, the positioning module 24 is specifically used to identify, based on the second spatiotemporal feature matrix, an area in the video frame sequence that has a significant motion difference from the background area outside the video element, and determine the area as the area where the dynamic element is located; in the area where the dynamic element is located, identify the category and bounding box of the dynamic element, and for each dynamic element, establish a motion trajectory of the dynamic element between the video frame sequence based on the category and bounding box of the dynamic element to generate a 2D motion trajectory corresponding to the dynamic element.

[0193] Optionally, in an embodiment of the present application, the generation module 25 is specifically used to extract 2D feature points on the 2D motion trajectory corresponding to the dynamic element, and match the 2D feature points between different video frames to determine matching 2D feature point pairs; using the matching 2D feature point pairs, calculate the corresponding points of the dynamic element in 3D space to generate the 3D motion trajectory of the dynamic element; based on the subject behavior pattern and the scene information in the first spatiotemporal feature matrix, construct an initial virtual 3D scene model, and integrate the 3D motion trajectory of the dynamic element into the initial virtual 3D scene model to generate a target virtual 3D scene model, wherein the dynamic element can move in the correct time and space position in the virtual 3D scene model; according to the target virtual 3D scene model and the acquired position information of the dynamic element in each of the video frames, render new video frames under preset camera parameters to form a predicted video frame sequence.

[0194] Optionally, in an embodiment of the present application, the adjustment module 26 is specifically used to calculate the difference loss function between the predicted video frame sequence and the video frame sequence decomposed by the video data, and the difference loss function includes a model parameter set, and the model parameter set includes at least: camera parameters or rendering parameters.

[0195] Optionally, in the embodiment of the present application, the adjustment module 26 is further configured to use the formula group:

[0196] , calculating a difference loss function between the predicted video frame sequence and the video frame sequence decomposed by the video data;

[0197] in, Expressed as a pixel-level loss function, Represented as a timestamp The predicted video frame corresponding to the frame is represents a sequence of video frames decomposed into the video data, It is expressed as the structural similarity index, which is used to measure the structural similarity between the predicted video frame sequence and the video frame sequence decomposed by the video data. The value range is between -1 and 1, where 1 means exactly the same and 0 means no correlation. It is calculated using the SSIM algorithm; Expressed as a hyperparameter, it is used to balance the weight of SSIM loss and L1 norm loss; Expressed as L1 norm loss, it is the sum of absolute errors between the predicted video frame sequence and the video frame sequence decomposed by the video data;

[0198] in, Expressed as geometric consistency loss function, Represented as the number of sampling points in the real 3D scene model, Represented as the number of sampling points in the target virtual 3D scene model, Represented as a real 3D scene model, Represented as the target virtual 3D scene model, Represented as a calculation point set Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a calculation point set Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a hyperparameter to adjust the normal consistency loss Loss of geometric consistency The weight in Expressed as normal consistency loss, it is used to measure and the differences between;

[0199] in, Expressed as a temporal consistency loss function, Represented as a timestamp The predicted video frame corresponding to the frame is Represented as a timestamp The predicted video frame corresponding to the frame is Expressed as optical flow error, it is used to measure the consistency of pixel motion between two predicted video frames. Expressed as the penalty coefficient of optical flow error, Expressed as optical smoothness loss, it is used to control the smooth transition of video frames in timestamps. Expressed as a hyperparameter, it is used to balance the influence of the optical smoothness loss and the optical flow error on the temporal consistency loss;

[0200] in, Expressed as the difference loss function, Represented as a set of model parameters, Expressed as the weight of the pixel-level loss function, Expressed as the weight of the geometric consistency loss function, is represented as the weight of the temporal consistency loss function.

[0201] Figure 2 The video data processing system based on the digital twin scenario can perform Figure 1 The implementation principle and technical effects of the video data processing method based on the digital twin scenario described in the illustrated embodiment will not be repeated here. The specific manner in which each module and unit performs operations in the video data processing system based on the digital twin scenario in the above embodiment has been described in detail in the embodiment of the method and will not be elaborated here.

[0202] In one possible design, Figure 2 The video data processing system based on the digital twin scenario of the embodiment shown can be implemented as a computing device, such as Figure 3 As shown, the computing device may include a storage component 31 and a processing component 32;

[0203] The storage component 31 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 32. The processing component 32 is used to: collect video data from an actual environment, decompose the video data into a video frame sequence, and perform spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes an object motion path, subject behavior pattern, and scene information corresponding to the video frame, and the object motion path, subject behavior pattern, and scene information corresponding to the video frame are determined by spatiotemporal feature extraction; based on the first spatiotemporal feature matrix, multiple video elements are segmented from the video frame sequence, and an element segmentation mask corresponding to each of the video elements is generated; the element segmentation mask is merged with the object motion paths corresponding to the multiple video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step, each column contains the segmentation mask features and motion features of all objects within the time step; based on the second spatiotemporal feature matrix, the dynamic elements in the video frame sequence are located and tracked in real time, and the 2D motion trajectory corresponding to the dynamic elements is obtained, and the dynamic elements are video elements with dynamic features in the video elements; the 2D motion trajectory of the dynamic elements is mapped into the 3D space to create or update the target virtual 3D scene model, and the first spatiotemporal feature matrix and the dynamic elements are input into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model; the predicted video frame sequence is compared frame by frame with the video frame sequence decomposed by the video data, and based on the comparison result, the model parameters in the target virtual 3D scene model are adjusted, and the model parameters include at least object motion parameters, camera parameters or rendering parameters.

[0204] The processing component 32 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0205] The storage component 31 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile memory device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0206] The display component 33 may be an electroluminescent (EL) element, a liquid crystal display or a micro display having a similar structure, or a retinal direct display or a similar laser scanning display.

[0207] Of course, a computing device may also include other components, such as input / output interfaces, communication components, etc.

[0208] The input / output interface provides an interface between the processing component and the peripheral interface module, which can be an output device, an input device, etc.

[0209] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0210] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0211] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The embodiment shown is based on a video data processing method in a digital twin scenario.

[0212] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0213] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0214] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0215] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video data processing method based on a digital twin scenario, characterized in that: include: Collecting video data from an actual environment, decomposing the video data into a video frame sequence, and performing spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes an object motion path, a subject behavior pattern, and scene information corresponding to the video frame, the object motion path, the subject behavior pattern, and the scene information corresponding to the video frame being determined by the spatiotemporal feature extraction; Segmenting a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix, and generating an element segmentation mask corresponding to each of the video elements; Fusing the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step, and each column contains the segmentation mask features and motion features of all objects in the time step; Based on the second spatiotemporal feature matrix, locating and tracking dynamic elements in the video frame sequence in real time, and obtaining 2D motion trajectories corresponding to the dynamic elements, wherein the dynamic elements are video elements with dynamic features in the video elements; Mapping the 2D motion trajectory of the dynamic element into a 3D space to create or update a target virtual 3D scene model, and inputting the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model; The predicted video frame sequence is compared frame by frame with the video frame sequence decomposed from the video data, and based on the comparison result, the model parameters in the target virtual 3D scene model are adjusted, wherein the model parameters include at least camera parameters or rendering parameters.

2. The method according to claim 1, characterized in that The performing spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix includes: Identify all objects in the video frame sequence and continuously track the position of each object in different frames to obtain the object motion path corresponding to the video frame; identifying action information of all subjects in the video frame sequence to determine a subject behavior pattern of each subject; Identifying scene change information that changes over time in the video frame sequence and static information in the video frame sequence, and forming scene information based on the scene change information and the static information; The static information includes at least: scene structure, object layout and lighting conditions; The object motion paths, subject behavior patterns, and scene information corresponding to the plurality of video frames are converted into a preset numerical form to form a first spatiotemporal feature matrix.

3. The method according to claim 1, characterized in that The step of segmenting a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix and generating an element segmentation mask corresponding to each of the video elements includes: Inputting the first spatiotemporal feature matrix into a pre-established semantic segmentation model to predict each video frame in the first spatiotemporal feature matrix through the semantic segmentation model and outputting a segmentation result, wherein the segmentation result includes a plurality of video elements and an element label corresponding to each of the video elements, the element label being used to indicate whether the video element belongs to the foreground, background, or a specific object category in the video frame, the semantic segmentation model being trained based on a plurality of video frame samples with element labels; Generate an element segmentation mask corresponding to each video element, wherein the element segmentation mask exists in the form of a binary image, and the binary image form includes black and white colors, wherein white represents the area where the video element is located, and black represents the background area outside the video element.

4. The method according to claim 1, wherein The step of fusing the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix includes: Extracting object motion paths corresponding to a plurality of the video frames from the first spatiotemporal feature matrix, wherein the object motion path corresponding to each video frame is consistent with a time step of the corresponding element segmentation mask; Converting each of the element segmentation masks into segmentation mask features, and converting the object motion path corresponding to each of the video frames into motion features; For each of the video frames, concatenating the segmentation mask features of the corresponding object with the motion features to form a comprehensive feature vector; According to the time step and the number of objects, the comprehensive feature vectors corresponding to each of the video frames are arranged in sequence to generate a second spatiotemporal feature matrix.

5. The method according to claim 1, wherein The method of locating and tracking dynamic elements in the video frame sequence in real time based on the second spatiotemporal feature matrix, and obtaining 2D motion trajectories corresponding to the dynamic elements, wherein the dynamic elements are video elements with dynamic features in the video elements, includes: Based on the second spatiotemporal feature matrix, identifying an area in the video frame sequence that has a significant motion difference from a background area outside the video element, and determining the area as an area where a dynamic element is located; In the area where the dynamic element is located, the category and bounding box of the dynamic element are identified, and for each dynamic element, the motion trajectory of the dynamic element is established between the video frame sequence based on the category and bounding box of the dynamic element to generate a 2D motion trajectory corresponding to the dynamic element.

6. The method according to claim 1, characterized in that Mapping the 2D motion trajectory of the dynamic element into a 3D space to create or update a target virtual 3D scene model, and inputting the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model, including: Extracting 2D feature points on the 2D motion trajectory corresponding to the dynamic element, and matching the 2D feature points between different video frames to determine matching 2D feature point pairs; Using the matched 2D feature point pairs, calculating corresponding points of the dynamic element in 3D space to generate a 3D motion trajectory of the dynamic element; constructing an initial virtual 3D scene model based on the subject behavior pattern and the scene information in the first spatiotemporal feature matrix, and integrating the 3D motion trajectory of the dynamic element into the initial virtual 3D scene model to generate a target virtual 3D scene model, wherein the dynamic element can move at a correct temporal and spatial position in the virtual 3D scene model; According to the target virtual 3D scene model and the acquired position information of the dynamic elements in each of the video frames, new video frames are rendered under preset camera parameters to form a predicted video frame sequence.

7. The method according to claim 1, characterized in that The step of comparing the predicted video frame sequence with the video frame sequence decomposed from the video data frame by frame, and adjusting the model parameters in the target virtual 3D scene model based on the comparison result, includes: A difference loss function is calculated between the predicted video frame sequence and the video frame sequence decomposed from the video data, wherein the difference loss function includes a model parameter set, and the model parameter set includes at least: camera parameters or rendering parameters.

8. The method according to claim 7, characterized in that The calculating a difference loss function between the predicted video frame sequence and the video frame sequence decomposed from the video data includes: By formula group: , calculating a difference loss function between the predicted video frame sequence and the video frame sequence decomposed by the video data; in, Expressed as a pixel-level loss function, Represented as a timestamp The predicted video frame corresponding to the frame is represents a sequence of video frames decomposed into the video data, It is expressed as the structural similarity index, which is used to measure the structural similarity between the predicted video frame sequence and the video frame sequence decomposed by the video data. The value range is between -1 and 1, where 1 means exactly the same and 0 means no correlation. It is calculated using the SSIM algorithm; Expressed as a hyperparameter, it is used to balance the weight of SSIM loss and L1 norm loss; Expressed as L1 norm loss, it is the sum of absolute errors between the predicted video frame sequence and the video frame sequence decomposed by the video data; in, Expressed as geometric consistency loss function, Represented as the number of sampling points in the real 3D scene model, Represented as the number of sampling points in the target virtual 3D scene model, Represented as a real 3D scene model, Represented as the target virtual 3D scene model, Represented as a set of computational points Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a set of computational points Every point Arrival Set Nearest point The sum of the squares of the distances, Represented as a hyperparameter to adjust the normal consistency loss Loss of geometric consistency The weight in Expressed as normal consistency loss, it is used to measure and the differences between; in, Expressed as a temporal consistency loss function, Represented as a timestamp The predicted video frame corresponding to the frame is Represented as a timestamp The predicted video frame corresponding to the frame is Expressed as optical flow error, it is used to measure the consistency of pixel motion between two predicted video frames. Expressed as the penalty coefficient of optical flow error, Expressed as optical smoothness loss, it is used to control the smooth transition of video frames in timestamps. Expressed as a hyperparameter, it is used to balance the influence of the optical smoothness loss and the optical flow error on the temporal consistency loss; in, Expressed as the difference loss function, Represented as a set of model parameters, Expressed as the weight of the pixel-level loss function, Expressed as the weight of the geometric consistency loss function, is represented as the weight of the temporal consistency loss function.

9. A video data processing system based on a digital twin scenario, characterized in that: include: an acquisition and processing module, configured to acquire video data from an actual environment, decompose the video data into a video frame sequence, and perform spatiotemporal feature extraction on the video frame sequence to determine a first spatiotemporal feature matrix, wherein each row of the first spatiotemporal feature matrix includes a video frame in the video frame sequence, and each column includes an object motion path, a subject behavior pattern, and scene information corresponding to the video frame, the object motion path, the subject behavior pattern, and scene information corresponding to the video frame being determined through the spatiotemporal feature extraction; a segmentation module, configured to segment a plurality of video elements from the video frame sequence based on the first spatiotemporal feature matrix, and generate an element segmentation mask corresponding to each of the video elements; a fusion module, configured to fuse the element segmentation mask with the object motion paths corresponding to the plurality of video frames in the first spatiotemporal feature matrix to generate a second spatiotemporal feature matrix, wherein each row of the second spatiotemporal feature matrix represents a time step, and each column contains the segmentation mask features and motion features of all objects in the time step; a positioning module, configured to locate and track dynamic elements in the video frame sequence in real time based on the second spatiotemporal feature matrix, and obtain 2D motion trajectories corresponding to the dynamic elements, wherein the dynamic elements are video elements with dynamic features in the video elements; a generation module, configured to map the 2D motion trajectory of the dynamic element into a 3D space to create or update a target virtual 3D scene model, and input the first spatiotemporal feature matrix and the dynamic element into the target virtual 3D scene model to generate a predicted video frame sequence through the target virtual 3D scene model; An adjustment module is used to compare the predicted video frame sequence with the video frame sequence decomposed by the video data frame by frame, and based on the comparison result, adjust the model parameters in the target virtual 3D scene model, wherein the model parameters include at least object motion parameters, camera parameters or rendering parameters.

10. A computing device, characterized in that It includes a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement a video data processing method based on a digital twin scenario as described in any one of claims 1 to 8.

Images