Artificial intelligence video processing method based on digital twinning

Abstract

This invention discloses an artificial intelligence video processing method based on digital twins, belonging to the field of intelligent video analysis technology. The invention solves the problems of temporal inconsistency and missing frames in multi-source video data through timestamp alignment mechanisms and temporal interpolation, ensuring data integrity and synchronization. It utilizes multi-source integration technology and 3D convolutional networks to fuse multi-view video data, constructing a high-precision virtual model. Real-time parameter updates and Kalman filtering optimize dynamic scene mapping, improving the model's accuracy and real-time performance. Furthermore, by leveraging long short-term memory networks and attention mechanisms, it accurately predicts target trajectories and classifies behavioral patterns, providing reliable support for intelligent decision-making in complex dynamic environments. This effectively improves the efficiency and accuracy of multi-source video data processing, enhancing the system's adaptability and intelligence in dynamic scenes.

Description

Technical Field

[0001] This invention relates to the field of intelligent video analysis technology, specifically an artificial intelligence video processing method based on digital twins. Background Technology

[0002] Video data processing and intelligent analysis play a crucial role in modern society, with widespread applications in key scenarios such as smart cities, industrial monitoring, and autonomous driving, directly impacting social efficiency and safety. With the rapid development of sensor technology and computing power, achieving accurate perception and understanding of the real world through video data has become an indispensable research direction; however, current solutions still exhibit significant limitations in practical applications. Existing video analysis methods often rely on single data sources or static models, making them difficult to adapt to complex and ever-changing real-world scenarios, especially lacking in real-time performance and accuracy in dynamic environments. Furthermore, traditional methods suffer from a disconnect between data processing and scene understanding, resulting in a lack of comprehensiveness and depth in the analysis results.

[0003] The core challenge in this field lies in effectively integrating multi-source video data and achieving dynamic synchronization with real-world scenes, while simultaneously enhancing the ability of AI algorithms to understand complex content. Specifically, a technological gap exists between the comprehensive acquisition of video data and the real-time mapping of virtual models; latency and distortion in data synchronization limit the accuracy of scene reconstruction. Furthermore, the performance of deep learning algorithms in target recognition and behavior prediction is highly dependent on the quality and quantity of labeled data, and their generalization ability is insufficient when faced with unfamiliar new scenes. These unresolved technical factors make it difficult to achieve efficient target trajectory prediction and behavior analysis in dynamic environments, thus affecting the reliability and practicality of intelligent processing. Summary of the Invention

[0004] The purpose of this invention is to provide an artificial intelligence video processing method based on digital twins. It solves the problems of data synchronization and integrity by using timestamp alignment and interpolation, constructs a high-precision virtual model by using multi-source integration and three-dimensional convolutional networks, and achieves accurate trajectory prediction and behavior classification by combining LSTM and attention mechanisms, thereby providing efficient data processing and intelligent decision support for complex dynamic scenes.

[0005] The objective of this invention can be achieved through the following technical solutions:

[0006] This application provides an artificial intelligence video processing method based on digital twins, including the following steps:

[0007] S1. Acquire multi-source video data from multiple sensors, align the time series of each data stream using a preset timestamp alignment mechanism, fill in missing frames using time interpolation, and obtain a synchronized multi-source video dataset.

[0008] S2. Based on the synchronized multi-source video dataset, video frames from different perspectives are fused using multi-source integration technology, and the feature weights of each frame are calculated using the weighted average method to obtain a unified multi-view feature representation.

[0009] S3. Based on a unified multi-view feature representation, construct a high-precision virtual model, extract spatiotemporal features using a 3D convolutional network, and determine the initial state of the virtual model by updating parameters in real time to map the dynamic scene; then test the data synchronization delay performance.

[0010] S4. Using the optimized dynamic virtual model, obtain the spatiotemporal coordinate sequence of the target object in the scene, use a long short-term memory network to analyze the coordinate change trend, and judge the prediction result of the target trajectory.

[0011] S5. Based on the prediction results of the target trajectory, apply deep learning algorithms to extract the behavioral features of the target object, enhance the key frame weights through an attention mechanism, and obtain the behavioral pattern classification output; when the dynamic environment changes beyond the preset adaptation range, retrieve similar scene features from the pre-established scene library, adjust the deep learning algorithm parameters, and determine the final behavioral analysis results.

[0012] S6. Based on the final behavior analysis results, the processing strategy for multi-source video data is updated in conjunction with real-time synchronization capabilities. Online learning methods are used to optimize and integrate technical parameters to obtain the feature representation input for the next cycle.

[0013] Furthermore, by aligning the time series of each data stream using a preset timestamp alignment mechanism and filling in missing frames using time interpolation, a synchronized multi-source video dataset is obtained, specifically including:

[0014] Multi-source video data is collected from multiple sensors, and each sensor's video stream carries its own timestamp information; a preset timestamp alignment mechanism is activated, and a unified time base is determined as the reference time point for all data streams;

[0015] The timestamps in each video data stream are analyzed one by one, the deviation value between each timestamp and the reference time is calculated, and the time series of each data stream is adjusted to be consistent with the reference time through translation and scaling mathematical transformations based on the deviation value.

[0016] After completing the timestamp alignment, time interpolation is used to process the data, analyze several adjacent frames before and after the missing frame, and extract key features, including the object's motion direction, speed, shape changes, and the color and brightness information of the corresponding pixels in adjacent frames.

[0017] Based on pixel-level features, linear or bilinear interpolation algorithms are used to calculate the pixel values ​​of the missing frames based on the pixel information of adjacent frames. Based on the angular features of object motion, the position and state of the object at the time of the missing frame are predicted by the motion model to construct a complete picture of the missing frame. Then, the processed multi-source video data are integrated into a synchronized dataset.

[0018] Furthermore, multi-source fusion technology is used to fuse video frames from different perspectives in multi-source video datasets from different synchronization points. The feature weights of each frame are calculated using a weighted average method to obtain a unified multi-view feature representation, specifically including:

[0019] By fusing sensor data and video frames from different perspectives, identifying features in each video frame, and matching feature points of the same object from different perspectives, the frames are spatially stitched and integrated to initially construct a comprehensive picture with multi-view information.

[0020] The weighted average method is used to calculate the feature weights of each frame. First, the color, texture and edge details of the objects in each frame are extracted. For each feature dimension, the changes in different view frames and the degree of contribution to the overall scene description are analyzed.

[0021] By analyzing and assigning values ​​to each feature dimension, a set of comprehensive weights is calculated for each frame. Based on the comprehensive weights, the features of each frame are weighted and summed, and the features of video frames from different perspectives are organically integrated to obtain a unified multi-view feature representation.

[0022] When the data synchronization delay exceeds a preset threshold, the current frame position is predicted using the Kalman filter algorithm, the model state is adjusted, and an optimized dynamic virtual model is obtained.

[0023] Furthermore, based on a unified multi-view feature representation, a high-precision virtual model is constructed, specifically including:

[0024] By utilizing the rich information in the unified multi-view feature representation, a virtual model architecture corresponding to the real-world scene is built.

[0025] Then, a three-dimensional convolutional network is used to extract spatiotemporal features. The unified multi-view feature data is reasonably organized according to the time series and spatial dimensions and then input into the three-dimensional convolutional network. The convolutional kernels in the network slide on the three-dimensional scale of time and space to perform convolution operations on the data.

[0026] By combining convolutional layers, pooling layers, and activation functions, the spatiotemporal features of the data are automatically extracted. By updating parameters in real time to map dynamic scenes, as the real scene changes and the multi-source video data is continuously updated, the model will acquire new data in real time and re-analyze and evaluate the previously extracted spatiotemporal features.

[0027] Based on the differences between the new data and the existing model parameters, the backpropagation algorithm is used to optimize the parameters in the model dynamically, and then the data synchronization delay performance is tested.

[0028] Furthermore, after performing data synchronization delay performance testing, when the data synchronization delay exceeds a preset threshold, the current frame position is predicted using the Kalman filter algorithm to obtain a preliminary prediction result. Based on the preliminary prediction result, the changing trend of the current frame position is obtained, and smoothed using a linear interpolation method to obtain smoothed position data.

[0029] By adjusting the key parameters in the model state using the smoothed position data, an updated state description is obtained. Then, the real-time offset of the dynamic virtual model is calculated to obtain the offset adjustment value.

[0030] The coordinates of the dynamic virtual model are corrected by adjusting the offset value to obtain the corrected model data. Then, the prediction output of the Kalman filter is fused to obtain the optimized dynamic virtual model.

[0031] Furthermore, the spatiotemporal coordinate sequence of the target object in the scene is obtained, and a Long Short-Term Memory (LSTM) network is used to analyze the coordinate change trend to determine the predicted trajectory of the target object. Specifically, this includes:

[0032] Data related to the target object is extracted from the optimized dynamic virtual model. The representation of the target object in the virtual model is located by the data extraction algorithm. The coordinate values ​​of the target object in three-dimensional space at different time points are obtained from the data. The coordinate values ​​are arranged in chronological order to obtain the spatiotemporal coordinate sequence of the target object.

[0033] The obtained spatiotemporal coordinate sequence is input into a long short-term memory network to analyze the coordinate change trend and obtain a trend description. Based on the coordinate change trend, the movement direction of the target object is calculated to obtain the direction vector.

[0034] By adjusting the state parameters in the dynamic virtual model using the direction vector, the updated model state is obtained. Then, the trend description of the long short-term memory network is fused to obtain the trajectory prediction output.

[0035] The trajectory prediction output is compared with the actual coordinate sequence to obtain a stable prediction result. The position of the target object in the scene analysis is updated based on the stable prediction result to obtain the final coordinate distribution.

[0036] Furthermore, based on the predicted trajectory of the target object, deep learning algorithms are applied to extract the behavioral features of the target object. Specifically, this includes: taking the predicted trajectory of the target object as input and processing the trajectory data through a deep learning model; extracting local features through convolutional layers, capturing long-term dependencies in the time series using recurrent layers, and performing nonlinear transformations using activation functions to extract the behavioral features of the target object.

[0037] Furthermore, the keyframe weights are enhanced through an attention mechanism to obtain the behavior pattern classification output, specifically including:

[0038] The extracted behavioral features are mapped to form a feature matrix, the attention weight matrix is ​​initialized, and the attention score for each frame is calculated by combining the dot product attention calculation function with the learnable query vector.

[0039] Then, the scores are normalized by the Softmax function to make the scores the relative weights of the frames. The original feature matrix is ​​then weighted and fused to highlight key frame information. A classification model is then constructed, and the model is trained with the comprehensive feature vector processed by the attention mechanism and the corresponding behavior pattern label data.

[0040] After training, new behavioral feature data are processed through the same attention mechanism to obtain a comprehensive feature vector, which is then input into the trained model. The model outputs a probability distribution vector, and the category with the highest probability is selected as the final behavioral pattern classification output.

[0041] Furthermore, based on the final behavioral analysis results, and combined with real-time synchronization capabilities, the processing strategy for multi-source video data is updated. Online learning methods are used to optimize and integrate technical parameters to obtain the feature representation input for the next cycle, specifically including:

[0042] By combining behavioral analysis results with real-time synchronization technology to process multi-source video data, an adjusted processing strategy is obtained. An online learning method is then used to optimize and integrate parameters for the adjusted processing strategy, resulting in an optimized parameter set.

[0043] Video data is processed by an optimized set of parameters to obtain a preliminary distribution of feature representations. When the deviation between the preliminary distribution of feature representations and the data obtained by the synchronization technique exceeds a preset threshold, an adjustment method is used to iteratively optimize the set of parameters to obtain a stable feature representation.

[0044] By combining stable feature representations with a periodic input update dynamic update strategy, feature data for the next period is obtained. The dynamic update strategy is used to process multi-source video data, determine the trend changes in behavior analysis, and obtain the final analysis output.

[0045] By combining the final analysis output with real-time synchronization technology to adjust the processing strategy, an updated video data processing workflow is obtained.

[0046] The beneficial effects of this invention are as follows:

[0047] By employing a pre-defined timestamp alignment mechanism and time interpolation, the inconsistencies and missing frames in the time series of multi-source video data were resolved. The timestamp alignment mechanism was used to adjust the time series of each data stream to a unified benchmark, and accurate temporal correspondence was achieved through translation and scaling mathematical transformations. Time interpolation was used to fill in missing frames, and the content of missing frames was predicted by combining pixel features and motion models. This not only eliminated temporal differences but also restored the integrity of the video data, providing a high-quality data foundation for subsequent multi-view feature fusion and behavior analysis, and ensuring the accuracy and reliability of data processing.

[0048] By employing multi-source integration technology and a 3D convolutional network, the complexity of multi-view video data fusion and dynamic scene mapping is addressed. Multi-source integration technology fuses video frames from different perspectives, and a weighted average method is used to calculate the feature weights of each frame, resulting in a unified multi-view feature representation. A 3D convolutional network extracts spatiotemporal features, and real-time parameter updates and backpropagation algorithms are used to dynamically adjust the model state, ensuring the virtual model accurately maps dynamic scenes. When data synchronization delays are detected, a Kalman filter algorithm optimizes the model state, improving not only the accuracy and real-time performance of the virtual model but also providing technical support for high-precision simulation of complex dynamic scenes, enhancing the system's adaptability and robustness.

[0049] By employing Long Short-Term Memory (LSTM) networks and attention mechanisms, the accuracy issues of target trajectory prediction and behavior pattern classification are addressed. The LSTM network analyzes the spatiotemporal coordinate sequence of the target object, capturing long-term dependencies, predicting the target trajectory, and adjusting the model state. The attention mechanism enhances keyframe weights, highlighting important feature information, and combines this with a deep learning model to extract and classify behavioral features. When the predicted results deviate significantly from the actual data, iterative parameter optimization ensures the stability and accuracy of trajectory prediction and behavior classification. This not only improves the accuracy of target trajectory prediction and behavior pattern classification but also provides a reliable basis for intelligent decision-making in complex dynamic environments, enhancing the system's intelligence level and practicality. Attached Figure Description

[0050] To better understand and implement this application, the technical solution is described in detail below with reference to the accompanying drawings.

[0051] Figure 1 A flowchart illustrating the AI ​​video processing method based on digital twins provided in this application;

[0052] Figure 2 A flowchart illustrating the process of obtaining a synchronized multi-source video dataset using the digital twin-based artificial intelligence video processing method provided in this application;

[0053] Figure 3 This is a flowchart illustrating the prediction results of the AI ​​video processing method based on digital twins provided in this application for determining the trajectory of a target. Detailed Implementation

[0054] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, exemplary embodiments will be described in detail below, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and systems consistent with some aspects of this application as detailed in the appended claims.

[0055] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0056] The following detailed description of the specific implementation methods, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided in detail.

[0057] Please see Figures 1-3 This embodiment provides an artificial intelligence video processing method based on digital twins, including the following steps:

[0058] S1. Acquire multi-source video data from multiple sensors, align the time series of each data stream using a preset timestamp alignment mechanism, fill in missing frames using time interpolation, and obtain a synchronized multi-source video dataset.

[0059] Furthermore, by aligning the time series of each data stream using a preset timestamp alignment mechanism and filling in missing frames using time interpolation, a synchronized multi-source video dataset is obtained, specifically including:

[0060] S11. Collect multi-source video data from multiple sensors. Each video stream acquired by a sensor carries its own timestamp information. Activate the preset timestamp alignment mechanism and then determine a unified time base as the reference time point for all data streams.

[0061] S12. Analyze the timestamps in each video data stream one by one, calculate the deviation value between each timestamp and the reference time, and adjust the time series of each data stream to be consistent with the reference time through translation and scaling mathematical transformations based on the deviation value, thereby achieving accurate time correspondence.

[0062] S13. After completing the timestamp alignment, for the problem of missing frames in some video data that may be caused by sensor failure, packet loss, etc., time interpolation is used to process them. The key features are extracted by deeply analyzing several adjacent frames before and after the missing frame, including the direction of motion, speed, shape change of the object, and the color and brightness information of the corresponding pixels in the adjacent frames.

[0063] S14. Based on pixel-level features, linear or bilinear interpolation algorithms are used to calculate the pixel values ​​of the missing frame according to the pixel information of adjacent frames; based on the angular features of object motion, the position and state of the object at the time of the missing frame are predicted through a motion model to construct a complete picture of the missing frame. These two methods can effectively fill in missing frames and restore the integrity of video data.

[0064] After timestamp alignment and missing frame processing, the time differences and data loss issues of the various data streams were resolved, and the processed multi-source video data was integrated into a synchronized dataset.

[0065] S2. Based on the synchronized multi-source video dataset, video frames from different perspectives are fused using multi-source integration technology, and the feature weights of each frame are calculated using the weighted average method to obtain a unified multi-view feature representation.

[0066] Furthermore, multi-source integration technology is used to fuse video frames from different sensors and different perspectives, identify the features in each video frame, and match the feature points of the same object under different perspectives to reasonably stitch and integrate the frames in space, thus initially constructing a comprehensive picture that covers multi-view information.

[0067] Subsequently, to highlight the importance of video frames from different perspectives to the overall feature representation, a weighted average method was used to calculate the feature weights of each frame. First, detailed features such as color, texture, and edges of objects in each frame were extracted. For each feature dimension, its variation across different perspective frames and its contribution to the overall scene description were analyzed. For example, if a frame from one perspective clearly displays the key details and texture of an object, while the texture feature is blurry in other perspectives, then this frame will have a relatively high weight in the texture feature dimension. Through this analysis and assignment of values ​​to each feature dimension, a set of comprehensive weights was calculated for each frame. Finally, the features of each frame were weighted and summed according to the comprehensive weights, organically fusing the features of video frames from different perspectives to obtain a unified multi-view feature representation that integrates the advantages of multi-source video data.

[0068] S3. Based on a unified multi-view feature representation, construct a high-precision virtual model, extract spatiotemporal features using a 3D convolutional network, and determine the initial state of the virtual model by updating parameters in real time to map the dynamic scene; then test the data synchronization delay performance.

[0069] When the data synchronization delay exceeds the preset threshold, the current frame position is predicted by the Kalman filter algorithm, the model state is adjusted, and an optimized dynamic virtual model is obtained.

[0070] Furthermore, based on a unified multi-view feature representation, a high-precision virtual model is constructed, specifically including:

[0071] By utilizing the rich information in the unified multi-view feature representation, a virtual model architecture corresponding to the real scene is built. The design of the model structure must fully consider the ability to carry the complex spatial and temporal dimension information contained in multi-source video data to ensure accurate simulation of the real scene.

[0072] Then, a three-dimensional convolutional network is used to extract spatiotemporal features. The unified multi-view feature data is reasonably organized according to the time series and spatial dimensions and then input into the three-dimensional convolutional network. The convolutional kernels in the network slide on the three-dimensional scale of time and space to perform convolution operations on the data.

[0073] By combining convolutional layers, pooling layers, and activation functions, the spatial shape, position, and temporal and spatial features of objects in the data, as well as their motion trajectories and dynamic behaviors that change over time, can be automatically extracted. For example, convolutional kernels can capture features such as changes in the spatial position of objects at different times and the motion trends of objects between adjacent time frames.

[0074] During model operation, parameters are updated in real time to map the dynamic scene. As the real-world scene changes continuously, the multi-source video data is also continuously updated. The model acquires new data in real time and re-analyzes and evaluates the previously extracted spatiotemporal features. Based on the differences between the new data and the existing model parameters, optimization methods such as backpropagation are used to dynamically adjust the parameters in the model, such as the weights and biases of the convolutional kernels. This ensures that the virtual model can accurately reflect the real-time changes of the real-world scene and always remain synchronized with the dynamic scene. After a series of parameter updates, the model reaches a stable state, at which point the initial state of the virtual model is determined. This initial state contains accurate simulation information of the real-world scene constructed based on the current multi-view feature representation.

[0075] Subsequently, data synchronization latency performance testing was conducted. During the virtual model's operation, a dedicated monitoring module continuously recorded the time interval between the acquisition of multi-source video data and its input into the virtual model for processing, as well as the time lag between the model's output and the actual situation in the real scene. Through statistical analysis of this time data, key indicators such as the average, maximum, and minimum values ​​of the data synchronization latency were calculated. For example, if multiple instances of long time intervals between data acquisition and model processing, or if the time lag between the model's output and the actual situation in the real scene exceeds an acceptable range, it indicates a data synchronization latency problem.

[0076] Furthermore, after detecting the data synchronization delay performance, when the data synchronization delay exceeds a preset threshold, the current frame position is predicted using the Kalman filter algorithm to obtain a preliminary prediction result. Based on the preliminary prediction result, the changing trend of the current frame position is obtained, and smoothed using a linear interpolation method to obtain smoothed position data.

[0077] By adjusting the key parameters in the model state using the smoothed position data, an updated state description is obtained. Then, the real-time offset of the dynamic virtual model is calculated to obtain the offset adjustment value.

[0078] The coordinates of the dynamic virtual model are corrected by adjusting the offset value to obtain the corrected model data. Then, the prediction output of the Kalman filter is fused to obtain the optimized dynamic virtual model.

[0079] S4. Using the optimized dynamic virtual model, obtain the spatiotemporal coordinate sequence of the target object in the scene, use a long short-term memory network to analyze the coordinate change trend, and judge the prediction result of the target trajectory.

[0080] Furthermore, the spatiotemporal coordinate sequence of the target object in the scene is obtained, and a Long Short-Term Memory (LSTM) network is used to analyze the coordinate change trend to determine the predicted trajectory of the target object. Specifically, this includes:

[0081] S41. Extract data information related to the target object from the optimized dynamic virtual model, locate the representation of the target object in the virtual model through the data extraction algorithm, and obtain the coordinate values ​​of the target object in three-dimensional space at different time points. Arrange the coordinate values ​​in chronological order to obtain the spatiotemporal coordinate sequence of the target object.

[0082] S42. The obtained spatiotemporal coordinate sequence is input into a Long Short-Term Memory (LSTM) network to analyze the coordinate change trend and obtain a trend description. Based on the coordinate change trend, the movement direction of the target object is calculated to obtain a direction vector. The LSTM network structure has unique advantages in processing time series data. Its internal memory units can effectively capture long-term dependencies. When processing the spatiotemporal coordinate sequence of the target object, the input gate, forget gate, and output gate in the network work together. The input gate determines which new coordinate information will be added to the memory unit; the forget gate controls which historical information in the memory unit needs to be retained or discarded; and the output gate generates the output result based on the state of the memory unit. As the coordinate sequence data is input into the network sequentially, the LSTM network gradually learns the change law of coordinate values ​​over time, such as whether the object's movement direction is stable over a period of time and whether the speed has changed significantly.

[0083] S43. By adjusting the state parameters in the dynamic virtual model through the direction vector, the updated model state is obtained. Then, the trend description of the long short-term memory network is fused to obtain the trajectory prediction output.

[0084] S44. Based on the trajectory prediction output and the actual coordinate sequence, a stable prediction result is obtained. The position of the target object in the scene analysis is updated using the stable prediction result to obtain the final coordinate distribution.

[0085] Specifically, when the deviation between the trajectory prediction output and the actual coordinate sequence exceeds a preset threshold, a stable prediction result is obtained by iteratively adjusting the state parameters.

[0086] Specifically, by combining an optimized dynamic virtual model with a Long Short-Term Memory (LSTM) network, the system can efficiently acquire the spatiotemporal coordinate sequence of a target object and accurately analyze its motion trend, thereby achieving accurate prediction of the target trajectory. This process not only captures the dynamic changes of the target object in three-dimensional space but also effectively handles long-term dependencies in the time series through the memory units of the LSTM network, solving the problems of information loss or inaccurate prediction that may occur when traditional methods deal with complex motion trajectories. By adjusting the model state through direction vectors and combining it with trend descriptions to generate trajectory prediction output, the system can update the position information of the target object in real time, ensuring the stability and accuracy of the prediction results. Furthermore, when the deviation between the prediction result and the actual coordinate sequence exceeds a preset threshold, the system self-corrects by iteratively adjusting the state parameters, further improving the reliability of the prediction.

[0087] S5. Based on the prediction results of the target trajectory, apply deep learning algorithms to extract the behavioral features of the target object, enhance the key frame weights through an attention mechanism, and obtain the behavioral pattern classification output; when the dynamic environment changes beyond the preset adaptation range, retrieve similar scene features from the pre-established scene library, adjust the deep learning algorithm parameters, and determine the final behavioral analysis results.

[0088] Furthermore, based on the predicted trajectory of the target object, deep learning algorithms are applied to extract its behavioral features, specifically including:

[0089] The predicted trajectory of the target is used as input, and the trajectory data is processed by deep learning models (such as recurrent neural networks (RNN), long short-term memory networks (LSTM), or convolutional neural networks (CNN). These models can capture the spatiotemporal features in the trajectory data, such as changes in the position of the object, velocity, acceleration, and motion patterns.

[0090] By extracting local features through convolutional layers, capturing long-term dependencies in time series using recurrent layers, and combining activation functions for nonlinear transformation, the behavioral features of the target object can be extracted.

[0091] Furthermore, the keyframe weights are enhanced through an attention mechanism to obtain the behavior pattern classification output, specifically including:

[0092] The extracted behavioral features are mapped to form a feature matrix, the attention weight matrix is ​​initialized, and the attention score for each frame is calculated by combining the learnable query vector with attention calculation functions such as dot product.

[0093] Then, after normalization by the Softmax function, the scores become the relative weights of the frames. This is used to weight and fuse the original feature matrix to highlight key frame information. Then, classification models such as support vector machines, multilayer perceptrons, or convolutional neural networks are constructed. The model is trained with the comprehensive feature vectors processed by the attention mechanism and the corresponding behavioral pattern label data.

[0094] After training, new behavioral feature data are processed through the same attention mechanism to obtain a comprehensive feature vector, which is then input into the trained model. The model outputs a probability distribution vector, and the category with the highest probability is selected as the final behavioral pattern classification output, thereby accurately judging the behavioral pattern of the target object.

[0095] S6. Based on the final behavior analysis results, the processing strategy for multi-source video data is updated in conjunction with real-time synchronization capabilities. Online learning methods are used to optimize and integrate technical parameters to obtain the feature representation input for the next cycle.

[0096] Furthermore, based on the final behavioral analysis results, and combined with real-time synchronization capabilities, the processing strategy for multi-source video data is updated. Online learning methods are used to optimize and integrate technical parameters to obtain the feature representation input for the next cycle, specifically including:

[0097] By combining behavioral analysis results with real-time synchronization technology to process multi-source video data, an adjusted processing strategy is obtained. An online learning method is then used to optimize and integrate parameters for the adjusted processing strategy, resulting in an optimized parameter set.

[0098] Video data is processed by an optimized set of parameters to obtain a preliminary distribution of feature representations. When the deviation between the preliminary distribution of feature representations and the data obtained by the synchronization technique exceeds a preset threshold, an adjustment method is used to iteratively optimize the set of parameters to obtain a stable feature representation.

[0099] By combining stable feature representations with a periodic input update dynamic update strategy, feature data for the next period is obtained. The dynamic update strategy is used to process multi-source video data, determine the trend changes in behavior analysis, and obtain the final analysis output.

[0100] By combining the final analysis output with real-time synchronization technology to adjust the processing strategy, an updated video data processing workflow is obtained.

[0101] This embodiment uses a digital twin-based AI video processing method. First, it acquires multi-source video data from multiple sensors, processes it using a timestamp alignment mechanism and time interpolation, and obtains a synchronized multi-source video dataset. Next, it fuses video frames from different perspectives and calculates weights to obtain a unified multi-view feature representation. Then, it constructs a high-precision virtual model, extracts spatiotemporal features to determine the initial state, and optimizes the model if the data synchronization delay exceeds a threshold. Using the optimized model, it obtains the spatiotemporal coordinate sequence of the target object, analyzes the coordinate change trend to determine the target trajectory prediction result, extracts behavioral features based on the prediction result, and classifies them. If environmental changes exceed the range, it adjusts the algorithm parameters to obtain the final behavioral analysis result. Finally, it combines this result with real-time synchronization capabilities to update the processing strategy, optimize and integrate technical parameters, and provide feature representation input for the next cycle. This achieves comprehensive, accurate, and intelligent processing of video data, effectively analyzing the trajectory and behavior of target objects and adapting to dynamic environmental changes.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An artificial intelligence video processing method based on digital twins, characterized in that: Includes the following steps: Multi-source video data is acquired from multiple sensors, and the time series of each data stream is aligned using a preset timestamp alignment mechanism. Missing frames are filled in using time interpolation to obtain a synchronized multi-source video dataset. Based on a synchronized multi-source video dataset, video frames from different perspectives are fused using multi-source integration technology, and the feature weights of each frame are calculated using a weighted average method to obtain a unified multi-view feature representation. This includes: fusing sensor data and video frames from different perspectives, identifying features in each video frame, matching feature points of the same object from different perspectives, and spatially stitching and integrating the frames to initially construct a comprehensive picture with multi-view information; The weighted average method is used to calculate the feature weights of each frame. First, the color, texture and edge details of the objects in each frame are extracted. For each feature dimension, the changes in different view frames and the degree of contribution to the overall scene description are analyzed. By analyzing and assigning values ​​to each feature dimension, a set of comprehensive weights is calculated for each frame. Based on the comprehensive weights, the features of each frame are weighted and summed, and the features of video frames from different perspectives are organically integrated to obtain a unified multi-view feature representation. When the data synchronization delay exceeds the preset threshold, the current frame position is predicted by the Kalman filter algorithm, the model state is adjusted, and an optimized dynamic virtual model is obtained. Based on a unified multi-view feature representation, a high-precision virtual model is constructed. Spatiotemporal features are extracted using a 3D convolutional network. The initial state of the virtual model is determined by dynamically mapping the parameters in real time. Then, the data synchronization latency performance is tested. The spatiotemporal coordinate sequence of target objects in the scene is obtained through the optimized dynamic virtual model. The long short-term memory network is used to analyze the coordinate change trend and judge the prediction result of the target trajectory. Based on the predicted trajectory of the target, a deep learning algorithm is applied to extract the behavioral features of the target object. The keyframe weights are enhanced through an attention mechanism to obtain the behavioral pattern classification output. When the dynamic environment changes beyond the preset adaptation range, similar scene features are retrieved from the pre-established scene library, the parameters of the deep learning algorithm are adjusted, and the final behavioral analysis result is determined. Based on the final behavioral analysis results, the processing strategy for multi-source video data is updated using real-time synchronization capabilities. Online learning methods are then employed to optimize and integrate technical parameters, resulting in the feature representation input for the next cycle.

2. The artificial intelligence video processing method based on digital twins according to claim 1, characterized in that: By aligning the time series of each data stream using a pre-defined timestamp alignment mechanism and filling in missing frames using time interpolation, a synchronized multi-source video dataset is obtained, specifically including: Multi-source video data is collected from multiple sensors, and each sensor's video stream carries its own timestamp information; a preset timestamp alignment mechanism is activated, and a unified time base is determined as the reference time point for all data streams; The timestamps in each video data stream are analyzed one by one, the deviation value between each timestamp and the reference time is calculated, and the time series of each data stream is adjusted to be consistent with the reference time through translation and scaling mathematical transformations based on the deviation value. After completing the timestamp alignment, time interpolation is used to process the data, analyze several adjacent frames before and after the missing frame, and extract key features, including the object's motion direction, speed, shape changes, and the color and brightness information of the corresponding pixels in adjacent frames. Based on pixel-level features, linear or bilinear interpolation algorithms are used to calculate the pixel values ​​of the missing frames based on the pixel information of adjacent frames. Based on the angular features of object motion, the position and state of the object at the time of the missing frame are predicted by the motion model to construct a complete picture of the missing frame. Then, the processed multi-source video data are integrated into a synchronized dataset.

3. The artificial intelligence video processing method based on digital twins according to claim 1, characterized in that: A high-precision virtual model is constructed based on a unified multi-view feature representation, specifically including: By utilizing the rich information in the unified multi-view feature representation, a virtual model architecture corresponding to the real-world scene is built. Then, a three-dimensional convolutional network is used to extract spatiotemporal features. The unified multi-view feature data is reasonably organized according to the time series and spatial dimensions and then input into the three-dimensional convolutional network. The convolutional kernels in the network slide on the three-dimensional scale of time and space to perform convolution operations on the data. By combining convolutional layers, pooling layers, and activation functions, the spatiotemporal features of the data are automatically extracted. By updating parameters in real time to map dynamic scenes, as the real scene changes and the multi-source video data is continuously updated, the model will acquire new data in real time and re-analyze and evaluate the previously extracted spatiotemporal features. Based on the differences between the new data and the existing model parameters, the backpropagation algorithm is used to optimize the parameters in the model dynamically, and then the data synchronization delay performance is tested.

4. The artificial intelligence video processing method based on digital twins according to claim 3, characterized in that: After performing data synchronization delay performance testing, when the data synchronization delay exceeds a preset threshold, the current frame position is predicted using the Kalman filter algorithm to obtain a preliminary prediction result. Based on the preliminary prediction result, the changing trend of the current frame position is obtained, and smoothed using a linear interpolation method to obtain smoothed position data. By adjusting the key parameters in the model state using the smoothed position data, an updated state description is obtained. Then, the real-time offset of the dynamic virtual model is calculated to obtain the offset adjustment value. The coordinates of the dynamic virtual model are corrected by adjusting the offset value to obtain the corrected model data. Then, the prediction output of the Kalman filter is fused to obtain the optimized dynamic virtual model.

5. The artificial intelligence video processing method based on digital twins according to claim 1, characterized in that: Obtain the spatiotemporal coordinate sequence of target objects in the scene, analyze the coordinate change trend using a Long Short-Term Memory (LSTM) network, and determine the predicted trajectory of the target. Specifically, this includes: Data related to the target object is extracted from the optimized dynamic virtual model. The representation of the target object in the virtual model is located by the data extraction algorithm. The coordinate values ​​of the target object in three-dimensional space at different time points are obtained from the data. The coordinate values ​​are arranged in chronological order to obtain the spatiotemporal coordinate sequence of the target object. The obtained spatiotemporal coordinate sequence is input into a long short-term memory network to analyze the coordinate change trend and obtain a trend description. Based on the coordinate change trend, the movement direction of the target object is calculated to obtain the direction vector. By adjusting the state parameters in the dynamic virtual model using the direction vector, the updated model state is obtained. Then, the trend description of the long short-term memory network is fused to obtain the trajectory prediction output. The trajectory prediction output is compared with the actual coordinate sequence to obtain a stable prediction result. The position of the target object in the scene analysis is updated based on the stable prediction result to obtain the final coordinate distribution.

6. The artificial intelligence video processing method based on digital twins according to claim 1, characterized in that: Based on the predicted trajectory of the target object, deep learning algorithms are applied to extract the behavioral features of the target object. Specifically, this includes: taking the predicted trajectory of the target object as input and processing the trajectory data through a deep learning model; extracting local features through convolutional layers, capturing long-term dependencies in the time series using recurrent layers, and performing nonlinear transformations using activation functions to extract the behavioral features of the target object.

7. The artificial intelligence video processing method based on digital twins according to claim 1, characterized in that: The keyframe weights are enhanced through an attention mechanism to obtain the behavior pattern classification output, specifically including: The extracted behavioral features are mapped to form a feature matrix, the attention weight matrix is ​​initialized, and the attention score for each frame is calculated by combining the dot product attention calculation function with the learnable query vector. Then, the scores are normalized by the Softmax function to make the scores the relative weights of the frames. The original feature matrix is ​​then weighted and fused to highlight key frame information. A classification model is then constructed, and the model is trained with the comprehensive feature vector processed by the attention mechanism and the corresponding behavior pattern label data. After training, new behavioral feature data are processed through the same attention mechanism to obtain a comprehensive feature vector, which is then input into the trained model. The model outputs a probability distribution vector, and the category with the highest probability is selected as the final behavioral pattern classification output.

8. The artificial intelligence video processing method based on digital twins according to claim 1, characterized in that: Based on the final behavioral analysis results, and combined with real-time synchronization capabilities, the processing strategy for multi-source video data is updated. Online learning methods are used to optimize and integrate technical parameters, yielding the feature representation input for the next cycle, specifically including: By combining behavioral analysis results with real-time synchronization technology to process multi-source video data, an adjusted processing strategy is obtained. An online learning method is then used to optimize and integrate parameters for the adjusted processing strategy, resulting in an optimized parameter set. Video data is processed by an optimized set of parameters to obtain a preliminary distribution of feature representations. When the deviation between the preliminary distribution of feature representations and the data obtained by the synchronization technique exceeds a preset threshold, an adjustment method is used to iteratively optimize the set of parameters to obtain a stable feature representation. By combining stable feature representations with a periodic input update dynamic update strategy, feature data for the next period is obtained. The dynamic update strategy is used to process multi-source video data, determine the trend changes in behavior analysis, and obtain the final analysis output. By combining the final analysis output with real-time synchronization technology to adjust the processing strategy, an updated video data processing workflow is obtained.

Images