Video image data processing method and system based on time sequence large model

Abstract

The invention provides a video image data processing method and system based on a time sequence large model, and relates to the technical field of video image data processing, and the method comprises the steps: collecting video image data, and carrying out the preprocessing of the video image data; performing key information capture on the preprocessed video data to obtain a key time point video frame of the video data, and performing interpolation operation on the key time point video frame to obtain an interpolated feature value; historical information is retrieved from the global memory pool, the interpolated feature values are fused, feature transformation operation is performed on the fused features, and transformed features are obtained; and inputting the transformed features into a video prediction model, and predicting the central position of the target area at the future moment.

Description

Technical Field

[0001] The present invention proposes a video image data processing method and system based on a time series large model, and relates to the technical field of video image data processing. Background Art

[0002] In today's digital age, video data has been widely used in many fields such as security monitoring, autonomous driving, and smart media. These scenarios place increasingly high demands on the analysis and understanding of video data, requiring accurate extraction of key information from videos, such as the location and behavior of objects.

[0003] Video data has distinct temporal characteristics, with temporal continuity and correlation between each frame. Traditional image processing methods often focus solely on the information of a single frame and fail to fully exploit the temporal nature of video data. Therefore, a technology that can effectively process temporal data is needed to mine dynamic information from videos that changes over time.

[0004] In recent years, large models have achieved tremendous success in fields such as natural language processing and computer vision. These models possess powerful feature representation learning capabilities, enabling them to automatically learn complex patterns and features from massive amounts of data. Introducing large models into the field of video and image data processing can leverage their powerful learning capabilities to handle the complexity and diversity of video data.

[0005] However, the following technical problems still exist in the prior art: 1. The challenge of extracting dynamic spatiotemporal features: Objects in video data constantly move in space and change over time. Effectively extracting these dynamic spatiotemporal features is a key challenge. Traditional methods are unable to effectively capture the dynamic changes of objects in videos and the spatiotemporal dependencies between different frames, resulting in an inaccurate and incomplete understanding of the video content.

[0006] 2. Inaccurate capture of key information: Video data often contains a large amount of redundant information, making it difficult to accurately capture key information. Failure to accurately capture key information can lead to deviations in subsequent analysis and prediction tasks, such as missing important targets in object detection tasks or making incorrect judgments in behavior analysis.

[0007] 3. Lack of historical information utilization: Existing processing methods fail to fully utilize the historical information of video data, resulting in a lack of contextual support for the model's analysis of the current frame. In many real-world applications, historical information is crucial for understanding current events and predicting future trends. For example, historical information such as a vehicle's previous travel trajectory is crucial for predicting its future location and direction.

[0008] 4. Inappropriate feature fusion and transformation: Even if features from different sources (such as current frame features and historical information features) can be obtained, how to effectively fuse them and then appropriately transform them to meet the needs of the prediction model remains an unresolved issue. Inappropriate feature fusion and transformation can lead to information loss or introduce noise, affecting the performance of the final prediction model. Summary of the Invention

[0009] In order to solve the above technical problems, the present invention proposes a video image data processing method based on a time series large model, comprising the following steps: Step S1, collecting video image data and preprocessing the video image data; Step S2: capturing key information of the pre-processed video data to obtain video frames at key time points of the video data, and performing interpolation operations on the video frames at the key time points to obtain interpolated feature values; Step S3: retrieve historical information from the global memory pool and fuse the interpolated feature values, perform feature transformation on the fused features to obtain transformed features; Step S4: Input the transformed features into the video prediction model to predict the center position of the target area at the future moment.

[0010] In a preferred embodiment, step S2 includes: S21, extracting key time point video frames of the video data; S22. Calculate the interpolated feature values ​​of the video frame at the key time point.

[0011] In a preferred embodiment, let the key time point video frame X S The characteristic value of each pixel at coordinate (i, j) is X ZS (i, j), use the interpolation method to calculate the interpolated eigenvalues ​​from adjacent frame images : ; f def Let be the interpolation function, , , for the key time point video frame X S , sampling at non-integer coordinates (u, v), the interpolation function is: ; where w mn is the weight coefficient calculated based on the decimal part of u and v, and are the floor values ​​of u and v respectively, and m and n are the index variables for summation.

[0012] In a preferred embodiment, in step S3, the interpolated eigenvalues Perform splicing operation with the retrieved historical information M: , Among them, Concat represents the concatenation operation; Perform feature transformation on the fused feature F to obtain the transformed feature F t : .

[0013] In a preferred embodiment, in step S4, the transformed feature F t The input is sent to the trained target detection model, which outputs the bounding box information of the detected target area. For two-dimensional images, the bounding box information is expressed in the upper left corner coordinates (X min ,Y min ) and the lower right corner coordinate (X max ,Y max ) is expressed as follows: the center position of the target area is obtained as the current position of the target area : ; .

[0014] In a preferred embodiment, the transformed feature F t Represented as a column vector , the transformed feature F t After inputting the neural network, the adjustment amount is obtained and : ; Among them, B a is the bias vector, Q a is the weight matrix; Expressed as: ; ,get ; make: ; ; in, is the weight matrix Q a The j-th matrix element of the first row of is the weight matrix Q a The j-th matrix element of the second row of is the transformed feature F tis represented as the jth eigenvector after the column vector.

[0015] In a preferred embodiment, the center position of the target area predicted at the next moment is for: ; .

[0016] In a preferred embodiment, in step S21, the video frame X with dimensions T×H×W×C is input into the dimension expansion layer to obtain a feature map F with dimensions T×H×W×C′. Z , for the feature map F Z Perform dimension averaging to obtain the averaging result F with dimension T×1×1×C′ avg ,T is the number of time frames, H is the height of the video frame, W is the width of the video frame, C is the number of channels, and C′ is the number of intermediate channels; The mean result F is identified by the data sequence recognition mechanism. avg Processing is performed to obtain the time focus weight S, which is multiplied by the video frame X by element-by-element multiplication to obtain the video frame at the key time point: ; Represents element-wise multiplication.

[0017] The present invention also proposes a video image data processing system based on a time series large model, which is used to implement the above-mentioned video image data processing method based on a time series large model, including: a data acquisition and preprocessing module, a key information capture module, an information fusion and transformation module, and a video prediction module; The data acquisition and preprocessing module is used to acquire video image data and preprocess the video image data; The key information capture module is used to capture key information of the pre-processed video data, obtain video frames at key time points of the video data, perform interpolation operations on the video frames at key time points, and obtain interpolated feature values; The information fusion and transformation module is used to retrieve historical information from the global memory pool and fuse the interpolated feature values, perform feature transformation operations on the fused features, and obtain transformed features; The video prediction module is used to input the transformed features into the video prediction model to predict the center position of the target area at a future moment.

[0018] Compared with the prior art, the present invention has the following beneficial technical effects: 1. Effectively extract dynamic spatiotemporal features and key information: Dynamic temporal feature extraction and key information capture from pre-processed video data fully exploits the dynamic temporal characteristics of video data and accurately captures key information. This allows the resulting feature representation to more comprehensively and accurately reflect the video content, providing a high-quality data foundation for subsequent processing and analysis. In video surveillance scenarios, this can effectively capture key information such as the target object's motion trajectory and behavioral changes.

[0019] 2. Fusion of historical information improves feature representation capabilities: By concatenating interpolated feature values ​​with retrieved historical information, the features of the current video frame are combined with historical information, leveraging the relevance of information in time series. This fusion approach enables the model to better understand the context of video content, further enrich feature representation, and enhance the model's ability to handle complex scenarios and long-term dependencies. In video behavior analysis, incorporating historical information can more accurately determine whether current behavior is abnormal.

[0020] 3. Optimizing prediction model input to improve prediction accuracy: Linear transformations are performed on the concatenated new feature vectors to adjust the feature dimensions and distribution, making them more consistent with the input requirements of the video prediction model. This improves the accuracy and efficiency of the model's target location predictions. This emphasizes information useful for the prediction task and reduces the interference of redundant information, thereby improving the performance of the entire video prediction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 Schematic diagram of the flow of the video image data processing method based on the time series large model of the present invention.

[0023] Figure 2 Schematic diagram of the process of obtaining the interpolated eigenvalues ​​of the present invention.

[0024] Figure 3 This is a structural diagram of the video image data processing system based on the time series large model of the present invention. DETAILED DESCRIPTION

[0025] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0026] In the drawings of the specific embodiments of the present invention, in order to better and more clearly describe the working principles of the various components in the system, the connection relationship of the various parts in the device is shown, which only clearly distinguishes the relative position relationship between the various components, and does not constitute a limitation on the signal transmission direction, connection sequence and size, dimension and shape of the components or structures.

[0027] Example 1

[0028] like Figure 1 FIG. 1 is a flowchart of a method for processing video image data based on a time series large model according to the present invention. The method for processing video image data based on a time series large model comprises the following steps: S1. Collect video image data and preprocess the video image data.

[0029] S11. Video image data acquisition.

[0030] Use high-resolution cameras (such as 4K / 8K), industrial cameras, or smartphones to ensure stable frame rates and support dynamic scene capture. For special scenarios (such as infrared and depth perception), multimodal sensors (such as Kinect and LiDAR) are required.

[0031] Cover diverse scenarios (such as indoor / outdoor, lighting changes, and occluded environments) according to task requirements; collect images from different perspectives (top-down, level, and multi-angle) and motion modes (constant speed, acceleration, and random motion).

[0032] Multi-camera systems require time synchronization and spatial calibration of the equipment to eliminate lens distortion.

[0033] Extract keyframes or evenly sample from the video to avoid redundant computation. Long videos can be split into segments of equal length (e.g., 16 frames / segment) to adapt to the model input length.

[0034] S12: Preprocess the video image data.

[0035] Normalize pixel values ​​(e.g., [-1, 1]), adjust resolution (e.g., 224x224), and apply spatiotemporal enhancement to improve generalization. It is preferred to use enhancement methods such as random cropping, temporal flipping, and color perturbation.

[0036] S13. Data time series annotation and metadata recording.

[0037] Timestamp key events in the video, such as the start / end frames of an action, and use dense annotation for continuous actions, such as once per second.

[0038] Save acquisition parameters: resolution, frame rate, light intensity, device model, etc.; record scene semantic labels such as "driving in rainy days" and "crowded crowds".

[0039] In a preferred embodiment, data quality control is performed.

[0040] Anomaly detection: Automatically filters black and blurry frames, and removes video segments with stuck or dropped frames; manually checks annotation consistency, for example, the annotation overlap rate IoU>0.8.

[0041] Data balancing: statistical category distribution, oversampling or synthesis of long-tail data.

[0042] S2. Capture key information of the preprocessed video data to obtain video frames at key time points of the video data, perform interpolation operations on the video frames at key time points, and obtain interpolated feature values.

[0043] like Figure 2 As shown, the specific steps include: S21, extracting the key time point video frame X of the video data S : The video frame of the input video data has the dimensions T × H × W × C, where T is the number of time frames, H is the height of the video frame, W is the width of the video frame, and C is the number of channels. Each pixel value in the video frame is a real number and is stored in the real number set R.

[0044] S211: Generate feature map.

[0045] The video frame X is input through the dimension expansion layer to obtain the feature map F with the dimension of T×H×W×C′ Z , where C′ is the number of intermediate channels.

[0046] F Z =Conv2D(X;W conv ) W conv is the dimension expansion kernel parameter, feature map F Z The original spatial dimension H×W is retained, but the number of channels becomes C′.

[0047] The function of the dimension expansion layer is to extract features from the video frames of the video data and obtain a more representative feature map F Z .

[0048] S212, dimensional averaging.

[0049] For the feature map F Z Perform dimension averaging to obtain the averaging result F with dimension T×1×1×C′ avg : For the feature map F Z For each time step t and channel C′, calculate the averaged result F avg : ; The purpose of dimensional averaging is to compress the information in the time dimension and retain only the global information in the channel dimension, which can reduce the amount of calculation and highlight the importance of each channel.

[0050] S213. Generate a time focus weight through a data sequence recognition mechanism.

[0051] The C′-dimensional vector is mapped to H×W dimensions through a neural network and the shape is adjusted through the Reshape function operation: ; Among them, W1 and W2 are weights, It is an activation function. The function of the Reshape function is to reshape the output flattened vector into a shape that matches the spatial dimension of the original video frame. b1 and b2 are constant terms.

[0052] The result of dimension averaging is processed through the data sequence recognition mechanism to obtain the time focus weight S, which is calculated as follows: ; in is the sigmoid activation function.

[0053] The sigmoid function maps the output value to the interval [0, 1] so that the weight of each position represents the importance of that position in space.

[0054] S214: Focus on the video frame at the key time point.

[0055] The temporal focus weight S is multiplied by the original video frame X by element-by-element multiplication to obtain the video frame at the key time point: .

[0056] Represents element-wise multiplication.

[0057] In this way, the video frames at key time points are enhanced, while the video frames at non-key time points are relatively weakened.

[0058] S22, calculate the key time point video frame XS Interpolated eigenvalue

[0059] To solve the problem of feature blurring caused by剧烈运动. The displacement information of the object is obtained through the fuzzy value estimation method, and then the attention area is dynamically adjusted according to the displacement information, so that the model can better focus on the important parts in the video and maintain the accuracy of features even when the object is剧烈运动. This method can effectively improve the performance of video analysis tasks, especially when dealing with moving objects.

[0060] S221. Obtain the central displacement information of the target area using the fuzzy value estimation method.

[0061] The fuzzy value estimation method is a method for calculating the movement of pixel points in an image. The movement of the target area is described by calculating the displacement of pixels between adjacent frames. In the embodiment, the displacement information of the center of the target area in the video frame is obtained using the fuzzy value estimation method.

[0062] Use (△p x (i,j), △p Y (i,j)) to represent the displacement vector of the pixel point at the center position coordinates (i,j) of the image frame from the (t - 1)-th frame to the t-th frame, and these displacement vectors will be used to determine the fuzzy value later.

[0063] S222. Determine the fuzzy value based on the displacement vector obtained by the fuzzy value estimation method.

[0064] For the feature of the pixel point at each coordinate (i,j), its fuzzy value △p(i,j) can be adjusted according to the displacement vector: △p(i,j)=(△p x (i,j), △p Y (i,j)).

[0065] S223. Perform interpolation operation based on the fuzzy value to obtain the interpolated eigenvalue.

[0066] Let the video frame X at the key time point S be a two-dimensional feature map with a size of H×W (H is the height, W is the width). The eigenvalue of the pixel point at each coordinate (i,j) in the video frame X at the key time point S is X ZS (i,j). Calculate the interpolated eigenvalue from the adjacent frame images through the interpolation method .

[0067] Calculate the interpolated eigenvalue from the adjacent frame images through the interpolation method.

[0068] Let the interpolated eigenvalue be , then: ; f def Let be the interpolation function, , , that is, for the key time point video frame X S , sampling at non-integer coordinates (u, v), the interpolation function is: ; where w mn is the weight coefficient calculated based on the decimal part of u and v, and The formula is obtained by rounding down u and v. ZS In and In a certain neighborhood based on the benchmark, according to the weight coefficient w mn Perform weighted summation to calculate f def The value of .

[0069] m and n are index variables for summation, and their values ​​range from 0 to 1. When calculating the interpolation at non-integer coordinates (u, v), double summation of m and n in the range of 0 to 1 is performed, combined with the weight coefficient w mn And after the coordinates are rounded down The final interpolation result is obtained by using the video frame value at . m and n are used to traverse the items related to the surrounding pixels involved in the interpolation calculation.

[0070] After the above interpolation operation, the interpolated eigenvalue is obtained , 0≤i <H,0≤j<W。

[0071] Through the above steps, the interpolated eigenvalues ​​are calculated. ,Therefore ,the feature blur problem caused by the violent motion of the target is solved.

[0072] S3. Retrieve historical information from the global memory pool and fuse the interpolated feature values, perform feature transformation on the fused features, and obtain the transformed features.

[0073] S31. Obtain historical information M from a global memory pool storing historical information.

[0074] The global memory pool here can be understood as a storage structure that stores information such as features related to past video frames for subsequent retrieval.

[0075] The global memory pool is a storage structure that stores historical information such as features related to past video frames. The historical information M is obtained through the following steps: Locate the storage location and clarify the storage location of the global memory pool in the system, such as a specific area in the computer memory, a table in the database, a specific storage module, or a cache area specially opened in the memory for storing historical features in the deep learning video processing framework.

[0076] Set search conditions and search rules based on current needs. To obtain historical information within a specific time period, you need to determine it based on the timestamp of the video frame; to obtain historical information with a scene similar to the current frame, you need to search based on the similarity of scene features.

[0077] Execute a search operation, executing a search instruction in the storage location of the global memory pool according to the set search conditions. Preferably, a SELECT statement is executed in the database to filter out historical information M that meets the conditions from the storage structure. In the deep learning framework, the corresponding function or method is called to implement the search.

[0078] Extract information and extract the retrieved historical information M from the storage location so that it can be used for subsequent fusion operation with the final feature.

[0079] S32. Obtain the interpolated eigenvalue at the current moment , the interpolated eigenvalues The splicing operation is performed with the retrieved historical information M, and the formula is expressed as: , Concat represents the concatenation operation, which connects two features according to a certain dimension to form a new feature vector.

[0080] Among them, let the interpolated eigenvalue It is a feature vector with a dimension of d1, and the retrieved historical information M is a feature vector with a dimension of d2. After the fusion and splicing operation, the dimension of the fused feature F is d3=d1+d2.

[0081] Perform feature transformation on the fused feature F, and the transformed feature F t for: ; The transformed feature F t The dimension is also d3.

[0082] S4. Input the transformed features into the video prediction model to predict the center position of the target area at the future moment.

[0083] First, the transformed feature F t Calculate the center position of the current target area of ​​the object as : The transformed feature Ft The input is sent to the trained target detection model, which outputs the bounding box information of the detected target area. For a two-dimensional image, the bounding box is represented by the upper left corner coordinate (X min ,Y min ) and the lower right corner coordinate (X max ,Y max ) is expressed as follows: the center position of the target area is obtained as the current position of the target area : ; ; Then, the transformed feature F t Represented as a column vector , The transformed feature F t After inputting the neural network, the adjustment amount is obtained and :

[0084] Among them, B a is the bias vector, Q a is the weight matrix; Expressed as: ; ,get ; make: ; ; in, is the weight matrix Q a The j-th matrix element of the first row of is the weight matrix Q a The j-th matrix element of the second row of is the transformed feature F t is represented as the jth eigenvector after the column vector.

[0085] Then DX and DY represent the moving values ​​of the predicted center position output by the fully connected neural network layer.

[0086] The predicted value of the center position of the target area at the next moment is: ; ; Through the above steps, the center movement value of the target area predicted by the fully connected neural network is combined with the center position of the target area at the previous moment to predict the center position of the target area at the next moment.

[0087] Example 2

[0088] like Figure 3 As shown in FIG, a structural diagram of a video data processing system for implementing a video image data processing method based on a time series large model according to the present invention, the video data processing system includes: a data acquisition and preprocessing module, a key information capture module, an information fusion and transformation module, and a video prediction module.

[0089] The data acquisition and preprocessing module is used to acquire video image data and preprocess the video image data; The key information capture module is used to capture key information of the pre-processed video data, obtain the video frames of the key time points of the video data, perform interpolation operations on the video frames of the key time points, and obtain the interpolated feature values; The information fusion and transformation module is used to retrieve historical information from the global memory pool and fuse the interpolated feature values, perform feature transformation operations on the fused features, and obtain the transformed features; The video prediction module is used to input the transformed features into the video prediction model to predict the center position of the target area at the future moment.

[0090] In a preferred embodiment, the data acquisition and preprocessing module includes a video image acquisition unit and a preprocessing unit. The video image acquisition unit is responsible for acquiring video image data from various video sources (such as cameras, video files, etc.). The video image acquisition unit includes a hardware interface adapter module for connecting to different types of video input devices to ensure stable data acquisition. The preprocessing unit performs preprocessing on the acquired video image data, including image denoising to remove noise introduced during the acquisition process; size normalization to resize images of different resolutions to a uniform size suitable for subsequent processing; and color space conversion to convert the image data into a suitable color space representation.

[0091] The key information capture module captures key information of the preprocessed video data to obtain video frames at key time points of the video data, and performs interpolation operations on the video frames at key time points to obtain interpolated feature values.

[0092] The information fusion and transformation module consists of a storage unit, a fusion unit, and a feature transformation unit. The storage unit stores historical information, such as key features of previously processed video data. The storage unit utilizes an efficient retrieval mechanism to quickly retrieve relevant historical information based on current needs. The fusion unit fuses the historical information retrieved from the storage unit with the interpolated feature values ​​obtained from the key information capture module and performs an interpolation operation. This fusion method can be simple feature concatenation or weighted fusion to highlight the importance of different features. The feature transformation unit performs feature transformation operations on the fused features to make them more suitable for subsequent prediction tasks.

[0093] The video prediction module predicts the location information of the center position of the target area in the video by learning and analyzing the input features. The prediction result can be the coordinates of the center position of the target area or other representations related to the center position of the target area, such as the position and size of the bounding box.

[0094] In a preferred embodiment, the video data processing system of the present invention's method for processing video image data based on a time-series large model also includes a system control and coordination module responsible for controlling the flow of data between the various modules of the system, ensuring that data is transmitted and processed sequentially between them. For example, after data acquisition is completed, the preprocessing module is triggered to begin operation; after preprocessing is completed, the key information capture module is activated, etc. Simultaneously, the system manages parameters related to each module, such as the frequency of data acquisition, parameters of preprocessing operations, and hyperparameters of feature extraction and prediction models, and adjusts and optimizes these parameters based on actual application scenarios and requirements.

[0095] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0096] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above-mentioned method embodiments when executed by a processor.

[0097] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of each of the above-described method embodiments.

[0098] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. The database involved in the embodiments provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited to this. The processor involved in the embodiments provided in this application may be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, etc., but is not limited to this.

[0099] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0100] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A video image data processing method based on a time series large model, characterized in that: The steps include: Step S1, collecting video image data and preprocessing the video image data; Step S2, capturing key information of the preprocessed video data to obtain video frames at key time points of the video data, and performing interpolation operations on the video frames at key time points to obtain interpolated feature values; Step S3, retrieving historical information from the global memory pool and fusing the interpolated feature values, performing feature transformation operations on the fused features to obtain transformed features; Step S4: input the transformed features into the video prediction model to predict the center position of the target area at the future moment.

2. The video image data processing method based on the time series large model according to claim 1 is characterized in that: The step S2 comprises: S21, extracting the key time point video frames of the video data; S22: Calculate the interpolated feature value of the video frame at the key time point.

3. The video image data processing method based on the time series large model according to claim 2 is characterized in that: In step S22, the video frame X at the key time point is assumed to be S The characteristic value of each pixel at coordinate (i, j) is X ZS (i, j), use the interpolation method to calculate the interpolated eigenvalues ​​from adjacent frame images : ； Among them, f def is the interpolation function, let , , for the key time point video frame X S , sampling at non-integer coordinates (u, v), the interpolation function is: ； Among them, w mn is the weight coefficient calculated based on the decimal part of u and v, and are the floor values ​​of u and v respectively, and m and n are the index variables for summation.

4. The video image data processing method based on the time series large model according to claim 3 is characterized in that: In step S3, the interpolated eigenvalues Perform splicing operation with the retrieved historical information M: , Among them, Concat represents the concatenation operation; Perform feature transformation on the fused feature F to obtain the transformed feature F t : 。 5. The video image data processing method based on the time series large model according to claim 4 is characterized in that: In step S4, the transformed feature F t The target detection model outputs the bounding box information of the detected target area. For a two-dimensional image, the bounding box information is expressed in the upper left corner coordinates (X min ,Y min ) and the lower right corner coordinate (X max ,Y max ) represents, the current position of the center of the target area is obtained by the following formula : ； 。 6. The video image data processing method based on the time series large model according to claim 5 is characterized in that: The transformed feature F t Represented as a column vector , the dimension of the column vector is d 3， The transformed feature F t After inputting into the neural network, the adjustment amount is obtained and : ； Among them, B a is the bias vector, Q a is the weight matrix; It is expressed as: ； It is expressed as: ; make: ; ； in, is the weight matrix Q a The j-th matrix element of the first row of is the weight matrix Q a The j-th matrix element of the second row of is the transformed feature F t The jth eigenvector after being expressed as a column vector, B a,x is the x-component of the bias vector, B a,y is the y component of the bias vector.

7. The video image data processing method based on the time series large model according to claim 6 is characterized in that: The center position of the target area predicted at the next moment t for: ； 。 8. The video image data processing method based on the time series large model according to claim 2 is characterized in that: In step S21, the video frame X with dimensions T×H×W×C is input into the dimension expansion layer to obtain a feature map F with dimensions T×H×W×C′. Z , for the feature map F Z Perform dimension averaging to obtain the averaging result F with dimension T×1×1×C′ avg ,T is the number of time frames, H is the height of the video frame, W is the width of the video frame, C is the number of channels, and C′ is the number of intermediate channels; The average result F is obtained by using the data sequence recognition mechanism avg Processing is performed to obtain the time focus weight S, and the time focus weight S is multiplied by the video frame X by element-by-element multiplication to obtain the video frame X at the key time point S : ； Represents element-wise multiplication.

9. A video image data processing system based on a time series large model, used to implement the video image data processing method based on a time series large model according to any one of claims 1 to 8, characterized in that: include: Data acquisition and preprocessing module, key information capture module, information fusion and transformation module, video prediction module; The data acquisition and preprocessing module is used to acquire video image data and preprocess the video image data; The key information capture module is used to capture key information of the preprocessed video data, obtain video frames at key time points of the video data, perform interpolation operations on the video frames at key time points, and obtain interpolated feature values; The information fusion and transformation module is used to retrieve historical information from the global memory pool and fuse the interpolated feature values, perform feature transformation operations on the fused features, and obtain transformed features; The video prediction module is used to input the transformed features into the video prediction model to predict the center position of the target area at a future moment.

Images