Data processing method and system based on artificial intelligence
By employing AI-based data processing methods and utilizing a dual-branch feature extraction and feature fusion model, the problem of low accuracy and efficiency in traditional video data processing has been solved, achieving a balance between efficient identification and real-time performance for massive amounts of video data.
Patent Information
- Application Number
- CN202511665813.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-13
- Publication Date
- 2026-02-10
AI Technical Summary
Traditional data processing techniques struggle to effectively handle massive amounts of video data, especially when identifying target behavior, where accuracy and efficiency are low, and computational resource requirements are high, making it difficult to balance real-time performance and accuracy.
An AI-based data processing method is adopted, including preprocessing, spatiotemporal feature extraction and recognition steps. It utilizes a dual-branch feature extraction model and a feature fusion model, and extracts the spatial and temporal features of video frames through a two-dimensional neural network and a dynamic temporal model. The target behavior is then identified by combining linear transformation and probabilistic transformation.
It enables accurate and rapid processing of massive amounts of video data, maintaining a balance between real-time performance and accuracy, and improving the efficiency and accuracy of video data processing.
Smart Images
Figure CN121509764A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a data processing method and system based on artificial intelligence. Background Technology
[0002] In the digital age, video data from various video software backends (such as video social media and autonomous driving) is growing exponentially. Video data is characterized by large volume, high information redundancy, and complex spatiotemporal dimensions. Traditional data processing technologies face enormous challenges when performing target behavior recognition on this explosively growing video data.
[0003] Traditional data processing techniques rely on manually designed features and rules. These features and rules can only cope with specific situations. Once the video data changes slightly, the manually designed features and rules can become ineffective. Moreover, video data is unstructured data with chaotic data formats and no unified "fields" or "labels". Therefore, using traditional data processing techniques to process unstructured video data to identify target behaviors will result in problems such as difficulty in feature extraction and difficulty in adapting to dynamic changes in the distribution of video data. This leads to low accuracy and low efficiency in processing video data to identify target behaviors.
[0004] Meanwhile, the processing of massive amounts of video data places extremely high demands on computing resources. Traditional data processing technologies struggle to balance real-time performance and accuracy, failing to meet the needs of many scenarios (such as financial risk control, medical diagnosis, and intelligent recommendation) for in-depth processing and analysis of video data.
[0005] Therefore, how to accurately and quickly process massive amounts of video data to identify target behaviors, while maintaining a balance between real-time performance and accuracy in video data processing, is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0006] This application provides an artificial intelligence-based data processing method and system to accurately and quickly process massive amounts of video data, thereby identifying target behaviors and maintaining a balance between real-time performance and accuracy in video data processing.
[0007] To solve the above-mentioned technical problems, this application provides the following technical solution:
[0008] An artificial intelligence-based data processing method includes the following steps: Step T10: Acquire a real-time video stream and preprocess the acquired real-time video stream; Step T20: Extract spatiotemporal features from the preprocessed video stream and form a spatiotemporal feature sequence from the extracted spatiotemporal feature set; Step T30: Recognize the spatiotemporal feature sequence to obtain the target behavior.
[0009] The data processing method based on artificial intelligence described above preferably includes preprocessing, which comprises the following sub-steps: sampling a video stream to obtain a video frame sequence; inputting the video frames in the video frame sequence into a video frame denoising processing model to perform filtering and denoising processing on the video frames; inputting the filtered and denoised video frames with missing segments into a missing segment filling model to generate substitute segments to repair the video frames; scaling the resolution of the repaired video frames to a predetermined size, and completing the preprocessing of the video frame sequence after scaling all video frames in the video frame sequence.
[0010] The data processing method based on artificial intelligence described above preferably includes spatiotemporal feature extraction, comprising the following sub-steps: inputting a video frame sequence into a static spatial feature extraction model of a dual-branch feature extraction model to extract spatial features from each video frame of the video frame sequence, and forming a spatial feature sequence from the extracted spatial feature set; inputting the spatial feature sequence into a dynamic temporal feature extraction model of the dual-branch feature extraction model to extract temporal features from the spatial feature sequence, and forming a temporal feature sequence from the extracted temporal feature set; and inputting the spatial feature sequence and the temporal feature sequence into a feature fusion model for fusion to obtain spatiotemporal features.
[0011] The artificial intelligence-based data processing method described above preferably includes the following sub-steps for identifying spatiotemporal feature sequences: converting the spatiotemporal feature sequence corresponding to the video stream into a global feature vector that can represent the video stream; performing a linear transformation on the global feature vector representing the entire video stream to obtain the log probability vector of the entire video stream for different behavior categories; converting the log probability vector of the entire video stream for different behavior categories into probabilities, and taking the behavior category corresponding to the highest probability as the target behavior.
[0012] The data processing method based on artificial intelligence described above preferably involves transforming the global feature vector, which includes the following sub-steps: matching recognition weights to each spatiotemporal feature in the spatiotemporal feature sequence corresponding to the video stream, and collecting all recognition weights to obtain a recognition weight sequence; and weighting and summing the recognition weights in the recognition weight sequence and the corresponding spatiotemporal features in the spatiotemporal feature sequence to obtain a global feature vector that can represent the video stream.
[0013] An artificial intelligence-based data processing system includes: a preprocessing unit, a feature extraction unit, and a recognition unit; the preprocessing unit acquires a real-time video stream and preprocesses the acquired real-time video stream; the feature extraction unit extracts spatiotemporal features from the preprocessed video stream and forms a spatiotemporal feature sequence from the extracted spatiotemporal feature set; the recognition unit recognizes the spatiotemporal feature sequence to obtain the target behavior.
[0014] In the artificial intelligence-based data processing system described above, preferably, the preprocessing unit includes: a sampling unit, a filtering and noise reduction unit, a repair unit, and a scaling unit; the sampling unit samples the video stream to obtain a video frame sequence; the filtering and noise reduction unit inputs the video frames in the video frame sequence into a video frame noise reduction processing model to perform filtering and noise reduction processing on the video frames; the repair unit inputs the video frames that have been filtered and noise-reduced and have missing segments into a missing segment filling model to generate substitute segments to repair the video frames; the scaling unit scales the resolution of the repaired video frames to a predetermined size, and after scaling all video frames in the video frame sequence, the preprocessing of the video frame sequence is completed.
[0015] In the artificial intelligence-based data processing system described above, preferably, the feature extraction unit inputs the video frame sequence into the static spatial feature extraction model of the dual-branch feature extraction model, extracts spatial features from each video frame of the video frame sequence, and forms a spatial feature sequence from the extracted spatial feature set; the feature extraction unit inputs the spatial feature sequence into the dynamic temporal feature extraction model of the dual-branch feature extraction model, extracts temporal features from the spatial feature sequence, and forms a temporal feature sequence from the extracted temporal feature set; the feature extraction unit inputs the spatial feature sequence and the temporal feature sequence into the feature fusion model for fusion to obtain spatiotemporal features.
[0016] In the artificial intelligence-based data processing system described above, preferably, the recognition unit includes: a feature vector conversion unit, a log-odds vector transformation unit, and a probability transformation and judgment unit; the feature vector conversion unit converts the spatiotemporal feature sequence corresponding to the video stream into a global feature vector that can represent the video stream; the log-odds vector transformation unit performs a linear transformation on the global feature vector representing the entire video stream to obtain the log-odds vector of the entire video stream for different behavior categories; the probability transformation and judgment unit converts the log-odds vector of the entire video stream for different behavior categories into probabilities, and takes the behavior category corresponding to the highest probability as the target behavior.
[0017] In the artificial intelligence-based data processing system described above, preferably, the feature vector conversion unit matches recognition weights to each spatiotemporal feature in the spatiotemporal feature sequence corresponding to the video stream, and sets all recognition weights to obtain a recognition weight sequence; the feature vector conversion unit performs a weighted summation of the recognition weights in the recognition weight sequence and the corresponding spatiotemporal features in the spatiotemporal feature sequence to obtain a global feature vector that can represent the video stream.
[0018] Compared to the aforementioned background technologies, the artificial intelligence-based data processing method and system provided in this application can accurately and quickly process massive amounts of video data, thereby identifying target behaviors and maintaining a balance between real-time performance and accuracy in video data processing. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.
[0020] Figure 1 This is a flowchart of the data processing method based on artificial intelligence provided in the embodiments of this application; Figure 2 This is a flowchart of the identification of spatiotemporal feature sequences provided in the embodiments of this application; Figure 3 This is a schematic diagram of an artificial intelligence-based data processing system provided in an embodiment of this application. Detailed Implementation
[0021] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0022] Example 1 like Figure 1 As shown, this application provides a data processing method based on artificial intelligence, including the following steps: Step T10: Acquire the real-time video stream and preprocess the acquired real-time video stream; A real-time video stream is acquired, and video frames are sampled at a rate of 1 frame per predetermined interval (e.g., per second) to obtain a sequence of video frames. A video frame noise reduction model runs in the background of the video software. The video frames in the sampled sequence are sequentially input into the video frame noise reduction model to perform filtering and noise reduction processing on each video frame, thereby reducing random noise in each video frame, enhancing the overall consistency of each video frame, and making each video frame smoother.
[0023] Specifically, the video frame noise reduction processing model is as follows: ; in, Pixels in a video frame Grayscale value after filtering and noise reduction processing; Pixels in a video frame The original grayscale value, Pixels in a video frame The original grayscale value; Is the filter window in The weight value of the location; It is the size of the filter window. It is the length of the filtering window. It is the width of the filtering window; It is the sum of all weights in the filter window, used to avoid images being too bright or too dark.
[0024] After filtering and denoising the video frames, the missing segments of the filtered and denoised video frames are identified. The video frames with missing segments are then input into the missing segment filling model running in the background of the video software. The missing segment filling model generates a substitute segment for the video frame and then adds the substitute segment to the position of the missing segment in the video frame to repair the video frame and form a complete video frame.
[0025] Specifically, the missing fragment imputation model is as follows: ; in, It is the complete version after repair. One video frame; It is the first The segment in a video frame that precedes the missing segment; It is the first The segment located after the missing segment in a video frame; This is the first A replacement clip generated from a video frame; It is an adjustable factor. , The transition from 1 to 0 achieves smooth splicing of overlapping areas.
[0026] The resolution of the complete video frame is scaled to a predetermined size, such as 224x224, to ensure that subsequent processing can be performed under the same conditions, avoid errors or deviations caused by different resolutions, and scale the resolution to speed up subsequent processing and make it more efficient.
[0027] Each video frame in the video frame sequence undergoes filtering and noise reduction, missing segment identification, missing segment filling, and resolution scaling according to the above processing procedure, thereby achieving preprocessing of the video frame sequence and thus preprocessing of the corresponding video stream. The preprocessed video stream can then be processed in the next step.
[0028] Step T20: Extract spatiotemporal features from the preprocessed video stream and form a spatiotemporal feature sequence from the extracted spatiotemporal feature set; A dual-branch feature extraction model runs in the background of the video software. After preprocessing the video frame sequence of the video stream, the preprocessed video frame sequence is... The input to the dual-branch feature extraction model is the preprocessed video frame sequence. , It is a video frame sequence The first video frame in It is a video frame sequence The second video frame in It is a video frame sequence The first in The spatial feature sequence and temporal feature sequence are extracted from the video frames.
[0029] The dual-branch feature extraction model comprises a first branch and a second branch. The first branch is a static spatial feature extraction sub-model, used to extract spatial features from each video frame and form a spatial feature sequence from the extracted spatial feature set. The second branch is a dynamic temporal feature extraction model, used to extract temporal features from the spatial feature sequence and form a temporal feature sequence from the extracted temporal feature set. The preprocessed video frame sequence... Input static spatial feature extraction sub-model, from video frame sequence Spatial features are extracted from each video frame, and the extracted spatial feature set is formed into a spatial feature sequence. The spatial feature sequence is input into a dynamic temporal feature extraction model to extract temporal features from the spatial feature sequence, and the extracted temporal feature set is formed into a temporal feature sequence.
[0030] Specifically, the static spatial feature extraction sub-model is as follows: ; in, It is a two-dimensional neural network model used to process two-dimensional data; It is a video frame sequence The first in One video frame; It is a video frame Extracted spatial features; video frame sequence The spatial feature sequence is obtained from the set of spatial features of all video frames. , It is a video frame The extracted spatial features, It is a video frame The extracted spatial features, It is a video frame The extracted spatial features.
[0031] This application utilizes a two-dimensional neural network model. Extracting video frame sequences The spatial characteristics of video frames in the model are processed more efficiently than the 3D neural network models used for processing 3D data. This allows for accurate and rapid processing of massive amounts of video data, thus maintaining a balance between real-time performance and accuracy in video data processing.
[0032] Specifically, the dynamic temporal feature extraction model is as follows: ; Formula (1) is a short-term time series model, which can capture short-term target behavior information between adjacent video frames (e.g., small joint movements, instantaneous velocity of objects). It is a one-dimensional convolution kernel in the time dimension, meaning it only slides in the time dimension; It is a one-dimensional convolutional layer; It is a linear rectified function; It is the output of the short-term time series model (1).
[0033] Formula (2) is a long-term time series model, which can be used to understand the evolution of target behavior in the entire video stream; , and It is a learnable projection matrix; It is a video frame sequence The number of video frames; yes The dimension is used for scaling; It is a normalized exponential function; It is a feedforward neural network; Spatial features Extract the obtained temporal features; combine all the obtained temporal features to obtain the temporal feature sequence. , Spatial features The extracted time-series features Spatial features The extracted time-series features Spatial features Extracted temporal features.
[0034] This application first uses a short-term time series model to initially obtain an output containing short-term target behavior information from the spatial feature sequence in the time dimension. Then, the output is analyzed using a long-term time series model. By deeply capturing temporal feature sequences that embody long-term target behavior information across the entire video stream, the evolution of target behavior throughout the video stream can be understood based on these temporal feature sequences. This allows the dynamic temporal feature extraction model to focus on processing the extracted spatial feature sequences, avoiding direct computation of 3D neural network models on the original video stream. This results in more refined temporal modeling with lower computational costs and maintains a balance between real-time performance and accuracy in video data processing.
[0035] A feature fusion model runs in the background of the video software, and spatial feature sequences are obtained according to the above method. and time series feature sequences Then, the spatial feature sequence and time series feature sequences The input feature fusion model is used to fuse the features to obtain spatiotemporal features that take into account both spatial and temporal details.
[0036] Specifically, the feature fusion model is as follows: ; in, This indicates that the concatenation is performed along the feature channel dimension; yes and The spatiotemporal features are obtained by splicing together; the set of all obtained spatiotemporal features is used to obtain the spatiotemporal feature sequence. , yes and The spatiotemporal features obtained by splicing yes and The spatiotemporal features obtained by splicing yes and The spatiotemporal features obtained by splicing.
[0037] This application decouples the complex spatiotemporal and temporal features in video streams, allowing the first and second branches of the dual-branch feature extraction model to perform their respective functions, thereby achieving more efficient and accurate processing of massive video data and maintaining a balance between real-time performance and accuracy in video data processing.
[0038] Step T30: Identify the spatiotemporal feature sequence to obtain the target behavior; like Figure 2 As shown, step T31 involves converting the spatiotemporal feature sequence corresponding to the video stream into a global feature vector that can represent the video stream. After obtaining the spatiotemporal feature sequence corresponding to the video stream: back, For each spatiotemporal feature in the spatiotemporal feature sequence, an identification weight is matched, and the set of all identification weights is obtained as the identification weight sequence. ,in for The corresponding recognition weights for The corresponding recognition weights for The corresponding recognition weights.
[0039] Specifically, in order to calculate the recognition weights, a learnable weight vector is introduced. Based on learning, weight vectors under different target behaviors can be obtained. For example, when the identified target behavior is "shooting", the weight vector... They will learn to focus on "arm raising" and "ball release," and to incorporate spatiotemporal features. With weight vector Performing the dot product yields the corresponding scalar fraction. ,in, , for The transpose of .
[0040] scalar fractions Transform into a probability distribution , It is a natural exponential function, and then based on the probability distribution The calculated spatiotemporal features Matching recognition weights , in, ,and as well as .
[0041] The recognition weights in the recognition weight sequence and the corresponding spatiotemporal features in the spatiotemporal feature sequence are weighted and summed to obtain a global feature vector that can represent the video stream. .
[0042] Step T32: Perform a linear transformation on the global feature vector representing the entire video stream to obtain the log odds vector of the entire video stream for different behavior categories. The global feature vector representing the entire video stream The input is fed into a pre-trained linear transformation model, which performs a linear transformation and outputs the log-odds vectors of the entire video stream for different behavior categories. ,in Let be the log-odds vector of the entire video stream for the first behavior category. Let be the log-odds vector of the entire video stream for the second behavior category. For the entire video stream for the first A log-odds vector of behavioral categories.
[0043] The linear change model is: , For the entire video stream for the first A log-odds vector of behavioral categories. These are the weights from the input layer to the hidden layer in a linear transformation model. The bias of the hidden layer in the linear transformation model. These are the weights from the hidden layer to the output layer of the linear transformation model. The bias of the output layer of the linear transformation model. , , , All of these can be obtained through learning and training. ,when hour, ,when hour, .
[0044] Step T33: Convert the log-probability vectors of the entire video stream for different behavior categories into probabilities, and take the behavior category corresponding to the highest probability as the target behavior, thereby completing the identification of the target behavior; Log-odds vectors of the entire video stream for different behavior categories Convert to probability ,in, Based on the log-odds vector The probability of transformation. Based on the log-odds vector The probability of transformation. Based on the log-odds vector The probability of transformation.
[0045] Specifically, the probability transformation is performed using the following formula: ; in, Based on the log-odds vector The probability of transformation. It is a normalized exponential function.
[0046] The log-odds vector of the entire video stream for different behavior types Convert all to probability Then, from these probabilities Choose the highest probability The corresponding behavior category is used as the target behavior, thereby achieving the purpose of identifying the target behavior from the video stream.
[0047] Example 2 like Figure 3 As shown, this application provides a data processing system 300 based on artificial intelligence, including: a preprocessing unit 310, a feature extraction unit 320, and a recognition unit 330.
[0048] The preprocessing unit 310 acquires the real-time video stream and performs preprocessing on the acquired real-time video stream.
[0049] The preprocessing unit 310 includes a sampling unit 311, a filtering and noise reduction unit 312, a repair unit 313, and a scaling unit 314.
[0050] The sampling unit 311 acquires the real-time video stream and samples video frames at a rate of 1 frame per predetermined interval (e.g., per second) to obtain a video frame sequence of the video stream. A video frame noise reduction processing model runs in the background of the video software. The filtering and noise reduction unit 312 sequentially inputs the video frames from the sampled video frame sequence into the video frame noise reduction processing model to perform filtering and noise reduction processing on each video frame, thereby reducing random noise in each video frame, enhancing the overall consistency of each video frame, and making each video frame smoother.
[0051] Specifically, the video frame noise reduction processing model is as follows: ; in, Pixels in a video frame Grayscale value after filtering and noise reduction processing; Pixels in a video frame The original grayscale value, Pixels in a video frame The original grayscale value; Is the filter window in The weight value of the location; It is the size of the filter window. It is the length of the filtering window. It is the width of the filtering window; It is the sum of all weights in the filter window, used to avoid images being too bright or too dark.
[0052] After the filtering and noise reduction unit 312 performs filtering and noise reduction on the video frame, the repair unit 313 identifies the missing segments of the filtered and noise-reduced video frame, and then inputs the video frame with missing segments into the missing segment filling model running in the background of the video software. The missing segment filling model generates a substitute segment for the video frame, and then adds the substitute segment to the position of the missing segment of the video frame to repair the video frame, thereby forming a complete video frame.
[0053] Specifically, the missing fragment imputation model is as follows: ; in, It is the complete version after repair. One video frame; It is the first The segment in a video frame that precedes the missing segment; It is the first The segment located after the missing segment in a video frame; This is the first A replacement clip generated from a video frame; It is an adjustable factor. , The transition from 1 to 0 achieves smooth splicing of overlapping areas.
[0054] The scaling unit 314 scales the resolution of the complete video frame to a predetermined size, such as 224×224, to ensure that subsequent processing can be performed under the same conditions, avoid errors or deviations caused by different resolutions, and scaling the resolution can speed up the subsequent processing and make it more efficient.
[0055] Each video frame in the video frame sequence undergoes filtering and noise reduction, missing segment identification, missing segment filling, and resolution scaling according to the above processing procedure, thereby achieving preprocessing of the video frame sequence and thus preprocessing of the corresponding video stream. The preprocessed video stream can then be processed in the next step.
[0056] The feature extraction unit 320 extracts spatiotemporal features from the preprocessed video stream and forms a spatiotemporal feature sequence from the extracted spatiotemporal feature set.
[0057] A dual-branch feature extraction model runs in the background of the video software. After the preprocessing unit 310 preprocesses the video frame sequence of the video stream, the feature extraction unit 320 extracts the preprocessed video frame sequence. The input to the dual-branch feature extraction model is the preprocessed video frame sequence. , It is a video frame sequence The first video frame in It is a video frame sequence The second video frame in It is a video frame sequence The first in The spatial feature sequence and temporal feature sequence are extracted from the video frames.
[0058] The dual-branch feature extraction model includes a first branch and a second branch. The first branch is a static spatial feature extraction sub-model, used to extract spatial features from each video frame and form a spatial feature sequence from the extracted spatial feature set. The second branch is a dynamic temporal feature extraction model, used to extract temporal features from the spatial feature sequence and form a temporal feature sequence from the extracted temporal feature set. The feature extraction unit 320 processes the preprocessed video frame sequence... Input static spatial feature extraction sub-model, from video frame sequence Spatial features are extracted within each video frame, and the extracted spatial feature set is formed into a spatial feature sequence. The feature extraction unit 320 inputs the spatial feature sequence into the dynamic temporal feature extraction model, extracts temporal features from the spatial feature sequence, and forms a temporal feature sequence from the extracted temporal feature set.
[0059] Specifically, the static spatial feature extraction sub-model is as follows: ; in, It is a two-dimensional neural network model used to process two-dimensional data; It is a video frame sequence The first in One video frame; It is a video frame Extracted spatial features; video frame sequence The spatial feature sequence is obtained from the set of spatial features of all video frames. , It is a video frame The extracted spatial features, It is a video frame The extracted spatial features, It is a video frame The extracted spatial features.
[0060] This application utilizes a two-dimensional neural network model. Extracting video frame sequences The spatial characteristics of video frames in the model are processed more efficiently than the 3D neural network models used for processing 3D data. This allows for accurate and rapid processing of massive amounts of video data, thus maintaining a balance between real-time performance and accuracy in video data processing.
[0061] Specifically, the dynamic temporal feature extraction model is as follows: ; Formula (1) is a short-term time series model, which can capture short-term target behavior information between adjacent video frames (e.g., small joint movements, instantaneous velocity of objects). It is a one-dimensional convolution kernel in the time dimension, meaning it only slides in the time dimension; It is a one-dimensional convolutional layer; It is a linear rectified function; It is the output of the short-term time series model (1).
[0062] Formula (2) is a long-term time series model, which can be used to understand the evolution of target behavior in the entire video stream; , and It is a learnable projection matrix; It is a video frame sequence The number of video frames; yes The dimension is used for scaling; It is a normalized exponential function; It is a feedforward neural network; Spatial features Extract the obtained temporal features; combine all the obtained temporal features to obtain the temporal feature sequence. , Spatial features The extracted time-series features Spatial features The extracted time-series features Spatial features Extracted temporal features.
[0063] This application first uses a short-term time series model to initially obtain an output containing short-term target behavior information from the spatial feature sequence in the time dimension. Then, the output is analyzed using a long-term time series model. By deeply capturing temporal feature sequences that embody long-term target behavior information across the entire video stream, the evolution of target behavior throughout the video stream can be understood based on these temporal feature sequences. This allows the dynamic temporal feature extraction model to focus on processing the extracted spatial feature sequences, avoiding direct computation of 3D neural network models on the original video stream. This results in more refined temporal modeling with lower computational costs and maintains a balance between real-time performance and accuracy in video data processing.
[0064] A feature fusion model runs in the background of the video software, and spatial feature sequences are obtained according to the above method. and time series feature sequences Then, the feature extraction unit 320 extracts the spatial feature sequence. and time series feature sequences The input feature fusion model is used to fuse the features to obtain spatiotemporal features that take into account both spatial and temporal details.
[0065] Specifically, the feature fusion model is as follows: ; in, This indicates that the concatenation is performed along the feature channel dimension; yes and The spatiotemporal features are obtained by splicing together; the set of all obtained spatiotemporal features is used to obtain the spatiotemporal feature sequence. , yes and The spatiotemporal features obtained by splicing yes and The spatiotemporal features obtained by splicing yes and The spatiotemporal features obtained by splicing.
[0066] This application decouples the complex spatiotemporal and temporal features in video streams, allowing the first and second branches of the dual-branch feature extraction model to perform their respective functions, thereby achieving more efficient and accurate processing of massive video data and maintaining a balance between real-time performance and accuracy in video data processing.
[0067] The recognition unit 330 identifies the spatiotemporal feature sequence to obtain the target behavior.
[0068] The identification unit 330 includes: a feature vector transformation unit 331, a log-odds vector transformation unit 332, and a probability transformation judgment unit 333.
[0069] The feature vector conversion unit 331 converts the spatiotemporal feature sequence corresponding to the video stream into a global feature vector that can represent the video stream.
[0070] After obtaining the spatiotemporal feature sequence corresponding to the video stream: back, For each spatiotemporal feature in the spatiotemporal feature sequence, an identification weight is matched, and the set of all identification weights is obtained as the identification weight sequence. ,in for The corresponding recognition weights for The corresponding recognition weights for The corresponding recognition weights.
[0071] Specifically, in order to calculate the recognition weights, a learnable weight vector is introduced. Based on learning, weight vectors under different target behaviors can be obtained. For example, when the identified target behavior is "shooting", the weight vector... They will learn to focus on "arm raising" and "ball release," and to incorporate spatiotemporal features. With weight vector Performing the dot product yields the corresponding scalar fraction. ,in, , for The transpose of .
[0072] scalar fractions Transform into a probability distribution , It is a natural exponential function, and then based on the probability distribution The calculated spatiotemporal features Matching recognition weights , in, ,and as well as .
[0073] The recognition weights in the recognition weight sequence and the corresponding spatiotemporal features in the spatiotemporal feature sequence are weighted and summed to obtain a global feature vector that can represent the video stream. .
[0074] The log-odds vector transformation unit 332 performs a linear transformation on the global feature vector representing the entire video stream to obtain the log-odds vectors of the entire video stream for different behavior categories.
[0075] The global feature vector representing the entire video stream The input is fed into a pre-trained linear transformation model, which performs a linear transformation and outputs the log-odds vectors of the entire video stream for different behavior categories. ,in Let be the log-odds vector of the entire video stream for the first behavior category. Let be the log-odds vector of the entire video stream for the second behavior category. For the entire video stream for the first A log-odds vector of behavioral categories.
[0076] The linear change model is: , For the entire video stream for the first A log-odds vector of behavioral categories. These are the weights from the input layer to the hidden layer in a linear transformation model. The bias of the hidden layer in the linear transformation model. These are the weights from the hidden layer to the output layer of the linear transformation model. The bias of the output layer of the linear transformation model. , , , All of these can be obtained through learning and training. ,when hour, ,when hour, .
[0077] The probability transformation and judgment unit 333 transforms the log probability vector of the entire video stream for different behavior categories into probabilities, and takes the behavior category corresponding to the highest probability as the target behavior, thereby completing the recognition of the target behavior.
[0078] Log-odds vectors of the entire video stream for different behavior categories Convert to probability ,in, Based on the log-odds vector The probability of transformation. Based on the log-odds vector The probability of transformation. Based on the log-odds vector The probability of transformation.
[0079] Specifically, the probability transformation is performed using the following formula: ; in, Based on the log-odds vector The probability of transformation. It is a normalized exponential function.
[0080] The log-odds vector of the entire video stream for different behavior types Convert all to probability Then, from these probabilities Choose the highest probability The corresponding behavior category is used as the target behavior, thereby achieving the purpose of identifying the target behavior from the video stream.
[0081] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0082] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A data processing method based on artificial intelligence, characterized in that, Includes the following steps: Step T10: Acquire the real-time video stream and preprocess the acquired real-time video stream; Step T20: Extract spatiotemporal features from the preprocessed video stream and form a spatiotemporal feature sequence from the extracted spatiotemporal feature set; Step T30: Identify the spatiotemporal feature sequence to obtain the target behavior.
2. The data processing method based on artificial intelligence according to claim 1, characterized in that, Preprocessing includes the following sub-steps: Video frame sequences are obtained by sampling the video stream; The video frames in the video frame sequence are input into the video frame noise reduction processing model to perform filtering and noise reduction processing on the video frames. The video frame that has been filtered and denoised but has missing segments is input into the missing segment filling model to generate a substitute segment to repair the video frame. The resolution of the repaired video frames is scaled to a predetermined size. After scaling all video frames in the video frame sequence, the preprocessing of the video frame sequence is completed.
3. The data processing method based on artificial intelligence according to claim 1 or 2, characterized in that, Spatiotemporal feature extraction includes the following sub-steps: The video frame sequence is input into the static spatial feature extraction model of the dual-branch feature extraction model. Spatial features are extracted from each video frame of the video frame sequence, and the extracted spatial feature set is formed into a spatial feature sequence. The spatial feature sequence is input into the dynamic temporal feature extraction model of the dual-branch feature extraction model to extract temporal features from the spatial feature sequence, and the extracted temporal feature set is formed into a temporal feature sequence. Spatial feature sequences and temporal feature sequences are input into a feature fusion model for fusion to obtain spatiotemporal features.
4. The data processing method based on artificial intelligence according to claim 1 or 2, characterized in that, Identification of spatiotemporal feature sequences includes the following sub-steps: The spatiotemporal feature sequence corresponding to the video stream is converted into a global feature vector that can represent the video stream; A linear transformation is performed on the global feature vector representing the entire video stream to obtain the log odds vector of the entire video stream for different behavior categories. The log-probability vectors of the entire video stream for different behavior categories are transformed into probabilities, and the behavior category corresponding to the highest probability is taken as the target behavior.
5. The data processing method based on artificial intelligence according to claim 4, characterized in that, Transforming the global feature vector includes the following sub-steps: For each spatiotemporal feature in the spatiotemporal feature sequence corresponding to the video stream, a recognition weight is matched, and all recognition weights are set together to obtain the recognition weight sequence. The recognition weights in the recognition weight sequence and the corresponding spatiotemporal features in the spatiotemporal feature sequence are weighted and summed to obtain a global feature vector that can represent the video stream.
6. A data processing system based on artificial intelligence, characterized in that, include: Preprocessing unit, feature extraction unit, and recognition unit; The preprocessing unit acquires the real-time video stream and performs preprocessing on the acquired real-time video stream; The feature extraction unit extracts spatiotemporal features from the preprocessed video stream and forms a spatiotemporal feature sequence from the extracted spatiotemporal feature set. The recognition unit identifies spatiotemporal feature sequences to obtain the target behavior.
7. The data processing system based on artificial intelligence according to claim 6, characterized in that, The preprocessing unit includes: a sampling unit, a filtering and noise reduction unit, a repair unit, and a scaling unit; The sampling unit samples the video stream to obtain a video frame sequence; The filtering and noise reduction unit inputs video frames from the video frame sequence into the video frame noise reduction processing model to perform filtering and noise reduction processing on the video frames; The repair unit inputs the video frame with missing segments after filtering and noise reduction into the missing segment filling model to generate replacement segments to repair the video frame. The scaling unit scales the resolution of the repaired video frames to a predetermined size. After scaling all video frames in the video frame sequence, the preprocessing of the video frame sequence is completed.
8. The data processing system based on artificial intelligence according to claim 6 or 7, characterized in that, The feature extraction unit inputs the video frame sequence into the static spatial feature extraction model of the dual-branch feature extraction model, extracts spatial features from each video frame of the video frame sequence, and forms a spatial feature sequence from the extracted set of spatial features. The feature extraction unit inputs the spatial feature sequence into the dynamic temporal feature extraction model of the dual-branch feature extraction model, extracts temporal features from the spatial feature sequence, and forms a temporal feature sequence from the extracted temporal feature set. The feature extraction unit inputs the spatial feature sequence and the temporal feature sequence into the feature fusion model for fusion to obtain spatiotemporal features.
9. The data processing system based on artificial intelligence according to claim 6 or 7, characterized in that, The recognition unit includes: a feature vector transformation unit, a log-odds vector transformation unit, and a probability transformation judgment unit; The feature vector conversion unit converts the spatiotemporal feature sequence corresponding to the video stream into a global feature vector that can represent the video stream; The log-odds vector transformation unit performs a linear transformation on the global feature vector representing the entire video stream to obtain the log-odds vectors of the entire video stream for different behavior categories. The probability transformation and judgment unit transforms the log probability vector of the entire video stream for different behavior categories into probabilities, and takes the behavior category corresponding to the highest probability as the target behavior.
10. The data processing system based on artificial intelligence according to claim 9, characterized in that, The feature vector transformation unit matches recognition weights to each spatiotemporal feature in the spatiotemporal feature sequence corresponding to the video stream, and sets all recognition weights to obtain the recognition weight sequence. The feature vector conversion unit performs a weighted summation of the recognition weights in the recognition weight sequence and the corresponding spatiotemporal features in the spatiotemporal feature sequence to obtain a global feature vector that can represent the video stream.