Video reinjection server system for video processing and analysis

Abstract

The invention relates to the technical field of video stream processing, in particular to a video reinjection server system for video processing and analysis, which is used for solving the problems that in the prior art, accurate real-time detection of abnormal behaviors in video streams cannot be realized, real-time processing and abnormal state marking cannot be supported, and the video reinjection efficiency is low. The event response speed and the management efficiency are reduced; according to the invention, spatial-temporal feature extraction and long-time dependence modeling are carried out by using a 3D convolutional neural network and an LSTM through a behavior abnormity identification module, accurate real-time detection of abnormal behaviors in a video stream is realized, a sliding window mechanism and abnormal score judgment are adopted, real-time processing and abnormal state marking are supported, detailed structured event data are generated, and the real-time detection of the abnormal behaviors in the video stream is realized. And the event response speed and the management efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video stream processing, and more particularly to a video re-injection server system for video processing and analysis. Background Art

[0002] With the advancement of artificial intelligence and cloud computing, video surveillance systems are shifting from traditional storage to intelligent analysis. However, existing servers are limited to storage or simple transmission when processing video streams. They lack real-time analysis and feedback mechanisms, and in particular cannot inject analysis results back into the video stream for visualization. The main problems are low analysis efficiency, no video injection function, poor scalability, and no integrated edge computing, resulting in heavy cloud burdens and slow responses. There is an urgent need for a server system that integrates efficient video processing, intelligent analysis, and result injection to meet the intelligent needs of modern video surveillance.

[0003] Patent application with reference publication number CN114332782A discloses a traffic data reinjection system, which includes: a radar data acquisition module, a video stream acquisition module, a traffic data storage module, and a reinjection operation execution device; the radar data acquisition module is connected to a radar detector and is used to acquire traffic detection data collected by the radar detector; the video stream acquisition module is connected to a road camera assembly and is used to acquire road traffic videos captured by the road camera assembly; the traffic data storage module is used to time-align traffic detection data and road traffic videos; and the reinjection operation execution device is used to inject the time-aligned traffic detection data and road traffic videos into the tested system. The technical solution provided by the embodiment of the present invention realizes the reinjection of real traffic flow data in road traffic scenarios, provides real traffic data such as traffic detection data and road traffic videos for the development of road detection devices, and ensures the accuracy of traffic data;

[0004] However, the above-mentioned reference patent collects data through radar and video modules and injects it back into the system under test after time alignment, realizing the collection, storage and injection of real road traffic data, and providing accurate data for the development of road detection devices. However, it cannot achieve accurate real-time detection of abnormal behavior in video streams, cannot support real-time processing and abnormal status marking, and reduces event response speed and management efficiency; at the same time, it cannot accurately identify undiscovered or misjudged events, and cannot combine time overlap, type matching and label confidence for classification and judgment, which reduces event tracing and management efficiency in security and industrial scenarios.

[0005] To this end, we propose a video back-injection server system for video processing and analysis to address the above problems. Summary of the Invention

[0006] The purpose of the present invention is to provide a video re-injection server system for video processing and analysis, which solves the problem that the existing technology cannot achieve accurate real-time detection of abnormal behavior in video streams, cannot support real-time processing and abnormal status marking, and reduces event response speed and management efficiency; at the same time, it cannot accurately identify undiscovered or misjudged events, and cannot combine time overlap, type matching and label confidence for classification and judgment, which reduces the problem of event tracing and management efficiency in security and industrial scenarios.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A video re-injection server system for video processing and analysis, applied to a video analysis management platform, comprising:

[0009] The video stream access processing module is used to implement multi-source access to RTSP, HLS, HTTP protocol video streams and local video files, and perform pre-processing operations on all input video streams;

[0010] The behavior anomaly recognition module, based on deep learning and time series analysis methods, identifies abnormal behaviors in video streams in real time and generates structured event data with timestamps and event labels;

[0011] The event back-annotation recognition module retrospectively identifies undetected or misjudged historical events based on historical video streams and event tags, and generates structured event back-annotation data with timestamps and unique identifiers.

[0012] The event back-annotation execution module is used to automatically identify key events in the video stream based on historical back-annotation requirements and perform real-time or batch back-annotation tasks.

[0013] As a preferred embodiment of the present invention, the process of the abnormal behavior recognition module processing the video stream based on the deep learning method includes:

[0014] After preprocessing, a series of video frames arranged in chronological order are obtained, and a 3D convolutional neural network is used to extract spatiotemporal features from the video frame sequences. The specific steps are as follows:

[0015] Input: A continuous video frame sequence with a length of T and an image size of H×W×C. The entire video frame sequence can be represented as a four-dimensional tensor X∈R T×H×W×C ;

[0016] Convolutional layer: Multiple layers of 3D convolutional layers are used to extract multi-level features of the video stream in the temporal and spatial dimensions layer by layer. Each convolutional layer contains multiple convolution kernels, and each convolution kernel performs a sliding operation in the spatiotemporal domain to capture local spatiotemporal features. The convolution operation is expressed by the following formula:

[0017]

[0018] where Y i,j,k,l is the output of the 3D convolution operation, representing the feature map after convolution, W is the convolution kernel, b is the bias term, f is the activation function, T′, H′, W′ are the sizes of the convolution kernel, i, j, k are the indexes of the output feature map in time, height, and width, respectively, l is the channel index of the feature map, m, n, o are the indexes of the convolution kernel in time, height, and width, respectively;

[0019] Pooling layer: The pooling layer is introduced after the convolution layer to reduce the dimension of the feature map and reduce the amount of subsequent calculations;

[0020] Output: After multiple layers of convolution and pooling operations, a high-level spatiotemporal feature vector F∈R is generated. D , where D represents the dimension of the feature vector.

[0021] As a preferred embodiment of the present invention, the process of the abnormal behavior recognition module performing real-time recognition of abnormal behavior in the video stream based on the time series analysis method includes:

[0022] The spatiotemporal feature vector sequence {F1, F2, ..., F N} is input into the LSTM network for time series modeling. The LSTM network consists of an input gate, a forget gate, and an output gate, which is used to control the information flow and store long-term dependency information through cell states. Its update process includes state calculation under the gate control mechanism, and finally outputs a hidden state sequence {h1, h2, ..., h N}, where each h i Represents the hidden state of the i-th time step, which is determined by the current input and the state of the previous moment;

[0023] The final hidden state h of the LTSM network N Input to a fully connected layer and calculate the anomaly score S through the sigmoid activation function:

[0024] S=σ(Wh N +b), where W is the weight matrix, b is the bias vector, and σ is the sigmoid function;

[0025] The hidden state vector output by LSTM is input to the classifier to identify the type of abnormal event. The classifier can adopt SVM or multi-layer perceptron structure. During training, the labeled data is used as input and output pairs, and the cross entropy loss function is used. The parameters are updated through the Adam optimization algorithm. The classification result is determined by the maximum value in the output probability distribution.

[0026] As a preferred embodiment of the present invention, the process of generating structured event data by the behavior anomaly identification module includes:

[0027] Use sliding windows to segment the video frame sequence, and the window size determines the time range of each processing;

[0028] Calculate the anomaly score for each sliding window. If the anomaly scores of multiple consecutive windows exceed the set threshold, it is marked as an abnormal state;

[0029] The start time of the first window in which the anomaly score exceeds the threshold is recorded as the event start time;

[0030] When the anomaly scores of multiple consecutive windows are lower than the set threshold, the end time of the last window below the threshold is recorded as the event end time;

[0031] Based on the feature vector extracted during the event duration, a classifier is used to predict the event category.

[0032] As a preferred embodiment of the present invention, the process of the event back-annotation recognition module training a recognition model based on historical video streams and event labels includes:

[0033] Obtain historical video streams and existing event labels, and use a deep learning model to perform feature modeling and behavior recognition on the historical video streams. The model training steps are as follows:

[0034] T1: Use preprocessed video clips and their event labels for model training. The samples include positive and negative samples.

[0035] T2: The data is divided into training set, validation set and test set, and the corresponding loss function is defined according to the task type;

[0036] T3: Use the optimization algorithm to update the model parameters. Using batch training, each round of forward propagation, loss calculation, gradient backpropagation, and parameter update is performed.

[0037] T4: Regularly record the output results on the validation set, adjust the learning rate or terminate training, and output a model with determined parameters.

[0038] As a preferred embodiment of the present invention, the process of the event back-annotation identification module retrospectively identifying undiscovered or misjudged historical events includes:

[0039] The historical video stream is divided into video segments according to fixed time windows. These segments are fed into the trained model in sequence. The model outputs the probability distribution or classification score of each event category. Based on the output, a score is generated for each segment. If the score is higher than the set threshold, it is marked as a candidate event segment.

[0040] Perform temporal aggregation on the candidate event segments and generate event intervals using the following method:

[0041] Sliding window statistics candidate segment density;

[0042] The DBSCAN density clustering method is used to cluster the time points of the candidate segments;

[0043] If the time interval between two candidate segments is less than a preset threshold, they are combined into a continuous event interval;

[0044] Output a set of event time intervals, extract the original event labels and compare them with the identified event intervals. The comparison includes time overlap, event type matching, and the confidence or annotation status of the original labels.

[0045] The steps for classifying and judging events according to preset rules are as follows:

[0046] S1: For each event interval identified by the model, calculate its overlap with all original label time intervals;

[0047] S2: If the temporal overlap between the model recognition interval and all original label intervals is lower than the set threshold, or there is no overlap at all, it is determined that no event was found;

[0048] S3: If the model recognition interval significantly overlaps with an original label interval, but the event type is inconsistent, or the original label confidence is low and the label status is pending review, it is determined to be a potential misidentification event;

[0049] S4: If the model recognition result is highly consistent with the high-confidence original label in terms of time and type, it is determined that no back-annotation event is required and is not included in the back-annotation event list.

[0050] As a preferred embodiment of the present invention, the process of the event back-annotation identification module generating structured event back-annotation data with a timestamp and a unique identifier includes:

[0051] The filtered backtracking events are organized into a standard format to form structured data that can be stored and used. The input is a list of filtered backtracking events. Each event includes the identified time interval, the event type predicted by the model, the model score, and the judgment status.

[0052] Generate a globally unique identifier for each event using a UUID or hash algorithm to identify the event record.

[0053] The event type is determined by the model output in the recognition phase. The final type is obtained by aggregating the prediction results of multiple fragments. If the model does not support multi-class classification, it is uniformly marked as an abnormal type.

[0054] This information is organized into a standard data structure with fields including: event identifier, event type, start time, end time, confidence level, and judgment status.

[0055] As a preferred embodiment of the present invention, the process of the event back-annotation execution module automatically identifying key events in the video stream according to historical back-annotation requirements and executing real-time back-annotation tasks includes:

[0056] A structured event library is built based on historical back-annotation records. Each event type includes temporal, spatial, visual content, and metadata dimensional features. A rule-matching engine is used to identify events with fixed conditions. Machine learning models are suitable for identifying complex pattern events. A composite recognition mechanism combines multiple methods for comprehensive judgment. The system continuously analyzes new back-annotation results and updates or optimizes event definitions and recognition strategies.

[0057] When the recognition module detects a key event that meets the conditions, the system triggers the real-time backfill task:

[0058] Extract the event's timestamp, type, camera number, confidence level, and related raw video clips;

[0059] According to the preset strategy, the video clips before and after the event are intercepted and format converted or key frame extracted;

[0060] Collect snapshot images, recognition outputs, rule trigger information, and associated sensor data, and package them together with the video clips into a complete back-annotation record;

[0061] The back-injection record is written to the designated storage location through a high-speed channel, generating a unique identifier and index information;

[0062] Send notifications to designated recipients and push data to the target interface or platform.

[0063] As a preferred embodiment of the present invention, the process of the event back-injection execution module executing the batch back-injection task includes:

[0064] Batch re-injection is applied to historical video streams to supplement and identify omissions or re-analyze existing data based on new strategies:

[0065] Specify the processing time period, identification strategy used, video sources involved, and target storage format;

[0066] The system scans historical video streams within a specified time period and reads files from archive storage;

[0067] Apply recognition rules or models to analyze video streams, generate event lists, and filter low-confidence or repeated events;

[0068] Use the task scheduler to split tasks and assign them to multiple processing units for parallel execution;

[0069] Extract video clips before and after the event and process them in a unified format;

[0070] Pack the video clips, metadata, and recognition results into back-annotation records and write them to the target storage location in batches;

[0071] The system monitors task progress, records success and failure, and has a retry or skip mechanism;

[0072] After the task is completed, a report is generated, summarizing the total amount processed, the number of identified events, the number of successfully reinserted records, and error information.

[0073] Compared with the prior art, the advantages of the present invention are:

[0074] (1) In the present invention, the behavior anomaly recognition module uses a 3D convolutional neural network and LSTM to extract spatiotemporal features and model long-term dependencies, thereby achieving accurate real-time detection of abnormal behaviors in video streams. The sliding window mechanism and anomaly score judgment are adopted to support real-time processing and abnormal state marking, generate detailed structured event data, and improve event response speed and management efficiency.

[0075] (2) In the present invention, deep learning is used to analyze historical video streams through the event back-injection recognition module to accurately identify undetected or misjudged events and generate structured data with timestamps and unique identifiers. Flexible training, dynamic thresholds, sliding window statistics and DBSCAN clustering are used to optimize detection and aggregation to ensure high accuracy and consistency. Classification and judgment are performed in combination with time overlap, type matching and label confidence, significantly improving the event tracing and management efficiency in security and industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 is a system block diagram of the present invention;

[0077] Figure 2 This is a flowchart of the steps of training a recognition model based on historical video streams and event labels in the present invention;

[0078] Figure 3 This is a flow chart of the steps for classifying and determining events according to preset rules in the present invention. DETAILED DESCRIPTION

[0079] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making creative work shall fall within the scope of protection of the present invention.

[0080] Example 1: Figure 1 、 Figure 2 and Figure 3 As shown, the present invention proposes a video re-injection server system for video processing and analysis, which is applied to a video analysis management platform and includes:

[0081] The video stream access processing module is used to implement multi-source access to RTSP, HLS, HTTP protocol video streams and local video files, and perform pre-processing operations on all input video streams. Pre-processing operations include but are not limited to format transcoding (to adapt to different encoding formats such as H.264, HEVC, etc.), frame extraction and timing alignment (to ensure the accurate timestamp of each frame), and video quality optimization (such as removing image blur, jitter and color distortion, and performing noise suppression and sharpening processing);

[0082] The video stream access processing module has multi-source access capabilities and supports unified processing of RTSP, HLS, HTTP protocols and local video files, improving system compatibility and flexibility; the module can transcode video streams and adapt to encoding formats such as H.264 and HEVC to ensure data consistency and optimize storage and transmission efficiency; through frame extraction and timing alignment, it ensures frame-level timestamp accuracy, providing a reliable foundation for timing analysis; at the same time, the module integrates video quality optimization functions to effectively remove problems such as blur, jitter, color distortion, etc., and suppress noise, enhance details, and improve visual effects and algorithm recognition accuracy.

[0083] The behavior anomaly recognition module, based on deep learning and time series analysis methods, identifies abnormal behaviors in video streams in real time and generates structured event data with timestamps and event labels;

[0084] The process of processing video streams by the behavior anomaly recognition module based on deep learning methods includes:

[0085] After preprocessing, a series of video frames arranged in chronological order are obtained, and a 3D convolutional neural network is used to extract spatiotemporal features from the video frame sequences. The specific steps are as follows:

[0086] Input: A continuous video frame sequence with a length of T and an image size of H×W×C. The entire video frame sequence can be represented as a four-dimensional tensor X∈R T×H×W×C ;

[0087] Convolutional layer: Multiple 3D convolutional layers are used to extract multi-level features of the video stream in the temporal and spatial dimensions layer by layer. Each convolutional layer contains multiple convolution kernels, and each convolution kernel slides in the spatiotemporal domain to capture local spatiotemporal features. The size, number, and step size of the convolution kernel need to be adjusted based on experimental results. The convolution operation is expressed by the following formula:

[0088]

[0089] where Y i,j,k,l is the output of the 3D convolution operation, representing the feature map after convolution, W is the convolution kernel, b is the bias term, f is the activation function, T′, H′, W′ are the sizes of the convolution kernel, i, j, k are the indexes of the output feature map in time, height, and width, respectively, l is the channel index of the feature map, m, n, o are the indexes of the convolution kernel in time, height, and width, respectively;

[0090] Pooling layer: A pooling layer (such as maximum pooling or average pooling) is introduced after the convolution layer to reduce the dimensionality of the feature map and reduce the amount of subsequent calculations. The pooling operation performs information aggregation in a local area and usually downsamples in time and space dimensions.

[0091] Output: After multiple layers of convolution and pooling operations, a high-level spatiotemporal feature vector F∈R is generated. D , where D represents the dimension of the feature vector;

[0092] The behavior anomaly recognition module uses the time series analysis method to identify abnormal behaviors in the video stream in real time. The process includes:

[0093] The spatiotemporal feature vector sequence {F1, F2, ..., F N} is input into the LSTM network for time series modeling. The LSTM network consists of an input gate, a forget gate, and an output gate, which is used to control the information flow and store long-term dependency information through cell states. Its update process includes state calculation under the gate control mechanism, and finally outputs a hidden state sequence {h1, h2, ..., h N}, where each h i Represents the hidden state of the i-th time step, which is determined by the current input and the state of the previous moment;

[0094] The final hidden state h of the LTSM network N Input to a fully connected layer and calculate the anomaly score S through the sigmoid activation function:

[0095] S=σ(Wh N+b), where W is the weight matrix, b is the bias vector, σ is the sigmoid function, S∈[0,1], and the higher the value of S, the more likely it is abnormal behavior;

[0096] The hidden state vector output by the LSTM is input into the classifier to identify the type of abnormal event. The classifier can use an SVM or multi-layer perceptron structure. During training, the labeled data is used as the input and output pairs, and the cross entropy loss function is used. The parameters are updated through the Adam optimization algorithm. The classification result is determined by the maximum value in the output probability distribution.

[0097] The process of generating structured event data by the behavior anomaly recognition module includes:

[0098] Use sliding windows to segment the video frame sequence, and the window size determines the time range of each processing;

[0099] Calculate the anomaly score for each sliding window. If the anomaly scores of multiple consecutive windows exceed the set threshold, it is marked as an abnormal state;

[0100] The start time of the first window in which the anomaly score exceeds the threshold is recorded as the event start time;

[0101] When the anomaly scores of multiple consecutive windows are lower than the set threshold, the end time of the last window below the threshold is recorded as the event end time;

[0102] Based on the feature vector extracted during the event duration, a classifier is used to predict the event category. This process is completed based on the labeled dataset.

[0103] The behavior anomaly recognition module uses 3D convolutional neural networks and LSTM for efficient spatiotemporal feature extraction and long-term dependency analysis, and can accurately detect abnormal behavior in video streams in real time. The system uses a sliding window mechanism to calculate anomaly scores and automatically marks events when consecutive high-scoring windows appear. At the same time, it generates structured data containing precise timestamps and event types, greatly improving event response speed and management efficiency. In addition, it supports flexible configuration of multiple classifiers such as SVM or multi-layer perceptron, which not only enhances adaptability to different application scenarios, but also ensures high accuracy and reliability. The entire system is trained based on labeled data and continuously optimizes parameters, ensuring excellent performance and expansion potential, making it a powerful assistant in security monitoring, industrial monitoring and other fields. This efficient, accurate and flexible abnormal behavior recognition solution effectively improves the overall efficiency of security management and incident handling.

[0104] The event back-annotation recognition module retrospectively identifies undetected or misjudged historical events based on historical video streams and event tags, and generates structured event back-annotation data with timestamps and unique identifiers.

[0105] The process of training the recognition model based on historical video streams and event labels in the event back-annotation recognition module includes:

[0106] Obtain historical video streams and existing event labels, and use a deep learning model to perform feature modeling and behavior recognition on the historical video streams. The model training steps are as follows:

[0107] T1: Use preprocessed video clips and their event labels for model training. The samples include positive samples (video clips containing the target event) and negative samples (video clips not containing the target event).

[0108] T2: The data is divided into training, validation, and test sets. The corresponding loss function is defined according to the task type: cross entropy loss is used for classification tasks; mean squared error is used for regression tasks; multi-task learning combines multiple loss functions and sets corresponding weights; sample weights are introduced based on category distribution to balance sample size differences;

[0109] T3: Use optimization algorithms (such as Adam and SGD) to update model parameters, using batch training. Each round of training performs forward propagation, loss calculation, gradient backpropagation, and parameter update.

[0110] T4: Regularly record the output results on the validation set, adjust the learning rate or terminate training, and output a model with determined parameters;

[0111] For specific deviation samples, training is continued based on the initial model. The data comes from unlabeled or classified abnormal clips in the historical video stream. After screening and annotation, a fine-tuning dataset is formed. A small learning rate is used for limited rounds of training. Some layer parameters can be selectively updated to output a model with updated parameters.

[0112] The process by which the event back-annotation recognition module retrospectively identifies undiscovered or misjudged historical events includes:

[0113] The historical video stream is divided into video segments according to fixed time windows. These segments are fed into the trained model in sequence. The model outputs the probability distribution or classification score of each event category. Based on the output, a score is generated for each segment. If the score is higher than the set threshold, it is marked as a candidate event segment. The threshold is dynamically adjusted based on the performance of the validation set, or an adaptive threshold mechanism is adopted.

[0114] Perform temporal aggregation on the candidate event segments and generate event intervals using the following method:

[0115] Sliding window statistics candidate segment density;

[0116] The DBSCAN density clustering method is used to cluster the time points of the candidate segments;

[0117] If the time interval between two candidate segments is less than a preset threshold, they are combined into a continuous event interval;

[0118] Output a set of event time intervals, extract the original event labels and compare them with the identified event intervals. The comparison includes time overlap, event type matching, and the confidence or annotation status of the original labels.

[0119] The steps for classifying and judging events according to preset rules are as follows:

[0120] S1: For each event interval identified by the model, calculate its overlap with all original label time intervals;

[0121] S2: If the temporal overlap between the model recognition interval and all original label intervals is lower than a set threshold (e.g., 70%), or there is no overlap at all, it is determined that no event was found;

[0122] S3: If the model recognition interval significantly overlaps with an original label interval (i.e., the overlap is higher than the set threshold), but the event type is inconsistent, or the original label confidence is low (e.g., less than 60%), and the labeling status is pending review, it is determined to be a potential misidentification event;

[0123] S4: If the model recognition result is highly consistent with the original label with high confidence (e.g., greater than 80%) in time and type (i.e., the overlap is higher than the set threshold and the event type is the same), it is determined that the event does not need to be reinserted and is not included in the reinserted event list;

[0124] The process by which the event back-annotation recognition module generates structured event back-annotation data with timestamps and unique identifiers includes:

[0125] The filtered backtracking events are organized into a standard format to form structured data that can be stored and used. The input is a list of filtered backtracking events. Each event includes the identified time interval (start time, end time), the event type predicted by the model, the model score, and the judgment status.

[0126] Generate a globally unique identifier for each event. This identifier is generated through a UUID or hash algorithm to identify the event record. The event time information is derived from the original timestamp of the video stream. The start and end times are expressed with high precision and are synchronized with the original video frame timestamp.

[0127] The event type is determined by the model output in the recognition phase. The final type is obtained by aggregating the prediction results of multiple fragments. If the model does not support multi-class classification, it is uniformly marked as an abnormal type.

[0128] The confidence level is calculated based on the model recognition results. The highest confidence level or the average score within the time window can be selected as the final confidence level.

[0129] This information is organized into a standard data structure with fields including: event identifier, event type, start time, end time, confidence level, and judgment status. The data format uses a common structure and supports JSON or XML formats.

[0130] The event back-annotation recognition module performs feature modeling and behavior recognition on historical video streams based on deep learning, accurately mines undiscovered or misjudged historical events, and generates structured back-annotation data with timestamps and unique identifiers. The module adopts a flexible training strategy, combining positive and negative samples, multi-task loss functions, and deviation sample fine-tuning to improve model adaptability and accuracy. It screens candidate events through dynamic thresholds, combines sliding window statistics with DBSCAN clustering to achieve efficient time series aggregation and enhance the coherence of event intervals. Classification and judgment integrate time overlap, type matching, and label confidence to ensure reliable results. Finally, it outputs standardized structured data, significantly improving event tracing and management capabilities in security, industrial and other scenarios.

[0131] Embodiment 2: The technical solution of this embodiment of the present invention differs from that of Embodiment 1 in that:

[0132] like Figure 1 As shown, the event back-annotation execution module is used to automatically identify key events in the video stream according to historical back-annotation requirements and perform real-time or batch back-annotation tasks;

[0133] The event backtracking execution module automatically identifies key events in the video stream based on historical backtracking requirements and performs real-time backtracking tasks. The process includes:

[0134] A structured event library is built based on historical back-annotation records. Each event type includes temporal, spatial, visual content, and metadata dimensional features. A rule-matching engine is used to identify events with fixed conditions. Machine learning models are suitable for identifying complex pattern events. A composite recognition mechanism combines multiple methods for comprehensive judgment. The system continuously analyzes new back-annotation results and updates or optimizes event definitions and recognition strategies.

[0135] When the recognition module detects a key event that meets the conditions, the system triggers the real-time backfill task:

[0136] Extract the event's timestamp, type, camera number, confidence level, and related raw video clips;

[0137] According to the preset strategy, the video clips before and after the event are intercepted and format converted or key frame extracted;

[0138] Collect snapshot images, recognition outputs, rule trigger information, and associated sensor data, and package them together with the video clips into a complete back-annotation record;

[0139] The back-injection record is written to the designated storage location via a high-speed channel, generating a unique identifier and index information (timestamp, event type, camera number);

[0140] Send notifications to designated recipients and push data to the target interface or platform;

[0141] The process of the event back-injection execution module executing batch back-injection tasks includes:

[0142] Batch re-injection is applied to historical video streams to supplement and identify omissions or re-analyze existing data based on new strategies:

[0143] Specify the processing time period, identification strategy used, video sources involved, and target storage format;

[0144] The system scans historical video streams within a specified time period and reads files from archive storage;

[0145] Apply recognition rules or models to analyze video streams, generate event lists, and filter low-confidence or repeated events;

[0146] Use the task scheduler to split tasks and assign them to multiple processing units for parallel execution;

[0147] Extract video clips before and after the event and process them in a unified format;

[0148] Pack the video clips, metadata, and recognition results into back-annotation records and write them to the target storage location in batches;

[0149] The system monitors task progress, records success and failure, and has a retry or skip mechanism;

[0150] Generate a report after the task is completed, summarizing the total amount processed, the number of identified events, the number of successfully reinserted records, and error information;

[0151] The event backtracking execution module accurately identifies key events through a structured event library and composite recognition mechanism, and supports real-time and batch backtracking tasks; the system automatically extracts complete event data, generates standardized backtracking records, and combines unique identification and index information to facilitate efficient retrieval and management; it adopts parallel task scheduling and high-speed writing mechanisms to improve processing efficiency and system stability, supports task monitoring, retries and report generation, realizes rapid data sharing and cross-platform collaboration, and significantly enhances event tracing and response capabilities in security and industrial scenarios.

[0152] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes based on the technical solutions and improved concepts of the present invention within the technical scope disclosed by the present invention, and they should be covered by the scope of protection of the present invention.

Claims

1. A video re-injection server system for video processing and analysis, applied to a video analysis management platform, characterized in that: include: The video stream access processing module is used to implement multi-source access to RTSP, HLS, HTTP protocol video streams and local video files, and perform pre-processing operations on all input video streams; The behavior anomaly recognition module, based on deep learning and time series analysis methods, identifies abnormal behaviors in video streams in real time and generates structured event data with timestamps and event labels; The event back-annotation recognition module retrospectively identifies undetected or misjudged historical events based on historical video streams and event tags, and generates structured event back-annotation data with timestamps and unique identifiers. The event back-annotation execution module is used to automatically identify key events in the video stream based on historical back-annotation requirements and perform real-time or batch back-annotation tasks.

2. A video re-injection server system for video processing and analysis according to claim 1, characterized in that: The process of the abnormal behavior recognition module processing the video stream based on the deep learning method includes: After preprocessing, a series of video frames arranged in chronological order are obtained, and a 3D convolutional neural network is used to extract spatiotemporal features from the video frame sequences. The specific steps are as follows: Input: A continuous video frame sequence with a length of T and an image size of H×W×C. The entire video frame sequence can be represented as a four-dimensional tensor X∈R T×H×W×C ; Convolutional layer: Multiple layers of 3D convolutional layers are used to extract multi-level features of the video stream in the temporal and spatial dimensions layer by layer. Each convolutional layer contains multiple convolution kernels, and each convolution kernel performs a sliding operation in the spatiotemporal domain to capture local spatiotemporal features. The convolution operation is expressed by the following formula: where Y i,j,k,l is the output of the 3D convolution operation, representing the feature map after convolution, W is the convolution kernel, b is the bias term, f is the activation function, T′, H′, W′ are the sizes of the convolution kernel, i, j, k are the indexes of the output feature map in time, height, and width, respectively, l is the channel index of the feature map, m, n, o are the indexes of the convolution kernel in time, height, and width, respectively; Pooling layer: The pooling layer is introduced after the convolution layer to reduce the dimension of the feature map and reduce the amount of subsequent calculations; Output: After multiple layers of convolution and pooling operations, a high-level spatiotemporal feature vector F∈R is generated. D , where D represents the dimension of the feature vector.

3. A video re-injection server system for video processing and analysis according to claim 2, characterized in that: The process of the behavior anomaly recognition module performing real-time recognition of abnormal behaviors in the video stream based on the time series analysis method includes: The spatiotemporal feature vector sequence {F1, F2, ..., F N } is input into the LSTM network for time series modeling. The LSTM network consists of an input gate, a forget gate, and an output gate, which is used to control the information flow and store long-term dependency information through cell states. Its update process includes state calculation under the gate control mechanism, and finally outputs a hidden state sequence {h1, h2, ..., h N }, where each h i Represents the hidden state of the i-th time step, which is determined by the current input and the state of the previous moment; The final hidden state h of the LTSM network N Input to a fully connected layer and calculate the anomaly score S through the sigmoid activation function: S=σ(Wh N +b), where W is the weight matrix, b is the bias vector, and σ is the sigmoid function; The hidden state vector output by LSTM is input to the classifier to identify the type of abnormal event. The classifier can adopt SVM or multi-layer perceptron structure. During training, the labeled data is used as input and output pairs, and the cross entropy loss function is used. The parameters are updated through the Adam optimization algorithm. The classification result is determined by the maximum value in the output probability distribution.

4. A video re-injection server system for video processing and analysis according to claim 3, characterized in that: The process of generating structured event data by the behavior anomaly identification module includes: Use sliding windows to segment the video frame sequence, and the window size determines the time range of each processing; Calculate the anomaly score for each sliding window. If the anomaly scores of multiple consecutive windows exceed the set threshold, it is marked as an abnormal state; The start time of the first window in which the anomaly score exceeds the threshold is recorded as the event start time; When the anomaly scores of multiple consecutive windows are lower than the set threshold, the end time of the last window below the threshold is recorded as the event end time; Based on the feature vector extracted during the event duration, a classifier is used to predict the event category.

5. A video re-injection server system for video processing and analysis according to claim 1, characterized in that: The process of training the recognition model of the event back-annotation recognition module based on the historical video stream and event labels includes: Obtain historical video streams and existing event labels, and use a deep learning model to perform feature modeling and behavior recognition on the historical video streams. The model training steps are as follows: T1: Use preprocessed video clips and their event labels for model training. The samples include positive and negative samples. T2: The data is divided into training set, validation set and test set, and the corresponding loss function is defined according to the task type; T3: Use the optimization algorithm to update the model parameters. Using batch training, each round of forward propagation, loss calculation, gradient backpropagation, and parameter update is performed. T4: Regularly record the output results on the validation set, adjust the learning rate or terminate training, and output a model with determined parameters.

6. A video re-injection server system for video processing and analysis according to claim 5, characterized in that: The process of the event back-annotation recognition module retrospectively identifying undiscovered or misjudged historical events includes: The historical video stream is divided into video segments according to fixed time windows. These segments are fed into the trained model in sequence. The model outputs the probability distribution or classification score of each event category. Based on the output, a score is generated for each segment. If the score is higher than the set threshold, it is marked as a candidate event segment. Perform temporal aggregation on the candidate event segments and generate event intervals using the following method: Sliding window statistics candidate segment density; The DBSCAN density clustering method is used to cluster the time points of the candidate segments; If the time interval between two candidate segments is less than a preset threshold, they are combined into a continuous event interval; Output a set of event time intervals, extract the original event labels and compare them with the identified event intervals. The comparison includes time overlap, event type matching, and the confidence or annotation status of the original labels. The steps for classifying and judging events according to preset rules are as follows: S1: For each event interval identified by the model, calculate its overlap with all original label time intervals; S2: If the temporal overlap between the model recognition interval and all original label intervals is lower than the set threshold, or there is no overlap at all, it is determined that no event was found; S3: If the model recognition interval significantly overlaps with an original label interval, but the event type is inconsistent, or the original label confidence is low and the label status is pending review, it is determined to be a potential misidentification event; S4: If the model recognition result is highly consistent with the high-confidence original label in terms of time and type, it is determined that no back-annotation event is required and is not included in the back-annotation event list.

7. A video re-injection server system for video processing and analysis according to claim 6, characterized in that: The process of the event back-annotation identification module generating structured event back-annotation data with a timestamp and a unique identifier includes: The filtered backtracking events are organized into a standard format to form structured data that can be stored and used. The input is a list of filtered backtracking events. Each event includes the identified time interval, the event type predicted by the model, the model score, and the judgment status. Generate a globally unique identifier for each event using a UUID or hash algorithm to identify the event record. The event type is determined by the model output in the recognition phase. The final type is obtained by aggregating the prediction results of multiple fragments. If the model does not support multi-class classification, it is uniformly marked as an abnormal type. This information is organized into a standard data structure with fields including: event identifier, event type, start time, end time, confidence level, and judgment status.

8. A video re-injection server system for video processing and analysis according to claim 1, characterized in that: The process of the event back-annotation execution module automatically identifying key events in the video stream according to the historical back-annotation requirements and executing the real-time back-annotation task includes: A structured event library is built based on historical back-annotation records. Each event type includes temporal, spatial, visual content, and metadata dimensional features. A rule-matching engine is used to identify events with fixed conditions. Machine learning models are suitable for identifying complex pattern events. A composite recognition mechanism combines multiple methods for comprehensive judgment. The system continuously analyzes new back-annotation results and updates or optimizes event definitions and recognition strategies. When the recognition module detects a key event that meets the conditions, the system triggers the real-time backfill task: Extract the event's timestamp, type, camera number, confidence level, and related raw video clips; According to the preset strategy, the video clips before and after the event are intercepted and format converted or key frame extracted; Collect snapshot images, recognition outputs, rule trigger information, and associated sensor data, and package them together with the video clips into a complete back-annotation record; The back-injection record is written to the designated storage location through a high-speed channel, generating a unique identifier and index information; Send notifications to designated recipients and push data to the target interface or platform.

9. A video re-injection server system for video processing and analysis according to claim 8, characterized in that: The process of the event back-injection execution module executing the batch back-injection task includes: Batch re-injection is applied to historical video streams to supplement and identify omissions or re-analyze existing data based on new strategies: Specify the processing time period, identification strategy used, video sources involved, and target storage format; The system scans historical video streams within a specified time period and reads files from archive storage; Apply recognition rules or models to analyze video streams, generate event lists, and filter low-confidence or repeated events; Use the task scheduler to split tasks and assign them to multiple processing units for parallel execution; Extract video clips before and after the event and process them in a unified format; Pack the video clips, metadata, and recognition results into back-annotation records and write them to the target storage location in batches; The system monitors task progress, records success and failure, and has a retry or skip mechanism; After the task is completed, a report is generated, summarizing the total amount processed, the number of identified events, the number of successfully reinserted records, and error information.

Images