Road video monitoring video data restoration method based on relational complementary constraint

Abstract

The application belongs to the technical field of image restoration, and relates to a road video monitoring video data restoration method based on relational complementary constraints, which comprises the following steps: first, constructing a data set, then performing multi-scale space-time feature coding on video sequences in the data set, then performing vehicle relationship modeling based on graph relationship, sequentially performing long-time sequence local vehicle area level repair and long-time sequence global vehicle object level repair, then repairing and fusing long-time sequence vehicle missing content, then modeling dynamic scenes in the video sequence by using a relationship adversarial network, introducing vehicle relationship adversarial training between a generator and a discriminator, enabling the generator to learn vehicle relationship information in the video sequence, thereby simulating changes of the dynamic scenes, generating new video data, and finally performing network training and testing; the method can improve the accuracy and robustness of data restoration, reduce the misjudgment rate of data restoration, and enhance the application scenarios of data restoration.

Description

Technical Field

[0001] This invention belongs to the field of image restoration technology and relates to a method for restoring road video surveillance video data based on relational complementary constraints. Background Technology

[0002] Road video surveillance data restoration refers to the use of deep learning methods to repair and restore images in road surveillance videos in order to restore lost, obscured, blurred, or damaged data.

[0003] Currently, significant progress has been made in road video surveillance data restoration methods in many aspects, but some challenges and limitations still exist. Road surveillance videos are typically captured in complex scenes and lighting conditions, such as at night, in rain, or in snow. Under these conditions, images may be affected by severe noise, lighting variations, and motion blur, making data restoration difficult. Furthermore, road surveillance videos often contain occluded targets, such as vehicles occluding each other, trees, or buildings. Restoring occluded targets is a complex task, requiring accurate estimation and reconstruction of the location, shape, and texture of the occluded targets. Road surveillance video data is typically characterized by its large scale, high resolution, and high frame rate; processing this large-scale and complex data requires efficient computing and storage resources. In addition, road surveillance videos often contain data from multiple cameras, involving data fusion and processing from multiple perspectives, further increasing the complexity of the algorithms.

[0004] Existing road video surveillance systems generate massive amounts of video data, which can be considered an advantage for road video surveillance data restoration algorithms. The large volume of video data provides ample samples for algorithm training and validation, helping deep learning models learn more accurate features and patterns. Through this vast amount of video data, more complex and accurate deep learning models can be built, thereby improving the accuracy and robustness of data restoration. Furthermore, the large amount of video data can be used for data augmentation techniques, such as data augmentation and data synthesis, to increase the diversity of training samples and improve the model's generalization performance. In addition, with the continuous improvement of hardware computing power, modern computing devices such as GPUs and TPUs can process large-scale video data and accelerate the training and inference process of deep learning algorithms, enabling restoration algorithms to process large amounts of video data in a shorter time and meet real-time requirements. Summary of the Invention

[0005] To overcome the problems existing in the prior art, this invention provides a road video surveillance data restoration method based on relational complementary constraints, which improves the accuracy and robustness of the restoration results by analyzing the relationships and motion patterns between vehicles.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for restoring road video surveillance data based on relational complementary constraints, specifically including the following steps: S1. Constructing the dataset: Collect video data containing road traffic monitoring, label the collected data with vehicle location information, and then artificially introduce missing or corrupted data into the labeled video data to obtain the constructed dataset. Then, divide the constructed dataset into training set, validation set and test set. S2. Multi-scale spatiotemporal feature encoding: The video sequences in the dataset are input into the backbone network Transformer. First, multi-scale spatial and temporal decomposition is performed. The extracted original features are decomposed into sub-features of different scales. The encoder encodes each sub-feature. Finally, the encoded sub-features are fused and integrated to obtain the multi-scale spatiotemporal feature encoding representation. S3. Vehicle Relationship Modeling Based on Graph Relationships: Graph modeling based on feature similarity (SIM) models the relationships between nodes as a similarity metric (SIM), thus obtaining the relationships between nodes; S4. Long-term local vehicle region-level restoration: Based on the multi-scale spatiotemporal feature encoding representation obtained in step S2, vehicle information in the local region of the video sequence is restored by processing the local region. S5. Long-term global vehicle object-level restoration: The object detection algorithm is used to detect objects in the video sequence, obtain the position information, width and height of the objects, convert the image-level position information to the object-level position information, calculate the object-level similarity and object-level inter-frame similarity, and restore the entire vehicle object. S6. Long-term vehicle missing content repair and fusion: The repair results of steps S4 and S5 are fused to obtain the enhanced features after fusion; S7. Relationship Adversarial Network: Relationship adversarial network is used to model dynamic scenes in video sequences. By introducing adversarial training of vehicle relationships between the generator and the discriminator, the generator learns the vehicle relationship information in the video sequence, thereby simulating changes in dynamic scenes and generating new video data. S8. Network Training and Testing: Perform generator training, discriminator training, and temporal adversarial training respectively. Use the validation set or test set to evaluate the trained generator and optimize and improve it based on the evaluation results.

[0007] As a further technical solution of the present invention, the video data containing road traffic monitoring mentioned in step S1 is an existing video dataset or video data collected at actual road intersections or road sections. The video data covers various situations such as different weather conditions, different time periods, and different traffic flows to ensure the diversity of the dataset.

[0008] As a further technical solution of the present invention, the multi-scale spatiotemporal feature encoding in step S2 is represented as follows: , Here, Devide(Frames) represents multi-scale spatial and temporal decomposition of the input video frames, decomposing the original features into sub-features of different scales; Mask represents adding a mask to the corresponding region to generate an image with lost vehicle information; Encoder represents encoding each sub-feature and extracting its important feature information; Fuse represents fusing and integrating the encoded sub-features, i.e., the outputs of the Transformer side of layers 2, 3, and 4.

[0009] As a further technical solution of the present invention, the relationship between the nodes in step S3 is as follows: , Where i and j represent the position indices of pixels in the node features. Represents a similarity matrix; Represents the current node. Represents adjacent nodes. This represents the similarity measure between nodes, which is implemented using the L2 distance function. This indicates normalization processing.

[0010] As a further technical solution of the present invention, the specific process of step S4 is as follows: , in, Represents regional similarity. This represents the current regional-level multi-scale spatiotemporal feature encoding representation. This represents the multi-scale spatiotemporal feature encoding representation of adjacent regions.

[0011] As a further technical solution of the present invention, step S5 involves using an object detection algorithm to detect objects in the video frame as follows: , in, Representative object detection algorithms, This represents classifying detected objects into different object numbers, where i is the number. The generated object's position information (x, y) and its width w and height h are given. Represents object-level position information; This represents the process of data flow operations; The process of calculating object-level similarity is as follows: , in, Represents object-level similarity. This represents the current multi-scale spatiotemporal feature encoding representation of the vehicle object. Represents the multi-scale spatiotemporal feature encoding of adjacent vehicle objects; The process of calculating object-level inter-frame similarity is as follows: , in, This represents the object-level inter-frame similarity between frame t and frame t+n. The multi-scale spatiotemporal feature encoding representation of the current vehicle at the object level in frame t. The multi-scale spatiotemporal feature encoding representation of the current vehicle at the object level in frame t+n.

[0012] As a further technical solution of the present invention, the enhanced feature after fusion in step S6 is as follows: , , This represents the enhanced features resulting from fusion. Representative features are superimposed along the channel dimension. Represents dimensionality reduction using convolutional layers. Represents matrix multiplication. This represents the final, repaired video output. This represents converting features into an output RGB image. Represents the loss in the reconstruction of spatial relationships. This represents the loss for reconstructing temporal relationships, and the loss function is L1+adversarial loss.

[0013] As a further technical solution of the present invention, the loss function adopted by the relational adversarial network in step S7, that is, minimizing the difference between the generator's generated result and the discriminator's evaluated result, is set as follows: , in, To maximize the training objective in the discriminator DNet, Minimize the training objective of the generator GNet.

[0014] As a further technical solution of the present invention, the generator training, discriminator training and temporal adversarial training processes in step S8 are specifically as follows: Generator Training: Use the generator GNet to generate Fake restored video data and calculate the spatial relationship reconstruction loss. Loss of reconstructing temporal relationships By minimizing these two loss functions, the generator parameters are updated to obtain a trained generator; Discriminator training: Using Fake video data generated by the generator and Real video data from the training set as input, the parameters of the discriminator are updated to evaluate the realism of the Fake video data generated by the generator. Temporal adversarial training: Alternately update the parameters of the generator and discriminator, and minimize the difference between the generator's output and the discriminator's evaluation. To conduct competitive training.

[0015] Compared with the prior art, the beneficial effects of the present invention are: (1) Improve the accuracy of data restoration: By introducing the relationship model between vehicles and time series information, missing or damaged data can be inferred more accurately, thereby improving the accuracy of data restoration.

[0016] (2) Improve the robustness of data restoration: Relational complementary constraints can help suppress noise interference in the data restoration process, thereby improving the robustness of the restoration algorithm. By considering the relationship between vehicles, restoration errors caused by errors or noise in single vehicle data can be reduced.

[0017] (3) Reduce the misjudgment rate of data restoration: Vehicle relationship complementary constraints can be used as a priori information to help reduce the misjudgment rate in data restoration.

[0018] (4) Enhance the application scenarios of data restoration: Data restoration methods based on vehicle relationship complementary constraints can be applied to road traffic monitoring, traffic flow analysis, accident research, and vehicle trajectory reconstruction. By improving the accuracy and robustness of data restoration, the reliability and practicality of these application scenarios can be enhanced. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the process framework for the road video surveillance video data restoration method based on relational complementary constraints provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the network framework used in the road video surveillance video data restoration method based on relational complementary constraints provided by the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0022] Example: like Figure 1 and Figure 2As shown, this embodiment provides a method for restoring road video surveillance data based on relational complementary constraints. The specific implementation includes the following steps: S1. Constructing the dataset: A large amount of video data containing road traffic monitoring data was collected. This data came from existing video datasets or was collected through actual video capture at road intersections or road sections. This embodiment uses the BDD100K autonomous driving dataset, which provides more than 1,100 hours of 100,000 high-definition video sequences. The provided video data covers different weather conditions, different time periods, and different traffic flow conditions. The collected video data was annotated in detail, including the accurate location information of vehicles. In the annotated video data, data missing or damaged situations were artificially introduced, such as occlusion, blurring, and loss, to simulate the data missing situations that may occur in actual monitoring scenarios. The constructed dataset was divided into training set, validation set, and test set for training, validating, and evaluating the road video monitoring data restoration algorithm based on relational complementary constraints.

[0023] S2. Multi-scale spatiotemporal feature encoding: Video data containing road traffic monitoring data typically includes information at different scales, such as vehicles on the road, the relationships between vehicles, and the relationship between vehicles and the road background. Multi-scale spatiotemporal feature encoding can extract feature representations at multiple scales from the input video data, thereby capturing information at different scales. This allows the network to process information at different scales and better recover missing data. This is because multi-scale feature encoding can capture contextual information at different scales by extracting features at different scales. Contextual information is crucial for video data recovery tasks, as it provides information about the surrounding environment of the missing area, thus helping the network to recover the missing data more accurately. The specific process is as follows: Given the input video sequence Frames, multi-scale spatiotemporal feature encoding calculates the encoded feature representation ST using the following formula: , In this context, Devide(Frames) represents multi-scale spatial and temporal decomposition of the input video frames, decomposing the original features into sub-features of different scales. In this embodiment, the video sequence Frames contains 15 consecutive frames of data. Mask represents adding a mask to the corresponding region to generate an image with lost vehicle information. In this embodiment, the Mask is generated by randomly generating the Mask position, and the length and width of the Mask are set to W=[0, 1 / 4×W] and H=[0, 1 / 4×H]. Encoder represents encoding each sub-feature and extracting its important feature information. The main network architecture of the entire process adopts Transformer. Fuse represents fusing and integrating the encoded sub-features, i.e., the outputs of the Transformer side of layers 2, 3, and 4, to obtain the final multi-scale spatiotemporal feature encoding representation ST.

[0024] S3. Vehicle relationship modeling based on graph relationships: Based on feature similarity Graph modeling is a relational modeling method used to model the relationships between nodes in a graph as a similarity metric. Nodes are typically represented by feature vectors, which can be node attributes, contextual information, or other available features. The similarity between nodes is calculated by examining these feature vectors. The relationships between nodes can be determined. In this embodiment, the L2 distance function is used to measure the feature similarity between nodes. For a given pair of nodes, the L2 distance between their feature vectors is calculated, and a similarity threshold is used to determine if they are related. Then, the relationships between nodes are inferred by aggregating the features of the node's neighbors. This method utilizes the graph's topology and node feature information to model the relationships between nodes. Specifically: , Where i and j represent the position indices of pixels in the node features. Represents a similarity matrix; Represents the current node. Represents adjacent nodes. This represents the similarity measure between nodes, which is implemented using the L2 distance function. This indicates normalization processing.

[0025] S4. Long-term local vehicle area-level repair: This step focuses on repairing specific local areas or specific vehicles, primarily processing local regions within a video sequence and repairing only vehicle information within those regions. It relies on the contextual information and geometric features of the local region to fill in missing vehicle information. Suitable for repairing specific regions or specific vehicles, it can selectively repair based on specific scene and task requirements. Specifically: , in, Represents regional similarity. This represents the current regional-level multi-scale spatiotemporal feature encoding representation. This represents the multi-scale spatiotemporal feature encoding representation of adjacent regions.

[0026] S5, Long-term global vehicle object-level repair: This step repairs the entire vehicle object, involving all vehicles in the entire video sequence. Its purpose is to recover missing or damaged portions of the vehicle object. This step utilizes global contextual information and extensive visual features to infer and repair missing vehicle object parts. It is suitable for situations where vehicle objects are significantly missing or damaged throughout the entire image frame or video sequence, and can improve the overall integrity and accuracy of vehicle monitoring data. Specifically: First, perform object detection: , in, This embodiment uses the Faster R-CNN algorithm to represent object detection algorithms. This represents classifying detected objects into different object numbers, where i is the number. The generated object's position information (x, y) and its width w and height h are given. Represents object-level position information; This represents the process of data flow operations; Then calculate object-level similarity: , in, Represents object-level similarity. This represents the current multi-scale spatiotemporal feature encoding representation of the vehicle object. Represents the multi-scale spatiotemporal feature encoding of adjacent vehicle objects; Then, calculate object-level inter-frame similarity: , in, This represents the object-level inter-frame similarity between frame t and frame t+n. The multi-scale spatiotemporal feature encoding representation of the current vehicle at the object level in frame t. The multi-scale spatiotemporal feature encoding representation of the current vehicle at the object level in frame t+n.

[0027] S6. Long-term vehicle missing content repair and integration: Global and local fusion complement each other in the restoration of missing vehicle data over long time periods. Global fusion ensures the consistency and coherence of the overall data, while local fusion focuses on the details and contextual relationships of local data. Combining these two fusion methods can improve the quality of restoration results, making data analysis and prediction more accurate and reliable. Specifically: , , This represents the enhanced features resulting from fusion. Representative features are superimposed along the channel dimension. Represents dimensionality reduction using convolutional layers. Represents matrix multiplication. This represents the final, repaired video output. This represents converting features into an output RGB image. Represents the loss in the reconstruction of spatial relationships. This represents the loss for reconstructing temporal relationships, and the loss function is L1+adversarial loss.

[0028] S7, Relational Adversarial Networks: Adversarial Relationship Networks (ARNs) maintain consistency in generated video sequences by introducing adversarial training of vehicle relationships (spatial and temporal) between the generator and discriminator. ARNs model dynamic scenes within the video sequences. By introducing adversarial training of vehicle relationships between the generator (GNet) and discriminator (DNet), the generator learns vehicle relationship information within the video sequence, thereby better simulating changes in dynamic scenes and generating more realistic and dynamic video data. The loss function of ARNs, which minimizes the difference between the generator's output and the discriminator's evaluation, is set as follows: , in, To maximize the training objective in the discriminator DNet, so that the generated vehicle data can more closely resemble real data, The goal of minimizing the training objective of the generator GNet is to generate vehicle data that is indistinguishable from real vehicles. By alternately optimizing the loss functions of the generator GNet and the discriminator DNet, the relation adversarial network can learn the relationships between data and generate new data samples with realistic relationships. During training, the generator and discriminator compete with each other to improve the generator's ability to generate realistic samples, while making the discriminator have better discrimination ability, thereby achieving better relation modeling results.

[0029] S8, Network Training and Testing: Generator training, discriminator training, and temporal adversarial training are performed separately. The trained generator is evaluated using a validation set or test set, and optimization and improvement are carried out based on the evaluation results. Specifically: Generator Training: Use the generator GNet to generate Fake restored video data and calculate the spatial relationship reconstruction loss. Loss of reconstructing temporal relationships By minimizing these two loss functions, the generator's parameters are updated to produce more reasonable and realistic restoration results. Discriminator training: Using the Fake restored video data generated by the generator and the Real video data in the training set as input, the parameters of the discriminator are updated so that it can accurately evaluate the realism of the restored video data generated by the generator. Temporal adversarial training: Alternately update the parameters of the generator and discriminator, and minimize the difference between the generator's output and the discriminator's evaluation. Adversarial training can be performed using optimization algorithms in adversarial training, such as the Adam optimizer in Generative Adversarial Networks (GANs). The performance of the trained generator GNet is evaluated using a validation set or a test set, including the realism of the restored results, temporal consistency, and consistency metrics of relational complementary constraints. The model is then tuned and improved based on the evaluation results.

[0030] The generator GNet trained in this embodiment can be applied to actual road video surveillance data restoration tasks to generate high-quality restored video data for subsequent monitoring and analysis applications.

[0031] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A method for restoring road video surveillance data based on relational complementary constraints, characterized in that, Specifically, the following steps are included: S1. Collect video data containing road traffic monitoring, and label the collected data with vehicle location information. Then, introduce missing or corrupted data into the labeled video data to obtain a constructed dataset. Finally, divide the constructed dataset into training set, validation set and test set. S2. Input the video sequences in the dataset into the backbone network Transformer. First, perform multi-scale spatial and temporal decomposition, decompose the extracted original features into sub-features of different scales, encode each sub-feature, and finally fuse and integrate the encoded sub-features to obtain a multi-scale spatiotemporal feature encoding representation. S3. Graph modeling based on feature similarity (SIM): The relationships between nodes are modeled as a similarity metric (SIM) to obtain the relationships between nodes; the relationships between nodes are as follows: ， Where i and j represent the position indices of pixels in the node features. Represents a similarity matrix; Represents the current node. Represents adjacent nodes. This represents the similarity measure between nodes, which is implemented using the L2 distance function. This indicates normalization processing; S4. Based on the multi-scale spatiotemporal feature encoding representation obtained in step S2, vehicle information within the local region of the video sequence is restored through processing. The specific process is as follows: , in, Represents regional similarity. This represents the current regional-level multi-scale spatiotemporal feature encoding representation. Represents multi-scale spatiotemporal feature encoding at the adjacent region level; S5. Object detection algorithms are used to detect objects in the video sequence, obtaining the object's location information, width, and height. After converting the image-level location information to object-level location information, object-level similarity and object-level inter-frame similarity are calculated to repair the entire vehicle object. The process of using object detection algorithms to detect objects in video frames is as follows: ， in, Representative object detection algorithms, This represents classifying detected objects into different object numbers, where i is the number. The generated object's position information (x, y) and its width w and height h are given. Represents object-level position information; This represents the process of data flow operations; The process of calculating object-level similarity is as follows: ， in, Represents object-level similarity. This represents the current multi-scale spatiotemporal feature encoding representation of the vehicle object. Represents the multi-scale spatiotemporal feature encoding of adjacent vehicle objects; The process of calculating object-level inter-frame similarity is as follows: ， in, This represents the object-level inter-frame similarity between frame t and frame t+n. The multi-scale spatiotemporal feature encoding representation of the current vehicle at the object level in frame t. The multi-scale spatiotemporal feature encoding representation of the current vehicle at the object level in frame t+n; S6. Fuse the repair results of steps S4 and S5 to obtain the enhanced features after fusion; S7. A relational adversarial network is used to model dynamic scenes in video sequences. By introducing adversarial training of vehicle relationships between the generator and the discriminator, the generator learns the vehicle relationship information in the video sequence, thereby simulating changes in dynamic scenes and generating new video data. S8. Perform generator training, discriminator training, and temporal adversarial training respectively. Use the validation set or test set to evaluate the trained generator and optimize and improve it based on the evaluation results.

2. The method for restoring road video surveillance video data based on relational complementary constraints according to claim 1, characterized in that, The video data containing road traffic monitoring mentioned in step S1 can be an existing video dataset or video data collected at actual road intersections or road sections.

3. The method for restoring road video surveillance data based on relational complementary constraints according to claim 1, characterized in that, The multi-scale spatiotemporal feature encoding described in step S2 is represented as follows: ， Here, Devide(Frames) represents multi-scale spatial and temporal decomposition of the input video frames, decomposing the original features into sub-features of different scales; Mask represents adding a mask to the corresponding region to generate an image with lost vehicle information; Encoder represents encoding each sub-feature and extracting its important feature information; Fuse represents fusing and integrating the encoded sub-features, i.e., the outputs of the Transformer side of layers 2, 3, and 4.

4. The method for restoring road video surveillance video data based on relational complementary constraints according to claim 3, characterized in that, The enhanced features after fusion described in step S6 are: ， ， This represents the enhanced features resulting from fusion. Representative features are superimposed along the channel dimension. Represents dimensionality reduction using convolutional layers. Represents matrix multiplication. 'e' represents the final, repaired video output. This represents converting features into an output RGB image. Represents the loss in the reconstruction of spatial relationships. This represents the loss for reconstructing temporal relationships, and the loss function is L1+adversarial loss.

5. The method for restoring road video surveillance data based on relational complementary constraints according to claim 4, characterized in that, The loss function used by the relational adversarial network in step S7, which minimizes the difference between the generator's generated result and the discriminator's evaluated result, is set as follows: ， in, To maximize the training objective in the discriminator DNet, Minimize the training objective of the generator GNet.

6. The method for restoring road video surveillance video data based on relational complementary constraints according to claim 5, characterized in that, The specific processes of generator training, discriminator training, and temporal adversarial training in step S8 are as follows: Generator Training: Use the generator GNet to generate Fake restored video data and calculate the spatial relationship reconstruction loss. Loss of reconstructing temporal relationships By minimizing these two loss functions, the generator parameters are updated to obtain a trained generator; Discriminator training: Using Fake video data generated by the generator and Real video data from the training set as input, the parameters of the discriminator are updated to evaluate the realism of the Fake video data generated by the generator. Temporal adversarial training: Alternately update the parameters of the generator and discriminator, and minimize the difference between the generator's output and the discriminator's evaluation. To conduct competitive training.

Images