Vehicle throwing object detecting and positioning method and system based on multi-channel spatial-temporal feature fusion

Abstract

The invention provides a vehicle throwing object detection and positioning method and system based on multi-channel spatial-temporal feature fusion, and the method comprises the steps: obtaining a continuous time sequence image frame sequence of a road scene, and generating an optical flow image sequence through an optical flow algorithm; outputting a detection frame of each vehicle in each standardized image through a deep neural network target detection model, and generating a region of interest according to the detection frames; a fusion feature vector is generated for each region of interest, a space-time fusion feature sequence is constructed, and a thrown object classification result and a positioning result of each region of interest are generated based on the space-time fusion feature sequence and the multi-branch full-connection network; and based on the classification result and the positioning result of the thrown object, inputting the obtained coordinates of the suspected area of the thrown object into a spherical camera for tracking. According to the method, the thrown object can be efficiently and accurately detected and positioned automatically, the accuracy and real-time performance of thrown object detection are improved, the false alarm rate is reduced, and the precision degree of positioning the position of the thrown object is improved.

Description

Technical Field

[0001] This application relates to the field of spillage detection technology, and in particular to a method and system for detecting and locating vehicle spillage based on multi-channel spatiotemporal feature fusion. Background Technology

[0002] With increasing road traffic volume, the incidence of cargo and debris spillage during motor vehicle transportation is gradually rising. This spillage not only poses traffic safety hazards but also increases the difficulty of road maintenance and management. Therefore, it is necessary to detect and locate spilled materials to facilitate timely removal.

[0003] In related technologies, traditional methods for detecting and locating spilled materials mainly rely on manual on-site inspections, which are inefficient and highly subjective. In recent years, automatic detection methods based on conventional video analysis have gradually emerged, but these detection schemes suffer from drawbacks in practical applications, such as high rates of missed detections and false alarms, as well as inaccurate location of spilled materials. Summary of the Invention

[0004] This application aims to at least partially address one of the technical problems in the related art.

[0005] Therefore, the first objective of this application is to propose a vehicle spill detection and positioning method based on multi-channel spatiotemporal feature fusion. This method can perform efficient and accurate automated detection and positioning of spills, improve the accuracy and real-time performance of spill detection, reduce the false alarm rate, and improve the accuracy of spill location positioning.

[0006] The second objective of this application is to propose a vehicle spillage detection and positioning method system based on multi-channel spatiotemporal feature fusion.

[0007] The third objective of this application is to propose an electronic device.

[0008] The fourth objective of this application is to provide a computer-readable storage medium.

[0009] To achieve the above objectives, the first aspect of this application is to propose a method for detecting and locating vehicle spillage based on multi-channel spatiotemporal feature fusion, comprising the following steps:

[0010] A continuous temporal image frame sequence of a road scene is acquired, the continuous temporal image frame sequence is normalized, and an optical flow algorithm is used to process the normalized continuous temporal image frame sequence to generate an optical flow image sequence.

[0011] The deep neural network target detection model outputs the detection box of each vehicle in the standardized continuous temporal image frame sequence, and generates the region of interest corresponding to each vehicle based on the detection box.

[0012] For each region of interest, features in the temporal image and features in the optical flow image are fused to generate a fused feature vector. Based on the fused feature vector and a preset time sliding window, a spatiotemporal fused feature sequence is constructed. Based on the spatiotemporal fused feature sequence and a multi-branch fully connected network, the projectile classification result and localization result for each region of interest are generated.

[0013] Based on the classification results of the spilled material, a target region of interest is determined, and the coordinates of the target region of interest are corrected based on the positioning results. The coordinates of the suspected spilled material area are then input into a spherical camera for tracking to locate the position coordinates of the spilled material.

[0014] Optionally, the step of outputting the detection box for each vehicle in the normalized continuous temporal image frame sequence includes: inputting all normalized images in the normalized continuous temporal image frame sequence into the deep neural network target detection model, and outputting the detection box for each vehicle in each normalized image frame, wherein the detection box adopts a format of center point coordinates and side lengths; the step of generating a region of interest corresponding to each vehicle based on the detection box includes: expanding the area of ​​the detection box based on the center point coordinates and side lengths to generate the region of interest, wherein if the expanded region exceeds the original region of the normalized image, the boundary information of the original region is filled.

[0015] Optionally, the step of fusing features in the temporal image and features in the optical flow image for each region of interest to generate a fused feature vector includes: cropping region sub-images corresponding to the region of interest from the temporal image and the optical flow image respectively; inputting the region sub-images from the temporal image and the optical flow image into the backbone network and the optical flow feature branch network of a convolutional neural network respectively to obtain image feature vectors and motion feature vectors; and fusing the image feature vectors and the motion feature vectors into the fused feature vector.

[0016] Optionally, the multi-branch fully connected network includes a classification branch and a regression branch. Constructing a spatiotemporal fusion feature sequence based on the fusion feature vector and a preset time window includes: within the time window, supplementing the fusion feature vectors of the same target according to target identifiers or spatiotemporal proximity relationships to generate the spatiotemporal fusion feature sequence; generating the debris classification and localization results for each region of interest based on the spatiotemporal fusion feature sequence and the multi-branch fully connected network includes: inputting the spatiotemporal fusion feature sequence into a temporal modeling network to generate a high-dimensional temporal feature vector; inputting the high-dimensional temporal feature vector into the multi-branch fully connected network, outputting the binary classification probability of the debris in the region of interest through the classification branch, and outputting the coordinate correction amount of the region of interest through the regression branch.

[0017] Optionally, determining the target region of interest based on the spill classification result includes: selecting regions of interest with a binary classification probability greater than a preset probability threshold as candidate regions of interest, and selecting the target region of interest from the candidate regions of interest; correcting the coordinates of the target region of interest based on the positioning result, and inputting the obtained coordinates of the suspected spill area into the spherical camera for tracking includes: correcting the initial coordinates of the region of interest using the coordinate correction amount to obtain the coordinates of the suspected spill area; and sending the coordinates of the suspected spill area to the spherical camera through an API interface with the spherical camera.

[0018] Optionally, acquiring a continuous temporal image frame sequence of the road scene includes: continuously acquiring a video stream of the road scene through a road video surveillance device, and periodically extracting frames from the video stream according to a set frame rate.

[0019] Optionally, the step of generating an optical flow image sequence by processing the normalized continuous temporal image frame sequence using an optical flow algorithm includes: inserting a zero-optical flow image into the first frame of the optical flow image sequence to be generated; calculating the dense optical flow between each frame in the normalized continuous temporal image frame sequence and the adjacent previous frame using an optical flow algorithm to generate a corresponding optical flow image; and performing normalization and filtering processing on the corresponding optical flow image based on the object motion scene reflected by the optical flow image.

[0020] To achieve the above objectives, a second aspect of this application also proposes a vehicle spillage detection and positioning system based on multi-channel spatiotemporal feature fusion, comprising the following modules:

[0021] The generation module is used to acquire a continuous temporal image frame sequence of a road scene, perform normalization processing on the continuous temporal image frame sequence, and perform optical flow algorithm on the normalized continuous temporal image frame sequence to generate an optical flow image sequence.

[0022] The detection module is used to output the detection box of each vehicle in the standardized continuous temporal image frame sequence through a deep neural network target detection model, and generate the region of interest corresponding to each vehicle based on the detection box;

[0023] The fusion module is used to fuse features in the temporal image and features in the optical flow image for each region of interest to generate a fused feature vector, and to construct a spatiotemporal fusion feature sequence based on the fused feature vector and a preset time sliding window. Based on the spatiotemporal fusion feature sequence and a multi-branch fully connected network, the module generates the projectile classification result and localization result for each region of interest.

[0024] The positioning module is used to determine the target region of interest based on the classification result of the spilled material, and to correct the coordinates of the target region of interest based on the positioning result. The obtained coordinates of the suspected area of ​​spilled material are then input into a spherical camera for tracking to locate the position coordinates of the spilled material.

[0025] To achieve the above objectives, a third aspect of this application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the vehicle spillage detection and positioning method based on multi-channel spatiotemporal feature fusion as described in any one of the first aspects above.

[0026] To achieve the above objectives, the fourth aspect of this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the vehicle spillage detection and positioning method based on multi-channel spatiotemporal feature fusion as described in any one of the first aspects.

[0027] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects: This application performs spatial and motion multimodal feature fusion, combining original image features and inter-frame optical flow features to compensate for the lack of single-channel feature information, and significantly improves the detection sensitivity for dynamic small targets, the separation of the object itself and the spilled material. Furthermore, it utilizes a temporal deep learning network to model the spilling process, using LSTM and other temporal neural networks to model the spatial-motion joint feature sequence of the vehicle's region of interest, achieving deep memory of the instantaneous spilling action and effectively reducing missed detections and false judgments. It also performs vehicle-level ROI dynamic expansion and instance tracking, automatically expanding the ROI range and dynamically adjusting the target detection area using the vehicle target as an anchor point, which can improve the probability of detecting anomalies while reducing background interference. Moreover, it precisely links the PTZ camera for automatic tracking and zooming. After detecting a high-confidence suspected area of ​​spilled material, it automatically and precisely guides the PTZ camera to rotate and zoom, acquiring higher resolution detail images, realizing a closed-loop process from discovery to tracking, to localization, and to verification. This application facilitates engineering deployment in practical applications. The algorithm is easy to integrate end-to-end, and the entire process can be seamlessly integrated with existing road video surveillance systems. It supports rapid deployment on general-purpose GPU / AI servers or edge boxes, making it easy to implement in real-world applications. Therefore, this application achieves efficient and accurate automated detection and location of debris thrown by motor vehicles during road traffic monitoring scenarios. This improves the accuracy and real-time performance of debris detection, reduces false alarm rates, and can be linked with camera equipment for tracking and detailed identification, improving the accuracy of debris location and facilitating traffic safety management and emergency response.

[0028] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0029] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0030] Figure 1 This is a flowchart of a vehicle spill detection and localization method based on multi-channel spatiotemporal feature fusion proposed in an embodiment of this application;

[0031] Figure 2 This is a schematic diagram illustrating the principle of a specific vehicle spillage detection and localization method based on multi-channel spatiotemporal feature fusion proposed in an embodiment of this application.

[0032] Figure 3 This is a flowchart of a feature fusion method proposed in an embodiment of this application;

[0033] Figure 4 This is a flowchart of a temporal feature modeling and classification localization method proposed in an embodiment of this application;

[0034] Figure 5 This is a schematic diagram of the structure of a vehicle spillage detection and positioning system based on multi-channel spatiotemporal feature fusion proposed in an embodiment of this application. Detailed Implementation

[0035] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0036] It should be noted that the automatic detection schemes for spilled materials based on video analysis in the relevant embodiments mainly fall into the following technical categories:

[0037] First, the single-frame object detection method. This method uses deep learning object detection models (such as YOLO, SSD, and Faster R-CNN) to directly detect abnormal objects or debris in road images. This type of method is effective for stationary objects and large-scale objects, but it is prone to missing detections of sudden, dynamic debris; and small objects are easily missed in complex backgrounds.

[0038] Second, there is the method based on motion anomaly analysis. This method uses optical flow, background modeling, and inter-frame differencing to detect motion anomaly regions in the image, and then uses traditional image features or a subsequent discriminator to screen whether they are spilled objects. This method can capture abnormal dynamics, but it is difficult to adapt to high-speed moving vehicles and scenes with changing backgrounds, and it does not fully incorporate the vehicle-spraying relationship.

[0039] Third, target tracking is combined with detection. This method is based on vehicle tracking (such as Kalman filtering and SORT algorithm) and abnormal behavior discrimination, combined with target detection, to output suspicious areas. Some solutions use deep temporal models to improve tracking and situational analysis capabilities, but most are limited to vehicle behavior analysis and do not combine multimodal information for screening.

[0040] Fourth, PTZ camera or multi-lens linkage. Some systems support linkage with high-magnification PTZ cameras for local magnification image acquisition after a front-end alarm, but lack precise linkage and positioning algorithms with abnormal points of spillage, resulting in limited automation.

[0041] Therefore, the spill detection schemes in the above-mentioned embodiments have limitations in terms of spatial accuracy, temporal coherence analysis, motion anomaly interpretation and multimodal fusion in practical applications. They have a high false negative rate and lack a system-wide, closed-loop, and efficient solution.

[0042] To address this, this application proposes a method and system for detecting and locating vehicle spilled materials based on multi-channel spatiotemporal feature fusion. This application can achieve the following objectives: accurately distinguishing motion anomalies in scenarios where the spilled material is small and the target is easily blended into the background; fully utilizing spatiotemporal continuity to reduce the rate of missed detections and false alarms, and improving the adaptability to complex road conditions; and achieving automatic linkage with a high-magnification camera for fine-grained tracking, thereby improving the accuracy of spilled material location capture.

[0043] The following description, with reference to the accompanying drawings, describes a method and system for detecting and locating vehicle spilled materials based on multi-channel spatiotemporal feature fusion, as proposed in an embodiment of this application.

[0044] Figure 1 This is a flowchart of a vehicle spill detection and localization method based on multi-channel spatiotemporal feature fusion proposed in an embodiment of this application, as shown below. Figure 1 As shown, the method includes the following steps:

[0045] Step S101: Obtain a continuous temporal image frame sequence of the road scene, perform standardization processing on the continuous temporal image frame sequence, and perform optical flow algorithm on the standardized continuous temporal image frame sequence to generate an optical flow image sequence.

[0046] Specifically, this application first collects video data of the road scene to be detected and located, and then processes the video data to obtain a continuous temporal image frame sequence of the road scene. The spilled objects detected and located in this application refer to goods, debris, or other objects that fall, splash, or are scattered onto the road surface from vehicles due to vehicle movement, improper loading, or other reasons in the road traffic environment. These objects are typically small in size, complex in shape, and often temporary, dynamic targets.

[0047] In one embodiment of this application, obtaining a continuous temporal image frame sequence of a road scene includes: continuously acquiring a video stream of the road scene through a road video surveillance device, and periodically extracting frames from the video stream according to a set frame rate.

[0048] Specifically, existing video surveillance equipment in road monitoring systems, such as wide-angle cameras or pan-tilt-zoom (PTZ) cameras, can be used to continuously capture video streams of the road scene to be monitored. Then, the raw video data is periodically extracted at a set frame rate (e.g., 2fps to 10fps), and the extracted frames are then aggregated to obtain image frames, resulting in a continuous temporal image frame sequence: {I1, I2, ..., I...} T}, where I T This represents the images at different times T.

[0049] Furthermore, the continuous temporal image frame sequence can be standardized. For example, various preprocessing methods, including but not limited to geometric correction (such as distortion correction), resolution unification, and histogram equalization, can be used to perform standardization on each extracted image frame, thereby improving the generalization ability and robustness of subsequent algorithms.

[0050] Furthermore, based on the obtained standardized continuous temporal image frame sequence, optical flow algorithms are used to calculate and generate optical flow image sequences.

[0051] Optical flow describes the motion vector field caused by the projection of each pixel as an object moves between consecutive video frames. The magnitude and direction of optical flow can reflect the motion of objects in the scene, and are used to analyze dynamic behavior and anomalous events.

[0052] In one embodiment of this application, an optical flow image sequence is generated by performing an optical flow algorithm on a normalized continuous temporal image frame sequence. This includes: inserting a zero-optical flow image into the first frame of the optical flow image sequence to be generated; calculating the dense optical flow between each frame in the normalized continuous temporal image frame sequence and the adjacent previous frame using the optical flow algorithm to generate a corresponding optical flow image; and performing normalization and filtering processing on the corresponding optical flow image based on the object motion scene reflected by the optical flow image.

[0053] Specifically, the optical flow image sequence constructed in this application is used to describe the motion information of pixels in a spatiotemporal image sequence. Each pixel value in the optical flow image represents the direction and magnitude of the pixel's motion between two frames. Since optical flow is calculated based on the difference between adjacent frames, the number of image frames in the optical flow image sequence is one less than that in the temporal image frame sequence. For an input N-frame temporal image sequence, N-1 optical flow images are generally obtained. To ensure that the generated optical flow image sequence has the same length as the original temporal image sequence, this application first pads the optical flow with zeros in the first frame. That is, an all-zero optical flow map is inserted into the first frame of the optical flow image sequence. This all-zero optical flow map indicates that there is no motion information before the first frame.

[0054] Then, using mainstream optical flow algorithms, dense optical flow calculations are performed on each frame to be analyzed (i.e., adjacent frames), calculating the relationship between each frame and its previous frame (I1). T-1 I T By analyzing the dense optical flow between the two frames, we can obtain optical flow images for each frame, thus obtaining a complete optical flow matrix sequence.

[0055] For example, mainstream optical flow methods such as Farneback, PWC-Net, and RAFT can be used to output a direction-amplitude dual-channel or pseudo-color image encoded optical flow matrix: {F1,F2,...,F...} T This is used to reflect the apparent motion information of the target area.

[0056] Finally, optical flow images can be selectively normalized and filtered. For example, optical flow images that reflect the motion of small objects under dynamic backgrounds or large scene changes can be normalized and filtered to suppress the negative impact of dynamic backgrounds or large scene changes on the detection of abnormal motion of small objects.

[0057] Step S102: Using a deep neural network target detection model, output the detection box of each vehicle in the standardized continuous temporal image frame sequence, and generate the region of interest corresponding to each vehicle based on the detection box.

[0058] Specifically, vehicles are detected in the image, and regions of interest (ROIs) are generated based on the detected vehicle regions. A region of interest refers to a key area in an image or video that the detection model determines is likely to contain a target. In this application, the ROI typically refers to a certain range of regions extending outwards from the vehicle detection bounding box.

[0059] In one embodiment of this application, outputting the detection box of each vehicle in a continuous temporal image frame sequence after normalization includes: inputting all normalized images in the continuous temporal image frame sequence after normalization into a deep neural network target detection model, and outputting the detection box of each vehicle in each normalized image frame, wherein the detection box adopts the format of center point coordinates and the length of each side.

[0060] Specifically, existing historical road vehicle datasets stored in various databases are collected in advance, and these datasets are used as training datasets to train a deep neural network object detection model. For example, the YOLO11 model is preferred in this application. The YOLO model adopts an end-to-end real-time object detection algorithm, which can quickly detect multiple target objects in a single image. This application uses this model to automatically identify vehicle entities in traffic scenes.

[0061] Then, as Figure 2 As shown, each frame of the standardized image in the continuous temporal image frame sequence after standardization is input into the YOLO11 model in batches, and a detection box (Bbox) is output in real time for all vehicle instances in each frame of standardized image.

[0062] The detection box is used to describe the spatial location information of the target in the image. The format is usually the center point coordinates and width and height, or the top left corner coordinates and width and height. For example, the output detection box in this embodiment is: B_i^t=[x,y,w,h], where (x,y) are the center point coordinates of the rectangular bounding box, and (w,h) are the width and height of the rectangular bounding box.

[0063] Furthermore, for each vehicle's detection bounding box, a corresponding region of interest is generated.

[0064] In one embodiment of this application, generating a region of interest for each vehicle based on the detection box includes: expanding the area of ​​the detection box based on the center point coordinates and the length of each side to generate the region of interest, wherein if the expanded region exceeds the original region of the standardized image, the boundary information of the original region is filled in.

[0065] For example, in this embodiment, for each vehicle's B_i^t, the bounding box is expanded outward proportionally along the line containing the center point, so that the area of ​​the expanded region is 1.5 times or more of the original Bbox, resulting in the corresponding Region of Interest (ROI_i^t). When the expanded region exceeds the effective boundary of the normalized image, a filling algorithm such as mirror padding or zero padding is used to ensure that no boundary information is lost. That is, by using a filling-based boundary processing method, extra pixels are added to the image edges, so that the boundary pixels of the image are expanded into the filling area, thus avoiding the loss of boundary information in subsequent operations and ensuring that the boundary parts of the image are not ignored or lost during processing.

[0066] Step S103: For each region of interest, the features in the temporal image and the features in the optical flow image are fused to generate a fused feature vector. Based on the fused feature vector and a preset time sliding window, a spatiotemporal fused feature sequence is constructed. Based on the spatiotemporal fused feature sequence and a multi-branch fully connected network, the debris classification result and localization result of each region of interest are generated.

[0067] Specifically, for each region of interest obtained in the previous step, the features of that region in the temporal image and optical flow image are extracted and then fused.

[0068] To more clearly illustrate the specific implementation process of feature extraction and fusion in this application, the following example illustrates a feature fusion method proposed in one embodiment of this application. Figure 3 This is a flowchart of a feature fusion method proposed in an embodiment of this application, such as... Figure 3 As shown, the method includes the following steps:

[0069] Step S301: Extract the region sub-images corresponding to the region of interest from the temporal image and the optical flow image, respectively.

[0070] Specifically, for each region of interest (ROI_i^t), patch_imgi and patch_flowi are cropped from the temporal image and optical flow image, respectively. Patch_imgi is the image patch / small piece corresponding to ROI_i^t in the original RGB image, and patch_flowi is the optical flow data patch / small piece of the same size at the same location in the optical flow image. Patch_imgi and patch_flowi are precisely aligned to obtain a region sub-map patch(patch_imgi, patch_flowi), which is used for subsequent feature extraction and fusion.

[0071] Step S302: Input the region sub-images in the temporal image and optical flow image into the backbone network and optical flow feature branch network of the convolutional neural network, respectively, to obtain image feature vectors and motion feature vectors.

[0072] Specifically, the patch_imgi obtained in step S301 is input into the backbone convolutional neural network, such as ResNet, MobileNet, or a lightweight backbone convolutional neural network CNN preferred in this application, to extract the spatial content features of patch_imgi and obtain the d1-dimensional image feature vector f_imgi.

[0073] Furthermore, by inputting patch_flowi into the branch dedicated to optical flow features in the aforementioned convolutional neural network (CNN), a d2-dimensional vector f_flowi representing motion features is obtained.

[0074] Step S303: The image feature vector and motion feature vector are fused into a fused feature vector.

[0075] Specifically, the two feature vectors obtained above are fused into a unified feature vector, namely f_fused, through various fusion methods such as multilayer perceptron (MLP) fusion, feature-level fusion (concatenation), or the introduction of attention mechanisms (such as SE modules and CBAM). i =Fusion(f_imgi,f_flowi).

[0076] Furthermore, the obtained fused feature vectors are used to model temporal features, and then the constructed temporal features are used to classify and locate each region of interest (ROI_i^t) through a fully connected network (FC).

[0077] To more clearly illustrate the specific implementation process of classifying and locating regions of interest in this application, the following example illustrates a temporal feature modeling and classification location method proposed in one embodiment of this application. Figure 4 This is a flowchart of a temporal feature modeling and classification localization method proposed in an embodiment of this application, as shown below. Figure 4 As shown, the method includes the following steps:

[0078] Step S401: Within the time sliding window, the fusion feature vectors of the same target are supplemented according to the target identifier or spatiotemporal proximity relationship to generate a spatiotemporal fusion feature sequence.

[0079] Specifically, for each region of interest, within a set time window (e.g., time window K = 8-16 frames), the fused feature vector of the same target, such as a vehicle or a region, is padded according to the target instance ID or the spatiotemporally nearest neighbor relationship with the target instance to construct the spatiotemporal fused feature sequence of that target: {f_fused i ^(tK),...,f_fused i ^t}.

[0080] In step S102 above, during the detection of targets in the image, a unique identifier (ID) is assigned to each detected target. Therefore, when completing a vehicle sequence, the vehicle ID can be used to complete the spatiotemporal fusion feature sequence of the vehicle.

[0081] Step S402: Input the spatiotemporal fusion feature sequence into the temporal modeling network to generate a high-dimensional temporal feature vector.

[0082] Specifically, in this embodiment, when generating the classification and localization results of the spilled material for each region of interest using spatiotemporal fusion feature sequences and multi-branch fully connected networks, the (K, D)-dimensional spatiotemporal fusion feature sequences are first input into the temporal modeling network. For example, ... Figure 2 As shown, a Long Short-Term Memory (LSTM) network can be used to capture the autocorrelation of cross-frame motion, structure, and appearance dynamics in the region of interest, thereby outputting a high-dimensional temporal feature vector f_seqi^t.

[0083] Among them, the high-dimensional temporal feature vector f_seqi^t output by the temporal modeling network LSTM in this application is a high-dimensional vector that integrates the spatiotemporal dynamics of the region of interest within K frames. This vector contains all the information about the appearance, action, movement and appearance of the region of interest, which is convenient for subsequent classification and localization of the region of interest.

[0084] Step S403: Input the high-dimensional temporal feature vector into the multi-branch fully connected network, output the binary classification probability of the projectile in the region of interest through the classification branch, and output the coordinate correction amount of the region of interest through the regression branch.

[0085] Specifically, the high-dimensional temporal feature vector f_seqi^t is input into a pre-trained multi-branch fully connected network (FC). The two branches of the fully connected network output in parallel the binary classification probability p_throwi^t of the projectiles corresponding to the region of interest (ROI) (output by the sigmoid / softmax layer) and the coordinate correction ΔBboxi^t of the bounding box (Bbox) of the ROI (output by the regression branch). Thus, the classification results and localization correction results can be used to perform joint optimized detection and localization of the projectiles within the ROI.

[0086] In this application, the fully connected network adopts a multi-branch output design. For the classification branch, it is usually an FC layer (or two FC stacks). The last layer uses a sigmoid / softmax layer to output the classification probability, which represents the probability that the current ROI is a projectile or a non-projectile.

[0087] For the regression branch, another two-layer FC can be used. This branch outputs a vector with a scale of 4, namely [Δx,Δy,Δw,Δh], which is used to fine-tune the coordinates of the detection box Bbox initially determined in step S102 for the region of interest.

[0088] As an example, the loss function of the fully connected network is jointly optimized beforehand during the training phase. The classification branch uses binary cross-entropy or softmax loss, while the regression branch uses L1 / L2 or IoU class loss. The total loss of the model is a weighted sum of the two objectives. For the aforementioned K-frame spatiotemporal patch sequence, after passing through a temporal sequence network (LSTM) to obtain a high-dimensional vector f_seqi^t, this high-dimensional vector is input into the trained fully connected network for multi-branch joint prediction.

[0089] The operation flow for branch 1 (classification) is: fc_cls(f_seqi^t) → sigmoid / softmax → projectile probability; the operation flow for branch 2 (regression) is: fc_reg(f_seqi^t) → ΔBbox (coordinate offset).

[0090] Step S104: Determine the target region of interest based on the classification results of the spilled material, and correct the coordinates of the target region of interest based on the positioning results. Input the obtained coordinates of the suspected spilled material area into the spherical camera for tracking to locate the position coordinates of the spilled material.

[0091] Specifically, using the classification and positioning results obtained in the previous step, the existing spherical camera is linked to perform further dynamic tracking in order to accurately locate the actual position of the spilled material.

[0092] In one embodiment of this application, determining the target region of interest based on the classification result of the spilled material includes: selecting regions of interest with a binary classification probability of the spilled material greater than a preset probability threshold as candidate regions of interest, and selecting the target region of interest from the candidate regions of interest.

[0093] Specifically, for all the regions of interest obtained in step S102, those regions with p_throwi^t greater than the dynamic or set threshold θ are selected as candidate regions of interest. Then, the region with the highest probability or all of the candidate regions of interest are selected to enter the target pool, thereby filtering out the target regions of interest.

[0094] Furthermore, in this embodiment, the coordinates of the target region of interest are corrected based on the positioning results, and the obtained coordinates of the suspected spilled area are input into the spherical camera for tracking. This includes: correcting the initial coordinates of the region of interest by a coordinate correction amount to obtain the coordinates of the suspected spilled area; and sending the coordinates of the suspected spilled area to the spherical camera by interfacing with the application programming interface (API) of the spherical camera.

[0095] Specifically, for each target region of interest, the coordinate correction amount ΔBboxi^t obtained in the previous step is used to correct the region ROI_i^t initially determined in step S102, which is then used as the real-time coordinates of the suspected spilled material area. The real-time coordinates of the suspected spilled material area are then sent to the spherical camera via the application programming interface (API) of the existing PTZ spherical camera in the road monitoring system.

[0096] Furthermore, the spherical camera can be triggered to rotate and zoom to the corresponding target (i.e., the coordinates of the suspected area of ​​the spilled material) according to the optimal tracking strategy, and the relevant area can be magnified in real time to obtain a high-resolution image sequence. The close-up image of the target acquired by the spherical camera is then fed back to the second-level detection or assisted by manual high-confidence verification to further accurately locate the actual spilled material position, thereby realizing a fully automatic and semi-automatic closed-loop control system.

[0097] Therefore, this application extracts multimodal features by fusing the original image and optical flow, combined with the vehicle detection ROI; it uses deep temporal networks such as LSTM / Transformer to learn and discriminate dynamic features; it adopts a refined ROI expansion strategy to accommodate vehicle shape differences in actual engineering scenarios; and finally, it achieves a fully closed-loop detection and positioning process from detection to focusing to detailed inspection by automatically connecting with the API of PTZ cameras or intelligent gimbals.

[0098] This application is applicable to work scenarios in traffic-intensive areas such as highways, urban arterial roads, tunnels, and bridges. It can be deployed on static video surveillance, cloud-edge collaborative AI platforms, or smart sensing boxes. Furthermore, this application is easily expandable, its algorithm model is compatible with mainstream AI chip platforms, and it can be linked with IoT front-end devices. Parameters such as detection granularity and thresholds can be flexibly adjusted according to actual conditions, and it supports integration with road intelligent robots and emergency response systems.

[0099] In summary, the vehicle spill detection and localization method based on multi-channel spatiotemporal feature fusion in this application integrates spatial and motion multimodal features, combining original image features with inter-frame optical flow features to compensate for the insufficient single-channel feature information, significantly improving the detection sensitivity for dynamic small targets and separating the vehicle body from the spilled material. Furthermore, it utilizes a temporal deep learning network to model the spilling process, employing LSTM and other temporal neural networks to model the spatial-motion joint feature sequence of the vehicle's region of interest, achieving deep memory of instantaneous spilling actions and effectively reducing missed detections and false positives. It also performs vehicle-level ROI dynamic expansion and instance tracking, automatically expanding the ROI range and dynamically adjusting the target detection area using the vehicle target as an anchor point, which improves the probability of anomaly detection while reducing background interference. Moreover, it precisely links the PTZ camera for automatic tracking and zooming; after detecting a high-confidence suspected spilled material area, it automatically and precisely guides the PTZ camera to rotate and zoom, acquiring higher-resolution detail images, realizing a closed-loop process from discovery to tracking, localization, and verification. This method facilitates engineering deployment in practical applications. The algorithm is easy to integrate end-to-end, and the entire process can be seamlessly integrated with existing road video surveillance systems. It supports rapid deployment on general-purpose GPU / AI servers or edge boxes, making it easy to implement in real-world applications. Therefore, this method achieves efficient and accurate automated detection and location of debris thrown by motor vehicles during road traffic monitoring scenarios. It improves the accuracy and real-time performance of debris detection, reduces false alarm rates, and can be linked with camera devices for tracking and detailed identification, improving the accuracy of debris location and facilitating traffic safety management and emergency response.

[0100] To achieve the above embodiments, this application also proposes a vehicle spillage detection and positioning system based on multi-channel spatiotemporal feature fusion. Figure 5 This is a schematic diagram of the structure of a vehicle spillage detection and positioning system based on multi-channel spatiotemporal feature fusion proposed in an embodiment of this application, as shown below. Figure 5 As shown, the system includes:

[0101] The generation module 100 is used to acquire a continuous temporal image frame sequence of a road scene, perform normalization processing on the continuous temporal image frame sequence, and perform optical flow algorithm on the normalized continuous temporal image frame sequence to generate an optical flow image sequence.

[0102] The detection module 200 is used to output the detection box of each vehicle in the standardized continuous temporal image frame sequence through a deep neural network target detection model, and generate the region of interest corresponding to each vehicle based on the detection box.

[0103] The fusion module 300 is used to fuse features in the temporal image and features in the optical flow image for each region of interest to generate a fused feature vector, and to construct a spatiotemporal fusion feature sequence based on the fused feature vector and a preset time sliding window. Based on the spatiotemporal fusion feature sequence and a multi-branch fully connected network, the module generates the debris classification result and localization result for each region of interest.

[0104] The positioning module 400 is used to determine the target region of interest based on the classification result of the spilled material, correct the coordinates of the target region of interest based on the positioning result, and input the obtained coordinates of the suspected area of ​​spilled material into the spherical camera for tracking in order to locate the position coordinates of the spilled material.

[0105] It should be noted that the explanation of the above-described embodiment of the vehicle spillage detection and positioning method based on multi-channel spatiotemporal feature fusion also applies to the system of this embodiment, and will not be repeated here.

[0106] In summary, the vehicle spillage detection and positioning system based on multi-channel spatiotemporal feature fusion in this application embodiment achieves efficient and accurate automated detection and positioning of spillage from motor vehicles during driving in road traffic monitoring scenarios. This improves the accuracy and real-time performance of spillage detection, reduces the false alarm rate, and can be linked with camera equipment for tracking and detailed identification, thereby improving the accuracy of spillage positioning. This is beneficial for traffic safety management and emergency response.

[0107] To implement the above embodiments, this application also proposes an electronic device, comprising: at least one processor; and

[0108] A memory communicatively connected to at least one processor; wherein the memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable at least one processor to perform the vehicle spill detection and location method based on multi-channel spatiotemporal feature fusion as described in any of the first aspects above.

[0109] To implement the above embodiments, this application also proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the vehicle spillage detection and positioning method based on multi-channel spatiotemporal feature fusion as described in any one of the first aspects of the embodiments above.

[0110] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0111] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0112] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0113] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0114] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0115] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0117] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for detecting and locating vehicle spillage based on multi-channel spatiotemporal feature fusion, characterized in that, Includes the following steps: A continuous temporal image frame sequence of a road scene is acquired, the continuous temporal image frame sequence is normalized, and an optical flow algorithm is used to process the normalized continuous temporal image frame sequence to generate an optical flow image sequence. The deep neural network target detection model outputs the detection box of each vehicle in the standardized continuous temporal image frame sequence, and generates the region of interest corresponding to each vehicle based on the detection box. For each region of interest, features in the temporal image and features in the optical flow image are fused to generate a fused feature vector. Based on the fused feature vector and a preset time sliding window, a spatiotemporal fused feature sequence is constructed. Based on the spatiotemporal fused feature sequence and a multi-branch fully connected network, the projectile classification result and localization result for each region of interest are generated. Based on the classification results of the spilled material, a target region of interest is determined, and the coordinates of the target region of interest are corrected based on the positioning results. The coordinates of the suspected spilled material area are then input into a spherical camera for tracking to locate the position coordinates of the spilled material.

2. The method according to claim 1, characterized in that, The step of outputting the detection bounding box for each vehicle in the normalized continuous temporal image frame sequence includes: All standardized images in the continuous temporal image frame sequence after the standardization process are input into the deep neural network target detection model, and the detection box of each vehicle in each standardized image frame is output. The detection box adopts the format of center point coordinates and the length of each side. The step of generating a region of interest for each vehicle based on the detection box includes: Based on the coordinates of the center point and the lengths of each side, the area of ​​the detection box is expanded to generate the region of interest. In cases where the expanded region exceeds the original region of the standardized image, the boundary information of the original region is filled in.

3. The method according to claim 1, characterized in that, For each region of interest, the features in the temporal image and the features in the optical flow image are fused to generate a fused feature vector, including: From the temporal image and the optical flow image, respectively, crop out the region sub-image corresponding to the region of interest; The region sub-images in the temporal image and optical flow image are respectively input into the backbone network and optical flow feature branch network of the convolutional neural network to obtain image feature vectors and motion feature vectors; The image feature vector and the motion feature vector are fused into the fused feature vector.

4. The method according to claim 1, characterized in that, The multi-branch fully connected network includes a classification branch and a regression branch. The step of constructing a spatiotemporal fusion feature sequence based on the fused feature vector and a preset time sliding window includes: Within the time sliding window, the fusion feature vectors of the same target are supplemented according to the target identifier or spatiotemporal proximity relationship to generate the spatiotemporal fusion feature sequence; The process of generating waste classification and localization results for each region of interest based on the spatiotemporal fusion feature sequence and multi-branch fully connected network includes: The spatiotemporal fusion feature sequence is input into a temporal modeling network to generate a high-dimensional temporal feature vector. The high-dimensional temporal feature vector is input into the multi-branch fully connected network. The classification branch outputs the binary classification probability of the projectile in the region of interest, and the regression branch outputs the coordinate correction of the region of interest.

5. The method according to claim 4, characterized in that, The determination of the target region of interest based on the classification results of the spilled material includes: The regions of interest whose binary classification probability of the spilled material is greater than a preset probability threshold are used as candidate regions of interest, and the target region of interest is selected from the candidate regions of interest. The step of correcting the coordinates of the target region of interest based on the positioning result and inputting the obtained coordinates of the suspected area of ​​the spilled material into a spherical camera for tracking includes: The initial coordinates of the region of interest are corrected using the coordinate correction amount to obtain the coordinates of the suspected area of ​​the spilled material. By interfacing with the spherical camera's application programming interface (API), the coordinates of the suspected area of ​​the spilled material are sent to the spherical camera.

6. The method according to claim 1, characterized in that, The acquisition of a continuous temporal image frame sequence of the road scene includes: The video stream of the road scene is continuously collected by the road video surveillance equipment, and the video stream is periodically extracted according to the set frame rate.

7. The method according to claim 1, characterized in that, The step of generating an optical flow image sequence by processing a normalized continuous temporal image frame sequence using an optical flow algorithm includes: Insert a zero-optical-flow image into the first frame of the optical-flow image sequence to be generated; The dense optical flow between each frame in the normalized continuous temporal image frame sequence and the adjacent previous frame is calculated using an optical flow algorithm to generate the corresponding optical flow image. Based on the object motion scene reflected by the optical flow image, the corresponding optical flow image is normalized and filtered.

8. A vehicle spillage detection and positioning system based on multi-channel spatiotemporal feature fusion, characterized in that, Includes the following modules: The generation module is used to acquire a continuous temporal image frame sequence of a road scene, perform normalization processing on the continuous temporal image frame sequence, and perform optical flow algorithm on the normalized continuous temporal image frame sequence to generate an optical flow image sequence. The detection module is used to output the detection box of each vehicle in the standardized continuous temporal image frame sequence through a deep neural network target detection model, and generate the region of interest corresponding to each vehicle based on the detection box; The fusion module is used to fuse features in the temporal image and features in the optical flow image for each region of interest to generate a fused feature vector, and to construct a spatiotemporal fusion feature sequence based on the fused feature vector and a preset time sliding window. Based on the spatiotemporal fusion feature sequence and a multi-branch fully connected network, the module generates the projectile classification result and localization result for each region of interest. The positioning module is used to determine the target region of interest based on the classification result of the spilled material, and to correct the coordinates of the target region of interest based on the positioning result. The obtained coordinates of the suspected area of ​​spilled material are then input into a spherical camera for tracking to locate the position coordinates of the spilled material.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the vehicle spillage detection and location method based on multi-channel spatiotemporal feature fusion as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the vehicle spillage detection and positioning method based on multi-channel spatiotemporal feature fusion as described in any one of claims 1-7.

Images