A 360-degree monitoring vehicle re-identification method based on infrared imaging

Abstract

This invention belongs to the field of vehicle re-identification technology and relates to a 360-degree monitoring vehicle re-identification method based on infrared imaging. First, a 360-degree infrared monitoring vehicle re-identification video dataset for urban areas is constructed and its features are extracted. Then, 360-degree vehicle detection is performed based on spherical convolution, and the detection results are subjected to secondary projection. Next, pseudo-infrared monitoring images are generated based on adversarial domain adaptation, and dual-stream vehicle re-detection features are extracted. Finally, an attention mechanism is used to weight the two types of vehicle features, and the vehicle re-identification result is output. This method enables intelligent vehicle re-identification, helps to broaden the monitoring range of vehicle re-identification, and can introduce infrared data in a multimodal form to ensure the accuracy of vehicle re-identification. It also overcomes the problem of decreased accuracy of vehicle re-identification algorithms in complex scenarios.

Description

Technical Field

[0001] This invention belongs to the field of vehicle re-identification technology and relates to a 360-degree monitoring vehicle re-identification method based on infrared imaging. Background Technology

[0002] Vehicle re-identification searches for the same vehicle from massive amounts of video data given an image of the vehicle. Essentially, vehicle re-identification is very similar to pedestrian re-identification; however, it faces more complex situations and requires consideration of more practical considerations. Currently, vehicle re-identification is mainly used for searching and tracking suspect vehicles in criminal investigations, traffic big data analysis, parking lot fee collection, and vehicle counting.

[0003] With the development of deep learning technology and the emergence of large datasets, vehicle re-identification has become a very hot research direction in the fields of computer vision and multimedia. With the rapid development of smart cities, China has basically achieved full coverage of surveillance equipment, especially the "Skynet" system, which integrates cameras from all areas into a unified system for management. How to effectively utilize these surveillance cameras is a very worthwhile research area.

[0004] Currently, equipment for 360-degree monitoring is gradually being deployed in areas requiring wide-range surveillance because it provides broader coverage and eliminates blind spots. However, research on vehicle detection based on 360-degree infrared imaging monitoring is almost nonexistent, primarily due to its greater complexity compared to single-angle monitoring. Summary of the Invention

[0005] To address the aforementioned issues, this invention proposes a 360-degree vehicle re-identification method based on infrared imaging. This method overcomes the problem of low-light detection failure in 360-degree infrared imaging monitoring, while also overcoming the problem of limited coverage in traditional fixed-view 2D prediction vehicle re-identification.

[0006] To achieve the above objectives, the specific process of vehicle re-identification in this invention is as follows:

[0007] (1) Collect a 360-degree vehicle detection dataset and use the domain adaptive method to generate pseudo-infrared monitoring data as a 360-degree infrared monitoring vehicle re-identification video dataset.

[0008] (2) The video dataset in step (1) is projected into ERP to obtain ERP images. Based on the distortion invariance of the sphere sampling strategy, the spherical pixels are sampled into planar convolutions. A CNN network is used to extract the 360-degree infrared monitoring vehicle re-identification video features.

[0009] (3) The vehicle location information, as well as the width and height of the vehicle, are obtained by processing 360-degree video based on spherical convolution;

[0010] (4) Based on the vehicle location information and the width and height of the vehicle obtained in step (3), the vehicle monitored by the sphere is projected into a 2D vehicle image by projection, and then cropped by width and height to obtain a refined vehicle block.

[0011] (5) Based on the refined vehicle block obtained in step (4), a pseudo-infrared monitoring image is generated by using a pseudo-infrared monitoring image generation method based on adversarial domain adaptation.

[0012] (6) Input the refined vehicle block obtained in step (4) and the pseudo-infrared monitoring image obtained in step (5) into a dual-stream ResNet network for feature extraction to obtain vehicle features based on RGB and NIR data respectively, and introduce a loss function. Improve the quality of generated pseudo-infrared surveillance images;

[0013] (7) Use an attention mechanism to weight the vehicle features obtained in step (6) based on RGB and NIR data, and output the vehicle re-identification results;

[0014] (8) Train and test the network. During the network testing phase, after locating the vehicle position and the size of the cropping region, calculate the similarity measure between different vehicle features based on RGB and NIR to determine whether the current vehicle is the vehicle being searched.

[0015] As a further technical solution of the present invention, the 360-degree vehicle detection dataset in step (1) adopts the VeRi-776, VehicleID and VERI-Wild datasets, and the vehicle position information is marked on the images in the dataset by manually marking points, and the projection data based on the position points is generated based on the vehicle position information, and then the outer bounding box is marked based on the projection data of the position points.

[0016] As a further technical solution of the present invention, the 2D convolution operation in the CNN network in step (2) is as follows:

[0017] ,

[0018] The operation of 3D convolution is as follows:

[0019] ,

[0020] in, Representative position Using 2D convolution; represent The neighborhood, Represents the weight parameter. Representative feature layer, Representative position Using 3D convolution, The transformation function representing the projection from the sphere to the tangent plane; This represents the transformation function from the projected tangent plane to the sphere.

[0021] As a further technical solution of the present invention, the specific process of step (3) is as follows:

[0022] (3-1) Input the ERP image from 360-degree infrared monitoring into the convolutional layer of the CNN network used in step (2). By introducing ConvBlock, the vehicle feature constraints are enhanced. After multiple ConvBlocks, the spherical coordinate position information is output. Projection is performed based on the spherical coordinate position information, and the outer bounding box generated by the projection is constrained by manual marking. The detailed operation is as follows:

[0023] ,

[0024] in, This represents the index image of the ERP system, representing the 360-degree infrared monitoring system. The representative will Encapsulate it in a form similar to ResBlock; represent The number of; Represents the vehicle's position coordinates on the sphere; This represents the width and height of the vehicle's outer bounding box after projection;

[0025] (3-2) After obtaining the vehicle's position coordinates and width and height information, the position loss function is used to calculate the loss between the vehicle's position on the sphere and the manually labeled position.

[0026] ,

[0027] in, The position loss function representing spherical coordinates, This represents the predicted spherical coordinate position information. The location coordinates are manually labeled.

[0028] The loss for the width and height of the outer bounding box is based on The loss between the bounding boxes w and h of the projected 2D vehicle image and the manually annotated w and h.

[0029] ,

[0030] in, This represents the loss function after projection; This represents the predicted width and height information. This represents the width and height information manually annotated.

[0031] As a further technical solution of the present invention, the specific process of step (4) is as follows: ,in, This represents vehicle location information generated based on the ERP image of the index frame; Representative based on The secondary projection produces an image containing the vehicle; Represents the refined vehicle block in the index.

[0032] As a further technical solution of the present invention, the specific process of step (5) is as follows: a trained network is obtained based on the existing infrared monitoring dataset, and the refined vehicle block obtained in step (4) is input. Generate infrared monitoring data ,Right now .

[0033] As a further technical solution of the present invention, the vehicle features based on RGB and NIR data in step (6) are as follows:

[0034] ,

[0035] ,

[0036] The loss function is:

[0037] ,

[0038] in, Represents the global average pooling layer. This represents the feature mapping layer, which maps feature dimensions to single-dimensional features. A matrix of 1 represents positive samples. The zero matrix represents negative samples.

[0039] As a further technical solution of the present invention, step (7) adopts an attention mechanism for weighting the process as follows:

[0040] ,

[0041] The output vehicle re-identification result is as follows:

[0042] ,

[0043] in, This represents the final output result. This represents a global average pooling layer, which transforms features into high-dimensional semantic information. This represents a linear mapping layer that maps high-dimensional information to low-dimensional information, thereby outputting vehicle re-identification results.

[0044] As a further technical solution of the present invention, the network training in step (8) includes three parts. The first part is the training of the vehicle detection network, which aims to obtain the location information of the 360-degree monitored vehicle and the size to be cropped based on the spherical convolution method. The second part is the training of the pseudo-infrared monitoring image generation network with adversarial domain adaptation, which aims to make the generated pseudo-infrared monitoring image of higher quality. The third part is the training of the vehicle re-identification network based on RGB and NIR data, which is the final vehicle re-detection network. By judging the similarity between vehicle feature dimensions, it outputs whether it is the vehicle that needs to be found.

[0045] Compared with the prior art, the beneficial effects of the present invention are:

[0046] (1) The method proposed in this invention is aimed at 360-degree monitoring, which can realize intelligent vehicle re-identification and help to broaden the scope of vehicle re-identification monitoring.

[0047] (2) This invention proposes a new 360-degree video surveillance method that can introduce infrared data in a multimodal form to ensure the accuracy of vehicle re-identification. At the same time, it can overcome the problem of decreased accuracy of vehicle re-identification algorithm in complex scenarios. Attached Figure Description

[0048] Figure 1 This is a flowchart of the vehicle re-identification method provided by the present invention. Detailed Implementation

[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments, but the scope of the invention is not limited in any way.

[0050] Example:

[0051] like Figure 1 As shown, this embodiment provides a 360-degree vehicle re-detection method based on infrared imaging, which specifically includes the following steps:

[0052] (1) Construction of a video dataset for re-identifying vehicles under 360-degree infrared surveillance in cities:

[0053] To construct a 360-degree infrared surveillance vehicle re-identification dataset for cities, a 360-degree vehicle detection dataset is first collected. Then, a pseudo-infrared vehicle re-identification dataset is generated based on the 360-degree vehicle detection dataset. Finally, a vehicle re-identification network is trained based on the generated pseudo-infrared vehicle re-identification dataset. Since there are very few 360-degree vehicle re-detection datasets based on infrared surveillance, this embodiment uses domain adaptation to generate pseudo-infrared surveillance data. This embodiment uses the VeRi-776, VehicleID, and VERI-Wild datasets as the 360-degree vehicle detection dataset and uses domain adaptation to generate pseudo-infrared surveillance data as the city's 360-degree infrared surveillance vehicle re-identification video dataset. The images in the 360-degree vehicle detection dataset are labeled with vehicle location information by manually annotating points. Based on the vehicle location information, projection data based on the location points is generated, and then the outer bounding box is labeled based on the projection data of the location points. This method can accurately label the vehicle detection dataset.

[0054] (2) 360-degree infrared vehicle re-identification video feature extraction:

[0055] Traditional 360-degree video processing involves projecting the image onto ERP or CMP images and extracting features for classification or prediction. However, ERP and CMP suffer from several drawbacks. For example, ERP can cause stretching during projection, and CMP may segment adjacent objects into different planes, leading to errors in the final result. To address these issues, this embodiment employs a distortion-invariant spherical sampling strategy. While convolution and pooling operations in traditional CNNs are also sampling operations, unlike planar uniform sampling, spherical sampling needs to consider the stretching caused by spherical projection. Therefore, while considering the stretching issue during sampling, spherical pixels are sampled as planar convolutions, allowing direct input into a 2D convolutional neural network for processing. Currently, CNN networks primarily rely on size-based sampling... 2D convolutions, even in large sizes 2D convolution can also be achieved through... To save parameters, convolution is used instead. In this embodiment, the following convolution operation is employed:

[0056] The main operations of 2D convolution are:

[0057] ,

[0058] The main operations of 3D convolution are:

[0059] ,

[0060] in, Representative position Using 2D convolution; represent The neighborhood, Represents the weight parameter. Representative feature layer, Representative position Using 3D convolution, The transformation function representing the projection from the sphere to the tangent plane; This represents the transformation function from the projected tangent plane to the sphere.

[0061] (3) 360-degree vehicle detection based on spherical convolution:

[0062] Predicting vehicle position information in 360-degree video using spherical convolution still presents problems because the predicted coordinates are 2D coordinates, which still have errors compared to the actual position of the vehicle on the sphere. To address this issue, this embodiment proposes a new method for processing 360-degree video based on spherical convolution, namely, processing the pixel information of coordinate points based on spherical coordinates. The main process is as follows:

[0063] (3-1) Input the ERP image from 360-degree infrared monitoring into the convolutional layer that performs the convolution operation in step (2), and introduce ConvBlock to enhance vehicle feature constraints. After multiple ConvBlocks, output spherical coordinate position information, perform projection based on the spherical coordinate position information, and manually mark the bounding box generated by the projection. The detailed operation is as follows.

[0064] ,

[0065] in, This represents the index image of the ERP system, representing the 360-degree infrared monitoring system. The representative will Encapsulate it in a form similar to ResBlock; represent The number of; Represents the vehicle's position coordinates on the sphere; This represents the width and height of the vehicle's outer bounding box after projection;

[0066] (3-2) After obtaining the vehicle's position and width / height information, calculate the loss between the predicted coordinates and the width and height of the outer bounding box and the manually labeled information. The position loss function is used to calculate the loss between the vehicle's position on the sphere and the manually labeled information.

[0067] ,

[0068] in, The position loss function representing spherical coordinates, This represents the predicted spherical coordinate position information. The location coordinates are manually labeled.

[0069] The loss for the width and height of the outer bounding box is based on The loss between the bounding boxes w and h of the projected 2D vehicle image and the manually annotated w and h.

[0070] ,

[0071] in, This represents the loss function after projection; This represents the predicted width and height information. This represents the width and height information manually annotated.

[0072] (4) Secondary projection of the predicted location and width / height based on 360° surveillance:

[0073] Step (3) provides accurate vehicle location information and precise vehicle width and height information. The system projects vehicles monitored on a sphere from the sphere into 2D vehicle images, and then crops the width and height to ensure that the vehicles are detailed vehicle blocks. :

[0074] ,in, This represents vehicle location information generated based on the ERP image of the index frame; Representative based on The secondary projection produces an image containing the vehicle; This represents the precise region of the vehicle image block in the index 1, i.e., the refined vehicle block.

[0075] (5) Generation of pseudo-infrared surveillance images adapted to the adversarial domain:

[0076] The refined vehicle block is obtained through step (4). Afterwards, re-identification is required based on infrared monitoring images, because infrared monitoring images can provide information missing in RGB monitoring, assisting RGB monitoring in more accurately identifying vehicles. However, there is a lack of sufficient infrared information for vehicles in the video. Therefore, this embodiment uses a pseudo-infrared monitoring image generation method based on adversarial domain adaptation to generate pseudo-infrared monitoring images, thereby alleviating the problem of model training failure caused by the shortage of infrared images. The main process is as follows: first, train the network based on the existing infrared monitoring dataset, and then input the obtained in step (4) based on the trained network. This generates vehicle images from pseudo-infrared surveillance data. That is, its detailed definition is: .

[0077] We can obtain the following through step (5) Therefore, vehicle re-detection input data based on NIR data and RGB data are obtained. Next, we need to... The input is fed into a vehicle feature extraction network to extract visual features. In this embodiment, a dual-stream ResNet network is used as the vehicle feature extraction network, and its detailed operation is as follows.

[0078] ,

[0079] ,

[0080] After feature extraction, in order to enable the domain adaptive network used in this embodiment to improve the generation quality of pseudo-infrared surveillance images, the following is introduced: loss function

[0081] ,

[0082] in, Represents the global average pooling layer. This represents the feature mapping layer, which maps feature dimensions to single-dimensional features. A matrix of 1 represents positive samples. The zero matrix represents negative samples.

[0083] (7) Output of vehicle re-identification results:

[0084] After the above steps, regardless of whether it is based on RGB information Still based on NIR information Both can be extracted. The next step is to fuse them to better leverage their complementary characteristics. This embodiment uses an attention mechanism to weight RGB and NIR.

[0085] ,

[0086] ,

[0087] in, This represents the final output result. This represents a global average pooling layer, which transforms features into high-dimensional semantic information. This represents a linear mapping layer that maps high-dimensional information to low-dimensional information, thereby outputting vehicle re-identification results.

[0088] (8) Network training and network testing:

[0089] The training of the entire network consists of three parts. The first part is the training of the vehicle detection network, whose main purpose is to obtain the location information of the monitored vehicle and the size to be cropped based on spherical convolution. The second part is the training of the adversarial domain-adaptive pseudo-infrared monitoring image generation network, whose main purpose is to improve the quality of the generated pseudo-infrared monitoring images. The third part is the training of the vehicle re-detection network based on RGB and NIR data. This is the final vehicle re-detection network, which judges the similarity between vehicle feature dimensions to output whether it is the vehicle that needs to be found.

[0090] Although network training consists of three main parts, its testing objective is very clear: to find the same vehicle with higher similarity in the current 360-degree video surveillance. Therefore, in the testing phase of the network, the vehicle detection network is used to locate the vehicle position and the size of the cropping region. Based on this, similarity measures between different vehicle features based on RGB and NIR are calculated to determine whether the current vehicle is the vehicle being searched for.

[0091] Algorithms and network structures not described in detail in this article are general techniques in this field.

[0092] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. A 360-degree vehicle re-identification method based on infrared imaging, characterized in that, The specific process is as follows: (1) Collect a 360-degree vehicle detection dataset and use the domain adaptive method to generate pseudo-infrared monitoring data as a 360-degree infrared monitoring vehicle re-identification video dataset. (2) The video dataset in step (1) is projected into ERP to obtain ERP images. Based on the distortion invariance of the sphere sampling strategy, the spherical pixels are sampled into planar convolutions. A CNN network is used to extract the 360-degree infrared monitoring vehicle re-identification video features. (3) The vehicle location information, width, and height are obtained by processing 360-degree video based on spherical convolution; the specific process is as follows: (3-1) Input the ERP image from 360-degree infrared monitoring into the convolutional layer of the CNN network used in step (2). By introducing ConvBlock, the vehicle feature constraints are enhanced. After multiple ConvBlocks, the spherical coordinate position information is output. Projection is performed based on the spherical coordinate position information, and the outer bounding box generated by the projection is constrained by manual marking. The detailed operation is as follows: ， in, This represents the index image of the ERP system, representing the 360-degree infrared monitoring system. The representative will Encapsulate it in a form similar to ResBlock; represent The number of; Represents the vehicle's position coordinates on the sphere; This represents the width and height of the vehicle's outer bounding box after projection; (3-2) After obtaining the vehicle's position coordinates and width and height information, the position loss function is used to calculate the loss between the vehicle's position on the sphere and the manually labeled position. ， in, The position loss function representing spherical coordinates, This represents the predicted spherical coordinate position information. The location coordinates are manually labeled. The loss for the width and height of the outer bounding box is based on The loss between the bounding boxes w and h of the projected 2D vehicle image and the manually annotated w and h. ， in, This represents the loss function after projection; This represents the predicted width and height information. Represents the width and height information manually annotated; (4) Based on the vehicle location information and the width and height of the vehicle obtained in step (3), the vehicle monitored by the sphere is projected into a 2D vehicle image by projection, and then cropped by width and height to obtain a refined vehicle block. (5) Based on the refined vehicle block obtained in step (4), a pseudo-infrared monitoring image is generated by using a pseudo-infrared monitoring image generation method based on adversarial domain adaptation. (6) Input the refined vehicle block obtained in step (4) and the pseudo-infrared monitoring image obtained in step (5) into a dual-stream ResNet network for feature extraction to obtain vehicle features based on RGB and NIR data respectively, and introduce them into the network. The loss function improves the quality of generated pseudo-infrared surveillance images; (7) Use an attention mechanism to weight the vehicle features obtained in step (6) based on RGB and NIR data, and output the vehicle re-identification results; (8) Train and test the network. During the network testing phase, after locating the vehicle position and the size of the cropping region, calculate the similarity measure between different vehicle features based on RGB and NIR to determine whether the current vehicle is the vehicle being searched.

2. The 360-degree vehicle re-identification method based on infrared imaging according to claim 1, characterized in that, Step (1) The 360-degree vehicle detection dataset uses the VeRi-776, VehicleID and VERI-Wild datasets. The vehicle position information is marked on the images in the dataset by manually marking points. Projection data based on the position points is generated based on the vehicle position information, and then the outer bounding box is marked based on the projection data of the position points.

3. The 360-degree vehicle re-identification method based on infrared imaging according to claim 1, characterized in that, The 2D convolution operation in the CNN network described in step (2) is as follows: ， The operation of 3D convolution is as follows: ， in, represent Position using 2D convolution; represent The neighborhood, Represents the weight parameter. Representative feature layer, represent Position using 3D convolution, The transformation function representing the projection from the sphere to the tangent plane; This represents the transformation function from the projected tangent plane to the sphere.

4. The 360-degree vehicle re-identification method based on infrared imaging according to claim 3, characterized in that, The specific process of step (4) is as follows: ,in, This represents vehicle location information generated based on the ERP image of the index frame; Representative based on The secondary projection produces an image containing the vehicle; Represents the refined vehicle block in the index.

5. The 360-degree vehicle re-identification method based on infrared imaging according to claim 4, characterized in that, The specific process of step (5) is as follows: training based on the existing infrared surveillance dataset. well trained The network, inputting the refined vehicle blocks obtained in step (4), Generate infrared monitoring data ,Right now .

6. The 360-degree vehicle re-identification method based on infrared imaging according to claim 5, characterized in that, Step (6) The vehicle features based on RGB and NIR data are as follows: ， ， The loss function is: ， in, This represents the global average pooling layer. This represents the feature mapping layer, which maps feature dimensions to single-dimensional features. A matrix of 1 represents positive samples. The zero matrix represents negative samples.

7. The 360-degree vehicle re-identification method based on infrared imaging according to claim 6, characterized in that, Step (7) uses an attention mechanism for weighting, which is as follows: ， The output vehicle re-identification result is as follows: ， in, This represents the final output result. This represents a global average pooling layer, which transforms features into high-dimensional semantic information. This represents a linear mapping layer that maps high-dimensional information to low-dimensional information, thereby outputting vehicle re-identification results.

8. The 360-degree vehicle re-identification method based on infrared imaging according to claim 7, characterized in that, The network training in step (8) consists of three parts. The first part is the training of the vehicle detection network, the purpose of which is to obtain the location information of the 360-degree monitored vehicle and the size to be cropped based on the spherical convolution method. The second part is the training of the adversarial domain adaptation pseudo-infrared surveillance image generation network, the purpose of which is to make the generated pseudo-infrared surveillance images of higher quality. The third part is the training of the vehicle re-identification network based on RGB and NIR data. This is the final vehicle re-identification network, which determines whether the vehicle being searched is the one that is similar to the vehicle feature dimensions.

Images