A vehicle re-identification method and device based on multiple cameras

Abstract

The application provides a vehicle re-identification method and device based on multiple cameras. The method comprises the following steps: acquiring an initial image, performing target detection, acquiring a category to which the target belongs, and extracting a target vehicle image in the initial image and saving the target vehicle image to a target image library; saving a vehicle image to be matched to a to-be-matched image library; selecting a target vehicle image from the target image library for target segmentation; introducing multiple vehicle features to encode the target vehicle image to obtain encoding information; inputting the encoding information into a multi-scale hierarchical feature extraction network based on a Transformer to obtain global features; extracting local features according to a convolutional neural network and the target segmentation result; and calculating the degree of approximation between the target vehicle image and all to-be-matched vehicle images in the to-be-matched image library according to the global features and the local features to re-identify the target vehicle. In this way, vehicle re-matching under multiple cameras is realized, the detection efficiency and quality are improved, manpower and material resources are saved, and costs and resources are saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle identification, and more particularly, to a multi-camera based vehicle re-identification method and device. BACKGROUND

[0002] Multi-camera vehicle re-identification refers to finding the same vehicle in multiple cameras at different times. By analyzing which cameras the vehicle appears in, the vehicle's driving trajectory can be determined from the positions of these cameras, thereby enabling tracking of the vehicle.

[0003] Currently, vehicle tracking and identification under multiple cameras mainly relies on manual analysis, with staff carefully checking images captured by each camera and finding the same vehicle to be tracked in each camera. In recent years, some technologies have used computer vision methods to achieve vehicle re-identification, including:

[0004] 1. Vehicle re-identification using vehicle license plates, identifying vehicle license plates under different cameras, and determining the same vehicle when the same license plate is detected under different cameras. The vehicle's driving trajectory can be determined from the positions of these cameras, thereby enabling vehicle re-identification.

[0005] 2. Vehicle re-identification using vehicle appearance color information, identifying vehicles with the most similar colors under different cameras, and determining the same vehicle. The vehicle's driving trajectory can be determined from the positions of these cameras, thereby enabling vehicle re-identification.

[0006] Existing manual detection techniques require a large number of staff to carefully check captured images and find the same vehicle captured under different cameras, consuming a large amount of manpower, resources, and financial resources. The biggest disadvantage of using license plate information for vehicle re-identification is that when the license plate is obscured or the camera cannot capture the license plate or cannot recognize the license plate, vehicle re-identification fails due to the inability to obtain license plate information, and the same vehicle cannot be found under multiple cameras. The biggest disadvantage of using vehicle color information for vehicle re-identification is that under different cameras, due to different environmental conditions such as camera ambient light, the color of the vehicle may not be the same under different lighting conditions, so using vehicle appearance color as a basis for vehicle re-identification may result in many detection errors. SUMMARY

[0007] According to an embodiment of the present application, a multi-camera based vehicle re-identification scheme is provided. This scheme replaces the traditional manual detection method, solves the problem of low accuracy of vehicle re-identification based on vehicle license plates, enables vehicle re-matching under multiple cameras, improves detection efficiency and quality, saves manpower and resources, and saves costs and resources.

[0008] In a first aspect of the present application, a multi-camera-based vehicle re-identification method is provided. The method comprises:

[0009] obtaining initial images collected by a plurality of cameras;

[0010] performing target detection on the initial images using a target detection algorithm, obtaining the category to which the target belongs, and extracting target vehicle images in the initial images and saving them to a target image library; and saving the to-be-matched vehicle images to a to-be-matched image library;

[0011] performing target segmentation on the images in the target image library and the to-be-matched image library to obtain target segmentation results;

[0012] encoding the images in the target image library and the to-be-matched image library, introducing a plurality of vehicle features to encode the vehicle, and obtaining encoding information of the images;

[0013] inputting the encoding information of the images into a multi-scale hierarchical feature extraction network based on Transformer to obtain global features of the images; and extracting local features of the images according to a convolutional neural network and the target segmentation results;

[0014] obtaining target vehicle images from the target image library, calculating the degree of approximation between the target vehicle images and all to-be-matched vehicle images in the to-be-matched image library according to the global features and the local features of the target vehicle images, and re-identifying the target vehicle according to the degree of approximation.

[0015] Further, the obtaining of the initial images collected by the plurality of cameras comprises:

[0016] selecting a plurality of cameras, one of which is used as a collection camera for obtaining target images, and the others are used as to-be-matched cameras for obtaining to-be-matched images;

[0017] setting a time interval for image collection, and collecting to-be-matched images and target images according to the time interval as initial images.

[0018] Further, the performing of target detection on the initial images using a target detection algorithm, obtaining the category to which the target belongs, and extracting target vehicle images in the initial images and saving them to a target image library; and saving the to-be-matched vehicle images to a to-be-matched image library, comprises:

[0019] inputting the initial images into a deep learning convolutional neural network for target detection, outputting the position information and category information of all vehicles in the initial images in the corresponding initial images, and marking all vehicles in the initial images with a rectangular frame;

[0020] Extracting the inner part of the rectangular frame to obtain a target vehicle image and a to-be-matched vehicle image.

[0021] Further, the target segmentation of the images in the target image library and the to-be-matched image library obtains a target segmentation result, including:

[0022] Each image in the target image library and the to-be-matched image library is segmented by using an image segmentation algorithm to obtain a plurality of region segmentation blocks as the target segmentation result of each image, and the region segmentation blocks are used to describe a front region, a roof region, a side region, a tail region and a background region in the image.

[0023] Further, the encoding of the images in the target image library and the to-be-matched image library obtains encoding information of the images, including:

[0024] Each image in the target image library and the to-be-matched image library is divided into a plurality of equal-area image blocks, and the position encoding information of the image blocks is obtained according to the position of the image blocks in the corresponding image.

[0025] The image information of the image blocks is subjected to convolution and linear normalization processing to obtain image encoding information.

[0026] The target category encoding information is obtained according to the category to which the target belongs.

[0027] The target segmentation result encoding information is obtained according to the target segmentation result.

[0028] The sum of the image encoding information, the target category encoding information, the target segmentation result encoding information and the position encoding information of the image blocks is taken as the encoding information of the current image.

[0029] Further, the feature extraction network based on the Transformer includes a first Transformer structure layer, a second Transformer structure layer, a third Transformer structure layer and a fourth Transformer structure layer, wherein the first Transformer structure layer, the second Transformer structure layer and the fourth Transformer structure layer each include 2 Transformer structures; the fourth Transformer structure layer includes 6 Transformer structures; and 1 up-sampling layer is included between adjacent Transformer structure layers.

[0030] Further, the local features of the image are extracted according to the convolutional neural network and the target segmentation result, including:

[0031] extracting image features of each image in the target image library and the image library to be matched through a ResNet convolutional neural network;

[0032] masking corresponding parts according to the target segmentation result;

[0033] multiplying the image features of each image with the mask of the corresponding part, and then performing linear transformation to obtain local features of the current image.

[0034] Further, the calculation of the approximation degree between the target vehicle image and all the vehicle images to be matched in the image library to be matched according to the global features and the local features of the target vehicle image comprises:

[0035] traversing all the vehicle images to be matched in the image library to be matched with the target vehicle image, calculating the local feature Euclidean distance between the target vehicle image and the corresponding part of the vehicle image to be matched, and accumulating the weight of the corresponding part to obtain a plurality of local feature distance values; and calculating the global feature Euclidean distance between the target vehicle image and the vehicle image to be matched to obtain a plurality of global feature distance values;

[0036] accumulating the local feature distance values and the global feature distance values corresponding to the same vehicle image to be matched to obtain a plurality of feature distance values between the target vehicle image and the vehicle image to be matched.

[0037] Further, the re-identification of the target vehicle according to the approximation degree comprises:

[0038] obtaining the minimum value in the plurality of feature distance values;

[0039] If the minimum value in the plurality of feature distance values is less than a preset threshold, the target vehicle re-identification is successful, and the vehicle image successfully re-identified is deleted from the corresponding image library;

[0040] If the minimum value in the plurality of feature distance values is not less than the preset threshold, the target vehicle re-identification fails, and the local feature information, the global feature information and the mask information of the target vehicle are saved

[0041] In a second aspect of the present application, an electronic device is provided. The electronic device comprises at least one processor; and a memory connected in communication with the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of the first aspect of the present application.

[0042] It is to be understood that the description in the Summary section is not intended to define key or essential features of embodiments of the application, nor is it intended to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0043] The above and other features, aspects, and advantages of embodiments of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings. In the drawings, like reference numerals refer to like elements, wherein:

[0044] Figure 1 A flow chart of a multi-camera-based vehicle re-identification method according to an embodiment of the present application is shown;

[0045] Figure 2 An improved YOLOv4 network structure diagram according to an embodiment of the present application is shown;

[0046] Figure 3 An initial image diagram showing a marked rectangular frame according to an embodiment of the present application is shown;

[0047] Figure 4 An image segmentation result diagram according to an embodiment of the present application is shown;

[0048] Figure 5 A mask diagram made according to the image segmentation result according to an embodiment of the present application is shown;

[0049] Figure 6 A Transformer-based feature extraction network structure diagram according to an embodiment of the present application is shown;

[0050] Figure 7 A structure diagram of a Transformer layer according to an embodiment of the present application is shown;

[0051] Figure 8 A block diagram of an exemplary electronic device capable of implementing an embodiment of the present application is shown;

[0052] Wherein, 800 is an electronic device, 801 is a CPU, 802 is a ROM, 803 is a RAM, 804 is a bus, 805 is an I / O interface, 806 is an input unit, 807 is an output unit, 808 is a storage unit, and 809 is a communication unit. DETAILED DESCRIPTION

[0053] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0054] In addition, the term "and / or" in this paper is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are in an "or" relationship.

[0055] The present application replaces the traditional manual detection method, can solve the problem of low accuracy of vehicle re-identification based on vehicle license plate, realizes vehicle re-matching under multiple cameras, improves detection efficiency and quality, saves manpower and material resources, saves cost and resources.

[0056] Figure 1 The flowchart of the vehicle re-identification method based on multiple cameras of the embodiments of the present application is shown.

[0057] The method comprises:

[0058] S101, acquiring initial images collected by a plurality of cameras.

[0059] As an embodiment of the present application, the acquiring of the initial images collected by the plurality of cameras comprises:

[0060] Firstly, a plurality of cameras are selected, wherein any one camera is selected as a collection camera for acquiring a target image, and the other cameras are selected as to-be-matched cameras for acquiring to-be-matched images.

[0061] Then, a time interval for image collection is set, and the to-be-matched images and the target image are collected according to the time interval as initial images.

[0062] In this embodiment, the plurality of cameras can be represented by N, N is a positive integer not less than 2. The cameras collect an image every q seconds, so N images can be obtained every q seconds. Any one camera is selected as a collection camera, and the remaining N-1 cameras are selected as to-be-matched cameras. Every q seconds, one target image and N-1 to-be-matched images can be collected.

[0063] S102, target detection is performed on the initial image by using a target detection algorithm, target vehicle images in the initial image are extracted and saved to a target image library, and the to-be-matched vehicle image is saved to a to-be-matched image library.

[0064] As an embodiment of the present application, the target detection performed on the initial image by using the target detection algorithm, the target category acquired and the target vehicle images in the initial image extracted and saved to the target image library include:

[0065] The initial image is input into a deep learning convolutional neural network for target detection, position information of all target vehicles in the initial image in the corresponding initial image is output, and a rectangular frame is used to mark the target vehicles in the initial image and the to-be-matched vehicles in the to-be-matched image; the part in the rectangular frame is extracted to obtain the target vehicle images and the to-be-matched vehicle images.

[0066] In the present embodiment, the deep learning convolutional neural network for target detection is a YOLO v4 network structure.

[0067] As an embodiment of the present application, as shown in Figure 2 YOLO v4 algorithm can be improved, and the improved YOLO v4 network structure is composed of CSPDarknet5, CBL, SPP, UpSample, Conv and three output modules.

[0068] The CSPDarknet53 network is composed of CSP and Darknet53; the CSP is Cross, Stage and Partial respectively; the CSP can enhance the learning ability of the CNN, and can maintain accuracy, reduce calculation difficulty and memory cost while being lightweight. Darknet53 is a network structure based on Darknet, which contains 5 large residual network blocks, and each large residual network block contains a certain number of residual network structures. CSPDarknet53 is to add CSP structure to each large residual block of Darknet53.

[0069] CBL is composed of Conv+BN+Leaky_relu activation function, wherein Conv represents convolution layer, BN represents normalization, and Leaky_relu is an activation function.

[0070] SPP is a spatial pyramid pooling layer, which can generate fixed output for any size of input, and solve the image distortion error caused by non-proportional compression of the input image.

[0071] UpSample represents the data up-sampling operation.

[0072] 3 output modules are 76*76*3*(4+1+class_num), 38*38*3*(4+1+class_num) and 19*19*3*(4+1+class_num) respectively.

[0073] In the above embodiment, the improved YOLO v4 network structure adds two SPP modules in the YOLO v4 network, reduces the error of image deformation, and improves the detection ability of small targets.

[0074] For N initial images collected in each collection cycle, first, target detection is performed on one target image to obtain vehicle position information, and a rectangular frame is used to mark the w target vehicles detected in the target image, as shown in the following formula: Figure 3 where w is a positive integer not less than 1. The image in the rectangular frame is extracted to obtain w target vehicle images, denoted as P 0w . P 0w is placed in a target image library for subsequent vehicle re-identification matching. Secondly, target detection is performed on N-1 images to be detected. For the i-th image to be detected, M i target vehicles are detected, and a rectangular frame is used to mark them. For the j-th image, it can be denoted as P ij (i=1~N-1; j=1~M i ). All the images to be matched are placed in a matching image library for subsequent vehicle re-identification matching with the target images in the target image library.

[0075] In this embodiment, a total of S vehicle images are extracted. Among them,

[0076]

[0077] where S is the number of extracted vehicle images; w is the number of target vehicles in the target image; M i is the number of target vehicles in the i-th image to be detected; and N is the number of vehicle images collected in each collection cycle.

[0078] S103, target segmentation is performed on the images in the target image library and the matching image library to obtain a target segmentation result.

[0079] As an embodiment of the present application, the images in the target image library and the matching image library are segmented by using an image segmentation algorithm, and each image obtains a plurality of region segmentation blocks as a target segmentation result, which are used to describe the front region, the top region, the side region, the tail region and the background region of the vehicle in the image.

[0080] In the embodiment, the image segmentation can adopt a deep learning-based image segmentation algorithm, such as a Deeplab series, a PSPNet network, a U-Net network, a U-Net++ network, and the like.

[0081] In the embodiment, the vehicle image can be divided into four parts, i.e., a front part, a side part, a top part, and a tail part of the vehicle. The front part of the vehicle can be a part displayed as a front view of the vehicle head; the side part of the vehicle can be a part displayed as a front view of two sides of the vehicle; the top part of the vehicle can be a part displayed as a front view of the vehicle roof; and the tail part of the vehicle can be a part displayed as a front view of the vehicle tail. The divided front part, side part, top part, and tail part do not have overlapping areas and cover all parts of the vehicle body. The remaining part of the image is a background part of the image, which is not considered in the calculation of the local feature. As shown in FIG. 1, Figure 4 Figure 4 (a) is a vehicle image, Figure 4 (b) is an image segmentation result, including a front part, a side part, and a top part of the vehicle.

[0082] S104, encoding the images in the target image library and the to-be-matched image library to obtain image encoding information.

[0083] As an embodiment of the present application, the image encoding includes:

[0084] Dividing each image in the target image library and the to-be-matched image library into a plurality of equal-area image blocks, encoding the positions of the image blocks in the corresponding images to obtain position encoding information of the image blocks;

[0085] Convolving and linearly normalizing the image information of the image blocks to obtain image encoding information;

[0086] Obtaining target category encoding information according to the category to which the target belongs;

[0087] Obtaining target segmentation result encoding information according to the target segmentation result;

[0088] Taking the sum of the image encoding information, the target category encoding information, the target segmentation result encoding information, and the position encoding information of the image blocks as the encoding information of the current image.

[0089] In the above embodiment, the image size obtained by target detection is unified to 224*224*3.

[0090] Position encoding of the image block:

[0091] ​The vehicle image is divided into equal-area image blocks, the input image has a width of 224, a height of 224, and 3 image channels. The input image is first divided into blocks, with 4*4 as a window, i.e., the 224*224 image is divided into 56*56=3136 image blocks. The 3136 image blocks are encoded from 0 to 3135 positions, i.e., the position encoding information of the image blocks is obtained.

[0092] Image encoding:

[0093] The input image is convolved and linearly normalized with a convolution kernel size of 4 and 96 channels, and the output result is B*96*56*56, which is rearranged to B*3136*96, i.e., image encoding information.

[0094] Target class encoding:

[0095] The target detection divides vehicles into 9 categories, namely, sedan (hatchback), SUV, van, sedan (hatchback), MPV, light truck (pickup), bus, truck, and multi-purpose vehicle. Their corresponding category IDs are 0-8. According to the vehicle categories, the category IDs are encoded to obtain the target class encoding information.

[0096] Image segmentation result encoding:

[0097] In the above embodiment, the vehicle image has been segmented by an image segmentation algorithm, and the vehicle has been divided into 5 parts, i.e., the front part, the roof part, the side part, the rear part, and the background part. The area occupied by each part in the image is counted. The front part, the roof part, the side part, and the rear part are sequentially binary encoded, and if all 4 parts are not detected, the encoding is 0000, corresponding to the decimal encoding 0. Let the area of the front part be S1, the area of the roof part be S2, the area of the side part be S3, and the area of the rear part be S4. When S1 is greater than 0.2*(S1+S2+S3+S4), it is considered that the front part is detected, and the binary encoding is 1000, i.e., the decimal encoding is 8. If the side part and the rear part are simultaneously detected in a target vehicle image, the binary encoding is 0011, i.e., the decimal encoding is 3. The segmentation results of all images are counted, and each image corresponds to a numerical encoding.

[0098] The above feature encoding is extended to a unified dimension, and is added in the same dimension, i.e., the overall feature encoding information of the target vehicle image is obtained, which is input into the feature extraction network based on Transformer.

[0099] S105, input the encoding information of the target vehicle image into the feature extraction network based on Transformer to obtain the global features of the image; and extract the local features of the image according to the convolutional neural network and the target segmentation result.

[0100] Specifically, in the present embodiment, global target extraction is performed by introducing the Transformer technology, which not only has excellent performance in extracting features, but also solves the problem that the convolutional neural network easily loses detailed features when extracting features of small pictures, thereby improving the ability of re-identifying small size pictures.

[0101] In the present embodiment, as shown in Figure 6 the Transformer-based feature extraction network includes a first Transformer structure layer, a second Transformer structure layer, a third Transformer structure layer and a fourth Transformer structure layer, wherein the first Transformer structure layer, the second Transformer structure layer and the fourth Transformer structure layer each include two Transformer structures; the fourth Transformer structure layer includes six Transformer structures; and one up-sampling layer is included between adjacent Transformer structure layers.

[0102] The encoded information of the image is first input into the first Transformer layer, which is composed of two Transformer structures, wherein the Transformer structure is as shown in Figure 7 The length and width of the vehicle image are both 224, so the input size is B*56*56*96. After the first Transformer layer, the output is also B*56*56*96. Then it is input into the first up-sampling layer, which is used for the operation of expanding the field of view, and the purpose is to obtain a larger field of view. The output result is subjected to convolution and linear transformation, and the width and height of the output are each reduced by half compared with the input, and the number of channels is doubled. Then it is sequentially passed through the second Transformer layer, the up-sampling layer, the third Transformer layer, the up-sampling layer and the fourth Transformer layer, and the operation of 3 times of Transformer layer transformation and 2 times of up-sampling layer expanding the field of view is performed, and the final global feature is output. The Transformer structure is as shown in Figure 7 which includes two linear transformation layers, a Mulit-headed Self-attention layer and a multi-layer perceptron, etc.

[0103] Through the above process, the global feature information of several images can be obtained, but it is difficult to accurately identify vehicles under different cameras by only using the global information of the image, so in addition to the global feature of the image, the local feature of the image also needs to be introduced.

[0104] As an embodiment of the present application, the local features of the image are extracted according to the convolutional neural network and the target segmentation result, which comprises:

[0105] The features of the target vehicle image are extracted by the ResNet convolutional neural network.

[0106] The mask corresponding to each part is made according to the target segmentation result.

[0107] The image features of each image are multiplied by the mask corresponding to the part, and then linearly transformed to obtain the local features of the target vehicle image.

[0108] In the embodiment, the mask corresponding to each part is made according to the target segmentation result, which comprises:

[0109] The masks of the front, side, top and tail of the vehicle are made according to the target segmentation result. For the part of the vehicle not detected by the image segmentation, the mask corresponding to the part is empty. As shown in Figure 5 Figure 5 (a) represents the mask of the front of the vehicle, Figure 5 (b) represents the mask of the tail of the vehicle, Figure 5 (c) represents the mask of the top of the vehicle, Figure 5 (d) represents the mask of the side of the vehicle. As can be seen, the mask corresponding to the tail of the vehicle in the above vehicle image is almost empty because the tail of the vehicle is not detected by the image segmentation.

[0110] The size of each mask is 32*32, so a total of S*4*32*32 masks are formed.

[0111] In the embodiment, the features of the S vehicle images are extracted by the convolutional neural network. The output layer dimension of the convolutional neural network is 32*32, so the total number of output features is S*32*32.

[0112] According to the above embodiment, for each vehicle image, the corresponding four masks are multiplied by the features with a size of 32*32 respectively, to obtain 4*32*32 features, which are linearly transformed to become 4*516 features. Therefore, for S vehicle images, there are S*4*516 local features.

[0113] S106, the target vehicle image is obtained from the target image library, the global features and local features of the target vehicle image are calculated, the approximate degree between the target vehicle image and all the vehicle images in the to-be-matched image library is calculated, and the target vehicle is re-identified according to the approximate degree.

[0114] ​As an embodiment of the present application, a target vehicle image is acquired from the target image library, and the approximation degree between the target vehicle image and all the vehicle images in the image library to be matched is calculated according to the global feature and the local feature, including:

[0115] The target vehicle image traverses all the vehicle images in the image library to be matched, the Euclidean distance of the local feature of the corresponding part of the target vehicle image and the vehicle image to be matched is calculated, and the weight of the corresponding part is accumulated to obtain a plurality of local feature distance values; the number of the local feature distance values is equal to the number of the vehicle images to be matched in the image library to be matched.

[0116] In the embodiment, for any vehicle image to be matched in the image library to be matched, the Euclidean distance of the local feature of the corresponding part of the target vehicle image and the vehicle image to be matched is calculated, and the weight of the corresponding part is accumulated to obtain a local feature distance value. For example, assuming that the target vehicle library has S1 vehicle images, the image library to be matched has S2 vehicle images, one vehicle image N3 is selected from the target vehicle library, and the Euclidean distance of the local feature of the vehicle image N3 and the S2 vehicle images in the image library to be matched is calculated. If the vehicle image N3 is divided into four parts, a total of 4*S2 Euclidean distance values are obtained, and S2 local feature distance values are obtained by weighted calculation according to the weight of each part.

[0117] When comparing the local features of two images, the weights of different local features should be different. The weight calculation formula is as follows:

[0118]

[0119] Among them, and are the areas of the i-th part of the target vehicle image and the vehicle image to be matched, respectively; is the weight of the i-th part of the target vehicle image and the vehicle image to be matched; z is the number of parts after image segmentation.

[0120] The size of each local mask and the size of the whole vehicle are not the same under different cameras for the same vehicle. For example, some vehicle images have front, top and side masks, among which the front and side masks have large areas, and the rear mask has almost no area. For example, some vehicle images have rear, top and side masks, among which the rear and side masks have large areas, and the front mask has almost no area. Therefore, when comparing the local features, the local feature information of the overlapping part (such as the top and side) should be considered, and the weight of this part of feature comparison should be increased; and for the part with very low repetition (such as the front and rear), the weight should be reduced. Through the above calculation formula, reasonable weights can be calculated for subsequent local feature comparison.

[0121] Further, the Euclidean distance of the global feature between the target vehicle image and the vehicle image to be matched is calculated as a global feature distance value by traversing all the vehicle images to be matched in the image library to be matched.

[0122] In the embodiment, it is assumed that the target vehicle library has S1 vehicle images, the image library to be matched has S2 vehicle images, and one vehicle image N3 is selected from the target vehicle library. The Euclidean distance of the global feature between the vehicle image N3 and the S2 vehicle images in the image library to be matched is calculated to obtain S2 global feature distance values.

[0123] Further, the local feature distance value and the global feature distance value corresponding to the same vehicle image to be matched are accumulated to obtain a plurality of feature distance values between the target vehicle image and the vehicle image to be matched.

[0124] In the embodiment, the feature distance value between the target vehicle image and the vehicle image to be matched is calculated as follows:

[0125] Feature distance value = 0.5 * local feature distance value + 0.5 * global feature distance value

[0126] In the above embodiment, the Euclidean distance of the global feature and the local feature of two images is calculated respectively, and the feature distance value between the two images is obtained by adding the two distances. The feature distance value between the target vehicle image and the vehicle image to be matched is used to represent the approximation degree between the two vehicle images. The smaller the feature distance value, the higher the approximation degree; on the contrary, the larger the feature distance value, the lower the approximation degree.

[0127] As an embodiment of the present application, the target vehicle is re-identified according to the approximation degree, which comprises:

[0128] The minimum value in the plurality of feature distance values is obtained.

[0129] If the minimum value in the plurality of feature distance values is less than a preset threshold value, the target vehicle re-identification is successful, and the vehicle image of the re-identified success is deleted from the corresponding image library.

[0130] In the embodiment, it is assumed that the target vehicle library has S1 vehicle images, and the image library to be matched has S2 vehicle images. For any target vehicle image in the target image library, S2 feature distance values are obtained. The minimum value in the S2 feature distance values is obtained, and it is judged whether the minimum value is less than a preset threshold value. The preset threshold value is set between 0.45 and 0.6. If the minimum value is less than the preset threshold value, it is indicated that the approximation degree of the two vehicle images is high, and the two vehicle images are successfully matched, that is, the vehicle re-identification is successful.

[0131] If the vehicle re-identification is successful, the camera information of the image of which the vehicle re-identification is successful is recorded, and the image of which the vehicle re-identification is successful is deleted from the library.

[0132] As another embodiment of the application, the re-identification of the target vehicle according to the approximation degree further includes:

[0133] The minimum value in the S2 feature distance values is obtained.

[0134] If the minimum value in the S2 feature distance values is not less than the preset threshold value, the re-identification of the target vehicle fails, and the local feature information, the global feature information and the mask information of the target vehicle are saved for subsequent vehicle re-identification.

[0135] In the above embodiment, for any target vehicle image in the target image library, S2 feature distance values are obtained. The minimum value in the S2 feature distance values is obtained, and it is judged whether the minimum value is less than a preset threshold value. The preset threshold value is set between 0.45 and 0.6. If the minimum value is not less than the preset threshold value, that is, greater than or equal to the preset threshold value, it is indicated that the approximation degree of the two vehicle images is low, and the two vehicle images are not successfully matched, that is, the vehicle re-identification fails.

[0136] In the next image acquisition cycle, the local feature and the global feature of the newly extracted image are obtained together with the features that are not successfully re-identified in the last cycle, and the approximation degree of the image features is calculated again to perform vehicle re-identification.

[0137] According to the embodiment of the application, the vehicle re-matching under multiple cameras is intelligently realized by using a computer vision algorithm, which replaces the traditional manual detection method and saves manpower and resources. Meanwhile, the vehicle re-matching is not dependent on basic information such as license plates and vehicle body colors.

[0138] According to the embodiment of the application, the local feature and the global feature are extracted respectively, and the two kinds of features are combined for vehicle re-identification. The performance and efficiency of vehicle re-identification are greatly improved.

[0139] According to an embodiment of the present application, the Transformer technology is introduced for global target extraction, which not only has excellent performance in feature extraction, but also solves the problem of loss of detail features when the convolutional neural network extracts features of small pictures, and improves the ability of re-identification of small size pictures.

[0140] According to an embodiment of the present application, when extracting local features, the target segmentation + local feature extraction technology is introduced, and the feature weight is added, which improves the ability of local feature comparison.

[0141] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0142] In the technical scheme of the present application, the acquisition, storage and application of user personal information comply with relevant laws and regulations and do not violate public order and good customs.

[0143] According to an embodiment of the present application, the present application also provides an electronic device.

[0144] Figure 8 A schematic block diagram of an electronic device 800 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application described and / or claimed in this document.

[0145] The device 800 includes a computing unit 801 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other through a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0146] A plurality of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0147] The computing unit 801 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the methods S101-S105. For example, in some embodiments, the methods S101-S105 can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the methods S101-S105 described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the methods S101-S105 by any other appropriate means, such as by means of firmware.

[0148] Various implementations of the systems and techniques described above herein can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0149] Program code for carrying out operations of the methods of the present application can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or entirely on a remote machine or server.

[0150] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable storage medium can include, without limitation, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the foregoing.

[0151] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0152] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0153] The computer system can include clients and servers. The clients and the servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0154] It should be understood that the steps shown in the various forms above can be reordered, added to, or removed. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, without limitation, as long as the desired results of the present disclosure are achieved.

[0155] The specific embodiments described above are not intended to limit the scope of the present application. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present application. Any further modifications, equivalents, and / or alternatives come within the scope of the present application as recited by the claims.

Claims

1. A multi-camera based vehicle re-identification method, characterized in that, The method comprises the following steps: acquiring initial images collected by a plurality of cameras; performing target detection on the initial images by using a target detection algorithm to acquire a category to which a target belongs and extract target vehicle images in the initial images and save the target vehicle images to a target image library; saving to-be-matched vehicle images to a to-be-matched image library; performing target segmentation on the images in the target image library and the to-be-matched image library to obtain target segmentation results; encoding the images in the target image library and the to-be-matched image library, introducing a plurality of vehicle features to encode the vehicles to obtain encoding information of the images; inputting the encoding information of the images into a multi-scale hierarchical feature extraction network based on a Transformer to acquire global features of the images; and extracting local features of the images according to a convolutional neural network and the target segmentation results; acquiring target vehicle images from the target image library, calculating the degrees of approximation between the target vehicle images and all to-be-matched vehicle images in the to-be-matched image library according to the global features and the local features of the target vehicle images, and re-identifying the target vehicle according to the degrees of approximation; the encoding of the images in the target image library and the to-be-matched image library to obtain the encoding information of the images comprises: dividing each image in the target image library and the to-be-matched image library into a plurality of equal-area image blocks, encoding the positions of the image blocks in the corresponding images to obtain position encoding information of the image blocks; performing convolution and linear normalization processing on image information of the image blocks to obtain image encoding information; obtaining target category encoding information according to the category to which the target belongs; obtaining target segmentation result encoding information according to the target segmentation results; summing up the image encoding information, the target category encoding information, the target segmentation result encoding information and the position encoding information of the image blocks as the encoding information of the current image.

2. The method of claim 1, wherein, the acquisition of the initial images collected by the plurality of cameras comprises: selecting a plurality of cameras, wherein one camera is used as a collection camera to acquire target images, and the other cameras are used as to-be-matched cameras to acquire to-be-matched images; setting a time interval for image collection, and collecting to-be-matched images and target images according to the time interval as initial images.

3. The method of claim 2, wherein, the target detection on the initial images by using a target detection algorithm to acquire a category to which a target belongs and extract target vehicle images in the initial images and save the target vehicle images to a target image library; and saving the to-be-matched vehicle images to a to-be-matched image library comprises: inputting the initial images into a deep learning convolutional neural network for target detection to output position information and category information of all vehicles in the initial images in the corresponding initial images, and marking all vehicles in the initial images with rectangular frames; extracting the parts in the rectangular frames to obtain target vehicle images and to-be-matched vehicle images.

4. The method of claim 1, wherein, the target segmentation on the images in the target image library and the to-be-matched image library to obtain target segmentation results comprises: The image segmentation algorithm is used to segment each image in the target image library and the to-be-matched image library to obtain a plurality of region segmentation blocks as the target segmentation result of each image, and the region segmentation blocks are used to describe the front part region, the top part region, the side part region, the tail part region and the background region in the image.

5. The method of claim 1, wherein, The Transformer-based feature extraction network comprises a first Transformer structure layer, a second Transformer structure layer, a third Transformer structure layer and a fourth Transformer structure layer, wherein the first Transformer structure layer, the second Transformer structure layer and the fourth Transformer structure layer each comprise two Transformer structures; the fourth Transformer structure layer comprises six Transformer structures; and one up-sampling layer is arranged between adjacent Transformer structure layers.

6. The method of claim 1, wherein, According to the convolutional neural network and the target segmentation result, local features of the image are extracted, comprising: The image features of each image in the target image library and the to-be-matched image library are extracted by the ResNet convolutional neural network; According to the target segmentation result, a mask corresponding to each part is made; The image features of each image are multiplied by the mask corresponding to the part, and then linearly transformed to obtain the local features of the current image.

7. The method of claim 1, wherein, According to the global features and the local features of the target vehicle image, the approximate degree between the target vehicle image and all to-be-matched vehicle images in the to-be-matched image library is calculated, comprising: The target vehicle image traverses all to-be-matched vehicle images in the to-be-matched image library, the local feature Euclidean distance between the target vehicle image and the to-be-matched vehicle image corresponding part is calculated, and the weight of the corresponding part is accumulated to obtain a plurality of local feature distance values; and the global feature Euclidean distance between the target vehicle image and the to-be-matched vehicle image is calculated to obtain a plurality of global feature distance values; The local feature distance values and the global feature distance values corresponding to the same to-be-matched vehicle image are accumulated to obtain a plurality of feature distance values between the target vehicle image and the to-be-matched vehicle image.

8. The method of claim 7, wherein, According to the approximate degree, the target vehicle is re-identified, comprising: The minimum value in the plurality of feature distance values is obtained; If the minimum value in the plurality of feature distance values is less than a preset threshold, the target vehicle re-identification is successful, and the vehicle image of the re-identification success is deleted from the corresponding image library; If the minimum value in the plurality of feature distance values is not less than the preset threshold, the target vehicle re-identification fails, and the local feature information, the global feature information and the mask information of the target vehicle are saved.

9. An electronic device, comprising at least one processor; and a memory connected in communication with the at least one processor; characterized in that, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method in any one of claims 1-8.

Images