A cross-camera vehicle re-identification method in highway scenes

Abstract

The embodiment of the present invention discloses a cross-camera vehicle re-identification method in a highway scene, including: using highway monitoring video to obtain two vehicle images to be matched on the highway, each vehicle image includes a vehicle; using a semantic segmentation network to semantically segment each vehicle image, and obtain images of multiple regions of each vehicle, wherein the multiple regions include: front face, side, roof and front windshield; inputting the images of the same region of the two vehicles into two branches of the twin network respectively, extracting the features before the last layer of the twin network as the difference features of the two vehicles in the same region; combining the difference features of the two vehicles in each region into the overall difference features, and calculating the vehicle similarity based on the overall difference features to achieve vehicle re-identification. This embodiment improves the accuracy of vehicle re-identification.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of intelligent transportation technology, and in particular to a cross-camera vehicle re-identification method in a highway scenario. Background Art

[0002] Vehicle re-identification is an important technology in highway scenarios, and its applications are in many fields such as security monitoring, intelligent transportation, vehicle pursuit and abnormal event detection. Currently, most of the research on vehicle re-identification methods directly adopts the twin network structure based on deep learning, divides the vehicle image into foreground and background, and uses different neural network branches to extract vehicle features and background features respectively.

[0003] However, these methods have the following problems when dealing with vehicle re-identification in highway scenarios: the vehicle perspective in highway scenarios is relatively single, which makes it difficult to extract vehicle features, and due to the influence of the distance factor of the surveillance camera, the acquired vehicle images often have low resolution and unclear features, resulting in the inability to accurately extract and compare features, affecting the accuracy of re-identification. Summary of the invention

[0004] The embodiment of the present invention provides a cross-camera vehicle re-identification method in a highway scene, so as to solve the problem that in the highway scene, the vehicle has a single viewing angle, the image resolution is low, and the feature cannot be accurately extracted, resulting in low re-identification accuracy.

[0005] In a first aspect, an embodiment of the present invention provides a cross-camera vehicle re-identification method in a highway scenario, comprising:

[0006] Using highway surveillance video, two vehicle images to be matched on the highway are obtained, each of which includes a vehicle;

[0007] Using a semantic segmentation network to perform semantic segmentation on each vehicle image, respectively, to obtain images of multiple regions of each vehicle, wherein the multiple regions include: front face, side, roof and front windshield;

[0008] Inputting the pictures of the same area of ​​the two vehicles into the two branches of the twin network respectively, extracting the features before the last layer of the twin network as the difference features of the two vehicles in the same area;

[0009] The difference features of the two vehicles in each area are combined into an overall difference feature, and the vehicle similarity is calculated based on the overall difference feature to achieve vehicle re-identification.

[0010] In a second aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0011] one or more processors;

[0012] a memory for storing one or more programs,

[0013] When the one or more programs are executed by the one or more processors, the one or more processors implement the cross-camera vehicle re-identification method in a highway scenario described in any embodiment.

[0014] The embodiment of the present invention provides a cross-camera vehicle re-identification method in a highway scene, which realizes vehicle re-identification through semantic segmentation, twin network and feature combination. Specifically, in view of the problem that the current re-identification method over-focuses on the features of a single image and ignores the features of the image pair, the image pair is obtained for vehicle re-identification; the vehicle is segmented into five regions, namely the front face, side, roof, front windshield and wheel, through the semantic segmentation network, so as to make more full use of the vehicle detail features; in view of the problem that the traditional CBAM attention mechanism does not pay enough attention to the key areas, the CBAM attention mechanism is improved, and the areas with poor learning effect are secondary learned, and the twin network is constructed using the improved CBAM and VGG1 6 as the backbone, and the features of the four parts of the vehicle pair, namely the front face, side, roof and front windshield, are learned respectively, and the vector generated before the last layer is extracted as the regional features of the vehicle image pair; the MLP is used to combine the regional features, learn the association between the parts, and output the final similarity. This method can use the vehicle appearance information to fully learn the regional features of each part of the vehicle when the vehicle perspective is single, the resolution is low, and the license plate cannot be recognized. The improved twin network is trained using the combined regional features, effectively realizing vehicle re-identification in highway scenarios with high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 A flowchart of a cross-camera vehicle re-identification method in a highway scenario provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the structure of a twin network provided by an embodiment of the present invention;

[0018] Figure 3 A comparison diagram of the effects of a traditional CBAM attention mechanism and an improved CBAM attention mechanism provided in an embodiment of the present invention;

[0019] Figure 4 A flowchart of another cross-camera vehicle re-identification method in a highway scenario provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0022] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0023] In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0024] Figure 1 1 is a flow chart of a method for re-identifying a vehicle across multiple cameras in a highway scenario provided by an embodiment of the present invention. The method is applicable to the case where vehicle re-identification is performed using vehicle images of multiple cameras on a highway, and is executed by an electronic device. Figure 1 As shown, the method specifically includes:

[0025] S110. Using the highway surveillance video, obtain two vehicle images to be matched on the highway, each vehicle image includes a vehicle.

[0026] Here, the two vehicle images are images from different cameras on the highway. This embodiment will identify whether the vehicles in the two images are the same vehicle. Optionally, several consecutive cameras on the side of the highway are selected, and all vehicles under the camera are detected using the target detection method. When the detection frame area of ​​a certain vehicle reaches the maximum, the area inside the detection frame is saved as an image, and the two vehicle images from different cameras are used as vehicle images to be matched. Exemplarily, the Yolov5 algorithm can be used for target detection.

[0027] More specifically, all vehicle images captured by the previous camera are saved in the image library. The images captured by the next camera are matched one by one with the vehicle images captured by the previous camera within a certain time range during the process of being saved in the image library as the objects of vehicle re-identification. During target detection, if there is only one vehicle in the detection frame, a screenshot can be taken when the detection frame is about to disappear (the vehicle drives out of the shooting range), which can ensure that only one vehicle is included in the screenshot.

[0028] S120, performing semantic segmentation on each vehicle image using a semantic segmentation network to obtain images of multiple regions of each vehicle, wherein the multiple regions include: a front face, a side face, a roof, and a front windshield.

[0029] As described above, the vehicle images captured by highway cameras have certain disadvantages in feature extraction. Therefore, this embodiment subdivides the vehicle images on the highway into multiple areas to facilitate more comprehensive learning of vehicle detail features in the subsequent process.

[0030] Optionally, this embodiment optimizes the PSPnet backbone network to Mobilenetv2. As a semantic segmentation network, Mobilenetv2 can learn the features of images in higher dimensions. The input of the semantic segmentation network is a vehicle picture, and the output is the probability that each pixel in the vehicle picture belongs to each region. The region with the highest probability of each pixel is taken as the region to which each pixel belongs; according to the region to which each pixel belongs, the picture of each region can be obtained. Optionally, the picture size of each region is the same as the vehicle picture. In the picture of a certain area, the pixels of the area keep the original value, and the pixel values ​​of other areas are 0. Exemplarily, in the picture of the front face area, only the front face area keeps the same pixel value as the vehicle picture, and the pixel values ​​of other areas are 0.

[0031] S130, respectively inputting pictures of the same area of ​​the two vehicles into two branches of the twin network, and extracting features before the last layer of the twin network as differential features of the two vehicles in the same area.

[0032] This embodiment constructs a twin network for each area, which is used to calculate the similarity of the same area of ​​the two vehicles. The specific front face area corresponds to a twin network, the network input is the picture of the front face area of ​​the two vehicles, and the output is the similarity of the front face area of ​​the two vehicles. This embodiment extracts the features before the last layer of the twin network as the difference features of the two vehicles in the front face area. This feature characterizes the difference of the front face area of ​​the two vehicles from multiple dimensions; similarly, the side area corresponds to another twin network, the network input is the picture of the side area of ​​the two vehicles, and the output is the similarity of the side area of ​​the two vehicles. This embodiment extracts the features before the last layer of the twin network as the difference features of the two vehicles in the side area. This feature characterizes the difference of the side area of ​​the two vehicles from multiple dimensions; the roof and the front windshield are similar, and the structures of each twin network are the same but the weights are different.

[0033] In a specific implementation, each subnetwork of the twin network includes an improved CBAM attention mechanism and a VGG network. Optionally, taking the twin network of the front windshield area as an example, its network structure is as follows: Figure 2 As shown in the figure, each subnetwork (i.e., each branch) includes an improved CBAM attention mechanism and a VGG16 network without the Flatten layer and the Dense layer, which are respectively used to learn the representation of the front windshield regions of the two vehicles embedded in the high-dimensional space. The two branches are connected to the L1 distance calculation layer and the Dense network, which are used to calculate the similarity of the front windshield regions of the two vehicles based on the two representations embedded in the high-dimensional space. The operation process of each part is described in detail below.

[0034] The traditional CBAM attention mechanism includes a channel attention layer and a spatial attention layer. In order to make up for the disadvantages of highway camera data in feature extraction, this embodiment proposes an improved CBAM attention mechanism, which performs secondary learning on the areas that are not sufficiently learned in the traditional CBAM attention mechanism to more fully extract image features. Specifically, after the original images of the same area of ​​the two vehicles are respectively input into the two branches of the twin network, for any original image F, first, it passes through the channel attention layer and the spatial attention layer in sequence to obtain an attention mask M 1 (F); Perform a dot product between the attention mask and the original image to obtain a feature map M of the region 2 (F), the formula is:

[0035] M 1 (F) = σ(M s (M c (F))) (2)

[0036] M 2 (F)=M 1 (F)·F (3)

[0037] Among them, Mc (·) is the operation of the channel attention layer, M s (·) is the operation of the spatial attention layer, σ(·) is the activation function, M 2 (F) is the output of the traditional CBAM attention mechanism. Through the CBAM attention mechanism, the interesting parts of the image can be enhanced while the less relevant parts can be weakened, making the key features in the area more obvious.

[0038] However, the portion M in the original image that is not sufficiently focused on is 3 (F), re-learning through the spatial attention layer to obtain another attention mask; dot product of the other attention mask and the original image to obtain another feature map M of the region 4 (F), the formula is as follows:

[0039] M 3 (F)=(1-M 1 (F))·F (4)

[0040] M 4 (F) = σ(M S (M 3 (F)))·F (5)

[0041] Among them, M 2 (F)=M 1 (F)·F represents the part of the original image that is fully focused on, then M 3 (F)=(1-M 1 (F))·F represents the part of the original image that is not sufficiently focused, or the part that is not fully learned. For the convenience of distinction and description, the one attention mask and the one feature map obtained in the traditional CBAM attention mechanism are called the first attention mask and the first feature map, and the other mask and the other feature map obtained by secondary learning here are called the second attention mask and the second feature map.

[0042] Finally, the first feature map and the second feature map are fused, and the spatial attention layer is used for three learnings to obtain another attention mask; the attention mask is dot-producted with the original image to obtain the final feature map of the region. The formula is as follows:

[0043] M(F)=σ(M S ([M 2 (F); M 4 (F)]))·F (6)

[0044] It should be noted that the above-mentioned insufficient learning means that the areas focused on by the traditional CBAM attention mechanism are not distinguishable enough, and more features need to be learned to enhance the distinguishability. Figure 3 An exemplary comparison diagram of the first feature map and the final feature map is given, where the left figure is the first feature map obtained by the traditional CBAM attention mechanism, and the right figure is the final feature map obtained by the improved CBAM attention mechanism. It can be seen that the distinguishable features paid attention to by the left figure are obviously not as sufficient as those in the right figure.

[0045] Continue to refer Figure 2 , after obtaining the final feature map of the two vehicles in the same area through the improved CBAM attention mechanism, it is processed by the VGG16 network and outputs its representation embedded in the high-dimensional space. Then, the two representations are input into the L1 distance calculation layer, and the L1 distances of the two representations in each dimension are calculated respectively, and the L1 distance vector is formed by the L1 distances in each dimension. Optionally, the calculation formula of L1 distance is:

[0046]

[0047] Among them, I 1 ,I 2 They represent the representation of two images embedded in a high-dimensional space, both of which are P-dimensional vectors; and Represents the i-th dimension data of the two representations, L1 i (I 1 , I 2 ) represents the L1 distance between two representations in the i-th dimension. 1 , L1 2 , ...L1 P The constructed P-dimensional distance vector is used as the output of the distance calculation layer, and the distance vector can characterize the difference (or similarity) between the two images in each dimension. Finally, the L1 distance vector is input into the Dense network, and the features before the last layer of the Dense network are extracted as the difference features of the two vehicles in the same area (referred to as regional features), wherein the Dense network is used to output the similarity of the two vehicles in the same area. Exemplarily, the vector generated before the last layer is extracted in this embodiment is 512-dimensional.

[0048] S140: Combining the difference features of the two vehicles in each region into an overall difference feature, and calculating the vehicle similarity based on the overall difference feature to achieve vehicle re-identification.

[0049] The entire process of S110-S140 in this embodiment can be used Figure 4The network shown in the figure shows that the "acquiring regional features" part includes the operations of S110-S130. After obtaining the difference features of the two vehicles in each region, the features are combined into the difference features of the two vehicles as a whole, which are used to characterize the difference of the two vehicles as a whole; and then the feature vector is processed into the similarity of the whole vehicle using the shallow network MLP autoencoder.

[0050] It should be noted that Figure 2 The similarity and loss functions in , and Figure 4 The loss functions in are all used in the network training phase. S110-S140 describe the operations in the network use phase and do not include Figure 2 Calculate the similarity and loss function in , and Figure 4 It can be seen that S110vS140 involves multiple neural networks, including semantic segmentation network, twin network and shallow network. In a specific implementation, the same vehicle image dataset can be used to train the above networks, which specifically includes the following steps:

[0051] Step 1: Construct a vehicle image dataset for sharing among all networks. Optionally, first select several continuous cameras on the side of the highway, use the target tracking method to detect and track all vehicles under the camera, and when the detection frame area of ​​a certain vehicle reaches the maximum, save the area inside the detection frame as a picture as the image sample of the vehicle. Exemplarily, the Yolov5-Deepsort algorithm can be used to track the vehicle. Then, find several groups of vehicle pair pictures in the image sample and construct a vehicle pair picture dataset, wherein each group of vehicle pair pictures includes two pictures of the same vehicle taken by different cameras. Exemplarily, two pictures of the same vehicle can be identified from pictures from different cameras by manual analysis and comparison as a group of vehicle pair pictures. At the same time, several single vehicle pictures (no need to appear in pairs) can also be collected for subsequent hard negative sample searches. Finally, all vehicle pictures are image enhanced, and the illumination, hue, and angle are randomly changed within a certain range to obtain the final vehicle picture dataset.

[0052] Step 2: Use the vehicle image dataset to train the semantic segmentation network. Optionally, first, manually annotate a sufficient number of images in the vehicle image dataset, divide the vehicle into five areas: front face, side, roof, front windshield, and wheel, and establish an image sample library for the semantic segmentation network. Then, load the official pre-trained weights of Mobilenetv2 in the semantic segmentation network to speed up the convergence of model training. Next, use the image sample library to train the semantic segmentation network after loading the pre-trained weights. The loss function consists of two parts, one part is Cross Entropy Loss, and its formula is:

[0053]

[0054] Where W is the width of the image, H is the height of the image, and t is the number of region categories (=5). is the probability that each pixel in the vehicle image actually belongs to each region, The semantic segmentation network predicts the probability that each pixel in the vehicle image belongs to each region.

[0055] The other part is Dice Loss, and its formula is:

[0056]

[0057] in, is the pixel set actually contained in each region, Yi is the pixel set actually contained in each region predicted by the semantic segmentation network, S(·) represents the operation of taking the area of ​​the pixel set, and ∩ represents the intersection of two pixel sets. For the convenience of distinction and description, the front face, side, roof, front windshield, and wheel can be divided into five areas called vehicle areas, and the connected area composed of multiple pixels in the image is called the image area. Then is the pixel area actually contained in each vehicle area, Y i The semantic segmentation network predicts the pixel area actually contained in each vehicle area. S(·) represents the operation of taking the area of ​​the pixel area, and ∩ represents the common area of ​​two pixel areas.

[0058] The final loss function is:

[0059] L1=λ 1 ×A+λ 2 ×B (10)

[0060] Step 3: Use the trained semantic segmentation network and vehicle image dataset to train each twin network. Optionally, this embodiment adopts a three-stage training method to enhance the network's learning effect on hard negative samples. Taking the twin network corresponding to the front windshield area as an example, its training process may include the following steps:

[0061] S1. Using vehicle images in the vehicle image dataset as samples, pre-train the sub-network of the twin network using the autoencoder method to minimize the difference between the restored image and the image input to the self-network. Figure 2, each sub-network includes an improved CBAM attention mechanism and a VGG16 network without the Flatten layer and the Dense layer. During training, a sub-network of any branch is regarded as an autoencoder ENCODER, and an autodecoder DECODER with a completely symmetrical structure with the autoencoder is added after the sub-network; a vehicle picture is input into the autoencoder to obtain the deep features of the vehicle; the deep features are then input into the autodecoder to obtain the restored image of the vehicle picture; the parameters of the autoencoder and the autodecoder are updated by the following loss function:

[0062]

[0063] Among them, W and H are the length and width of the picture respectively. is the value of each pixel in the vehicle image, P ij To restore the value of each pixel in the image. After the parameters are updated, the vehicle pictures on the highway are used as samples to input the operation of the autoencoder, and the next sample cycle is entered until the set termination condition is reached. After the autoencoder training is completed, only the parameters of the ENCODER part are retained to initialize the branches of the twin network, and the DECODER is discarded.

[0064] S2. Take two pictures of the same vehicle and the same area as positive sample pairs, and two pictures of different vehicles in the same area as negative sample pairs, and perform rough training on the pre-trained twin network to maximize the similarity of the positive sample pairs and minimize the similarity of the negative sample pairs. Optionally, first, take the vehicle pair images extracted from the above vehicle image dataset as the positive sample pairs of the vehicles, and extract one picture from each positive sample as the negative sample pair of the vehicles; then, use the trained semantic segmentation network to perform semantic segmentation on each vehicle image in each sample pair, obtain the positive sample pairs and negative sample pairs of the front windshield area, input them into the pre-trained twin network, and calculate the similarity of the two vehicles in the front windshield area. After each calculation, the weights of each sub-network and the Dense network are updated through the following loss function until the set termination condition is reached to complete the rough training of the twin network:

[0065]

[0066] in, and This is a picture of the same area for two vehicles. Y is the similarity of two vehicles in the same area predicted by the twin network; i is the label, if the two vehicles are the same, Y i is 1, otherwise Y i is 0; N is the number of samples in the same batch, and m is a hyperparameter.

[0067] S3. Calculate the similarity of multiple negative sample pairs using the coarsely trained twin network, select the negative sample pairs with similarity greater than the set threshold as hard negative sample pairs; use the positive sample pairs and the hard negative sample pairs to perform fine training on the coarsely trained twin network to maximize the similarity of the positive sample pairs and minimize the similarity of the hard negative sample pairs. Optionally, select any two single vehicle images from the above-mentioned vehicle image dataset as negative sample pairs of the vehicle, obtain negative sample pairs of the front windshield area after processing by the semantic segmentation network, and input them into the coarsely trained twin network to obtain the similarity of the front windshield area of ​​the two vehicles. If the similarity is greater than the set threshold (for example, 0.7), the negative sample pair is regarded as a hard negative sample pair, which is mostly a pair of images with similar colors and models, but not of the same vehicle. Take the hard negative sample pair as the new negative sample pair, still use the original positive sample pair as the positive sample pair, input the coarsely trained twin network, and train the twin network again. The loss function is the same as the coarse training process.

[0068] The training process of the twin networks in the remaining regions is the same. Only the front windshield area needs to be replaced by the other three areas, which will not be described in detail.

[0069] Step three, use the trained semantic segmentation network and twin network, as well as the vehicle image data set, to train the shallow network. Optionally, in step two, after the vehicle-level positive and negative sample pairs pass through the semantic segmentation network and each twin network, extract the features before the last layer of each twin network as the differential features of each vehicle pair in each area, and combine the differential features of the same vehicle pair in each area into the overall differential features as the samples of the shallow network; input each sample into the shallow network to obtain the overall similarity of the two vehicles. During the training process, the following loss function is used to update the parameters of the shallow network to maximize the similarity of the same vehicle and minimize the similarity of different vehicles:

[0070]

[0071] in, and They are two pictures of vehicles on the highway. is the overall difference feature of the two vehicle images, for The vehicle similarity obtained after the shallow network; Y i is the label, if the vehicle is the same, Y i is 1, otherwise Y i is 0; N is the number of samples in the same batch.

[0072] In summary, this embodiment provides a cross-camera vehicle re-identification method in a highway scenario, and realizes vehicle re-identification through PSPnet semantic segmentation, twin network and MLP feature combination. Specifically, in view of the problem that the current re-identification method over-focuses on the features of a single image and ignores the features of the image pair, a picture pair dataset and a hard negative sample set are constructed to enhance the network recognition ability; by optimizing the backbone network, a PSPnet semantic segmentation model is constructed for Mobilenetv2, and the vehicle is segmented into five regions: front face, side, roof, front windshield and wheel, so as to make more full use of the vehicle detail features; in view of the problem that the traditional CBAM attention mechanism does not pay enough attention to the key areas, the CBAM attention mechanism is improved, and the areas with poor learning effect are secondary learned, and the twin network is constructed using the improved CBAM and VGG16 as the backbone; the features of the four parts of the front face, side, roof and front windshield of the vehicle pair are learned separately, and the vector generated before the last layer is extracted as the regional features of the vehicle image pair; the MLP is used to combine the regional features, learn the association between the parts, and output the final similarity. This method can use the vehicle appearance information to fully learn the regional features of each part of the vehicle when the vehicle perspective is single, the resolution is low, and the license plate cannot be recognized. The improved twin network is trained using the combined regional features, effectively realizing vehicle re-identification in highway scenarios with high accuracy.

[0073] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the device includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more. Figure 5 A processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0074] The memory 61 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the cross-camera vehicle re-identification method in the highway scenario in the embodiment of the present invention. The processor 60 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 61, that is, realizing the cross-camera vehicle re-identification method in the highway scenario.

[0075] The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 61 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some examples, the memory 61 may further include a memory remotely arranged relative to the processor 60, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0076] The input device 62 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 63 may include a display device such as a display screen.

[0077] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-camera vehicle re-identification method in a highway scenario of any embodiment.

[0078] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0079] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0080] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0081] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. A cross-camera vehicle re-identification method in a highway scene, characterized in that: include: Using highway surveillance video, two vehicle images to be matched on the highway are obtained, each of which includes a vehicle; Using a semantic segmentation network to perform semantic segmentation on each vehicle image, respectively, to obtain images of multiple regions of each vehicle, wherein the multiple regions include: front face, side, roof and front windshield; The pictures of the same area of ​​the two vehicles are respectively input into the two branches of the twin network, and the features before the last layer of the twin network are extracted as the differential features of the two vehicles in the same area; wherein the twin network includes two feature extraction branches, a distance calculation layer and a dense network; specifically, the pictures of the same area of ​​the two vehicles are respectively input into the two feature extraction branches of the twin network, and the representations of the two pictures embedded in the high-dimensional space are respectively extracted; the two representations are input into the distance calculation layer, and the distances of the two representations in each dimension are respectively calculated, and the distance vector is formed by the distances in each dimension, wherein, Among them, I1 and I2 represent the representation of two pictures embedded in high-dimensional space, both of which are P-dimensional vectors. and They represent the i-th dimension data of the two representations respectively, L1i(I1,I2) represents the L1 distance between the two representations on the i-th dimension, and L11,L12,…L1 P The constructed P-dimensional distance vector is used as the output of the distance calculation layer; the distance vector is input into the Dense network, and the features before the last layer of the Dense network are extracted as the difference features of the two vehicles in the same area, wherein the Dense network is used to output the similarity of the two vehicles in the same area; The difference features of the two vehicles in each area are combined into an overall difference feature, and the vehicle similarity is calculated based on the overall difference feature to achieve vehicle re-identification; The steps of inputting the images of the same area of ​​the two vehicles into two feature extraction branches of the twin network respectively, and extracting the representations of the two images embedded in the high-dimensional space respectively, include: The original image F of any area of ​​any vehicle is passed through the channel attention layer and the spatial attention module layer of any branch in sequence to obtain the first attention mask M1(F); the first attention mask is dot-producted with the original image to obtain the first feature map M2(F) of the area, which is expressed by the formula: M1(F)=σ(M s (M c (F))) (2) M2(F)=M1(F)·F (3) Among them, M c (·) is the operation of the channel attention layer, M s (·) is the operation of the spatial attention layer, σ(·) is the activation function; The part M3(F) that is not sufficiently focused in the original image is re-learned through the spatial attention layer to obtain a second attention mask; the second attention mask is dot-producted with the original image to obtain a second feature map M4(F) of the region, which is expressed as follows: M3(F)=(1-M1(F))·F (4) M4(F)=σ(M S (M3(F)))·F (5) The first feature map and the second feature map are fused, and the spatial attention layer is used for three learning again to obtain a third attention mask; the third attention mask is dot-producted with the original image to obtain the final feature map M(F) of the region, which is expressed as follows: M(F)=σ(M S ([M2(F);M4(F)]))·F (6)。 2. The method according to claim 1, characterized in that The semantic segmentation network is used to perform semantic segmentation on each vehicle image, and images of multiple regions of each vehicle are obtained, including: Optimize the PSPnet backbone network to the Mobilenetv2 structure as a semantic segmentation network; Using the semantic segmentation network to perform semantic segmentation on any vehicle image, and obtain the probability that each pixel in the vehicle image belongs to each region; The region with the highest probability for each pixel is regarded as the region to which each pixel belongs; According to the area to which each pixel belongs, a picture of each area is obtained.

3. The method according to claim 1, characterized in that Before using the semantic segmentation network to perform semantic segmentation on each vehicle image and obtain images of multiple regions of each vehicle, the following steps are also included: Optimize the PSPnet backbone network to the Mobilenetv2 structure as a semantic segmentation network, and initialize the semantic segmentation network according to the official pre-trained weights of Mobilenetv2; Taking vehicle images on the highway as samples, the initialized semantic segmentation network is trained using the following loss function: Where W is the width of the image, H is the height of the image, and t is the number of region categories. is the probability that each pixel in the vehicle image actually belongs to each region, The semantic segmentation network predicts the probability that each pixel in the vehicle image belongs to each region; is the pixel set actually contained in each region, Y i The semantic segmentation network predicts the pixel sets actually contained in each region, S(·) represents the operation of taking the area of ​​the pixel set, and ∩ represents the intersection of two pixel sets.

4. The method according to claim 1, characterized in that: Before respectively inputting the pictures of the same area of ​​the two vehicles into the two branches of the twin network and extracting the features before the last layer of the twin network as the difference features of the two vehicles in the same area, the method further includes: Taking vehicle images on the highway as samples, the sub-network of the twin network is pre-trained using the autoencoder method to minimize the difference between the restored image and the image input into the self-network; Two pictures of the same vehicle and the same area are used as positive sample pairs, and two pictures of different vehicles and the same area are used as negative sample pairs. The pre-trained twin network is roughly trained to maximize the similarity of the positive sample pairs and minimize the similarity of the negative sample pairs. The coarsely trained Siamese network is used to calculate the similarity of multiple negative sample pairs, and the negative sample pairs with similarity greater than a set threshold are selected as hard negative sample pairs; the coarsely trained Siamese network is finely trained using the positive sample pairs and the hard negative sample pairs to maximize the similarity of the positive sample pairs and minimize the similarity of the hard negative sample pairs.

5. The method according to claim 4, characterized in that The method uses the vehicle pictures on the highway as samples and uses the autoencoder method to pre-train the sub-network of the twin network to minimize the difference between the restored pictures and the pictures input into the self-network, including: A sub-network of any regional twin network is used as an autoencoder, and an autodecoder having a symmetrical structure with the autoencoder is added after the sub-network; Inputting a vehicle picture on a highway as a sample into the autoencoder to obtain a deep feature of the vehicle; inputting the deep feature into the autodecoder to obtain a restored picture of the vehicle; The parameters of the autoencoder and autodecoder are updated by the following loss function: Among them, W and H are the length and width of the picture respectively. is the value of each pixel in the vehicle image, P ij To restore the value of each pixel in the image; Return to the operation of inputting the autoencoder with the vehicle picture on the highway as a sample, and enter the loop of the next sample until the set termination condition is reached.

6. The method according to claim 4, characterized in that The method uses two pictures of the same area of ​​the same vehicle as positive samples and two pictures of the same area of ​​different vehicles as negative samples, and performs rough training on the pre-trained twin network to maximize the similarity of positive samples and minimize the similarity of negative samples, including: The pre-trained twin network is roughly trained using the following loss function L2: in, and This is a picture of the same area for two vehicles. Y is the similarity of two vehicles in the same area predicted by the twin network; i is the label, if the two vehicles are the same, Y i is 1, otherwise Y i is 0; N is the number of samples in the same batch, and m is a hyperparameter.

7. The method according to claim 1, characterized in that The calculating the vehicle similarity according to the overall difference feature comprises: inputting the overall difference feature into a shallow network to obtain the vehicle similarity; Before inputting the overall difference features into the shallow network to obtain the vehicle similarity, the method further includes: taking the overall difference features of two vehicle images on the highway as samples, training the shallow network through the following loss function to maximize the similarity of the same vehicle and minimize the similarity of different vehicles: in, and They are two pictures of vehicles on the highway. is the overall difference feature of the two vehicle images, for The vehicle similarity obtained after the shallow network; Y i is the label, if the vehicle is the same, Y i is 1, otherwise Y i is 0; N is the number of samples in the same batch.

8. An electronic device, characterized in that: include: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the cross-camera vehicle re-identification method in a highway scenario as described in any one of claims 1-7.

Images