A vehicle retrieval method, device, equipment and medium based on multi-task learning

Abstract

The application provides a vehicle retrieval method based on multi-task learning, comprising the following steps: obtaining a training data set, a verification data set and a test data set containing vehicle image label information; inputting data of the training data set according to image serial numbers into an initial vehicle retrieval model based on multi-task learning to train a vehicle retrieval model; inputting data in the verification data set into the trained vehicle retrieval model to obtain a preliminary sorting result; and re-sorting the preliminary sorting result by using a multi-task retrieval strategy to obtain a final sorting result. The application guides the extraction of vehicle image features in a multi-task learning manner, designs camera perspective retrieval and vehicle orientation retrieval as auxiliary tasks, eliminates the deviation caused by different backgrounds and perspectives during retrieval, and effectively solves the problems caused by camera perspective and vehicle orientation transformation, so that the vehicle retrieval with strong interpretability, high efficiency and high retrieval precision is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image retrieval, in particular to a vehicle retrieval method and device based on multi-task learning, an electronic device and a storage medium. BACKGROUND

[0002] Vehicle retrieval belongs to the sub-problem of image retrieval, which refers to finding a vehicle with the same identity as a given vehicle image in a large-scale vehicle image library. With the acceleration of urbanization and the rapid growth of the number of vehicles, the increasingly severe traffic conditions and complex monitoring scenarios have brought great challenges to vehicle retrieval.

[0003] The research method related to vehicle retrieval in the related art is a vehicle retrieval algorithm based on vehicle meta information, which accurately mines the unique distinguishing appearance information of different vehicle images by using rich labeled information of the vehicle (such as vehicle body color, vehicle type, vehicle brand, space-time information, etc.), and extracts more robust vehicle features. However, such methods stay at the level of vehicle appearance attributes such as vehicle body color and vehicle type in the use of vehicle meta information, and are difficult to solve the challenge of small inter-class difference caused by the appearance similarity of vehicles with the same color, brand and style. SUMMARY

[0004] In view of the problem that the related method only stays at the level of vehicle appearance attributes or local area in the mining of vehicle image features and the complexity of the network model, the present application provides a vehicle retrieval method, device, electronic device and storage medium based on multi-task learning, to realize vehicle retrieval with strong explainability, high efficiency and high retrieval accuracy.

[0005] In an aspect of an embodiment of the present application, a vehicle retrieval method based on multi-task learning is provided, which comprises:

[0006] Obtaining a training data set, a verification data set and a test data set containing vehicle image label information, the data in each data set is saved in a dictionary file with an image sequence number as a query key and a tuple containing an image tensor, a vehicle ID, a vehicle orientation and a camera view angle as a value;

[0007] Inputting the data of the training data set into an initial vehicle retrieval model based on multi-task learning according to the image sequence number to train a vehicle retrieval model;

[0008] Inputting the data in the verification data set into the trained vehicle retrieval model to obtain a preliminary ranking result;

[0009] Re-ranking the preliminary ranking result by using a multi-task retrieval strategy to obtain a final ranking result.

[0010] In some embodiments, the file name of the dictionary file at least contains an image sequence number.

[0011] In some embodiments, the vehicle retrieval model comprises a global feature extraction model and a multi-task learning branch;

[0012] The global feature extraction model is configured to obtain an image feature vector wherein n represents the number of images, i is a positive integer, f i represents the i-th element of the image feature vector f;

[0013] The multi-task learning branch comprises a vehicle retrieval branch, a camera view retrieval branch, and a vehicle orientation retrieval branch,

[0014] The vehicle retrieval branch is configured to map the image feature vector to a corresponding vehicle feature vector and calculate a similarity wherein S v (V i ,V j ) represents the cosine similarity between vehicle feature vectors V i ,V j , i is a positive integer, and V i , V j represent the i-th and j-th elements of the vehicle feature vector V, respectively;

[0015] The camera view retrieval branch is configured to map the image feature vector to a camera view feature vector and calculate a similarity wherein C i , C j represent the i-th and j-th elements of the camera view feature vector C, respectively, S C (C i ,C j ) represents the cosine similarity between camera view feature vectors C i ,C j ;

[0016] The vehicle orientation retrieval branch is configured to map the image feature vector to a vehicle orientation feature vector and calculate a similarity wherein Oi, Oj represent the i-th and j-th elements of the vehicle orientation vector O, respectively, S O (O i , O j ) represents the cosine similarity between vehicle orientation feature vectors O i ,O j .

[0017] In some embodiments, each multi-task learning branch also includes a triplet loss function:

[0018] L triplet_V =max(μ-S) v (V a V p )+S v (V a V n ),0),

[0019] L triplet_C =max(μ-S) C (C a C p )+S C (C a C n ),0),

[0020] L triplet_O =max(μ-S) O (O a O p )+S O (O a O n ),0),

[0021] Among them, L triplet_V L triplet_ L triplet_ Let V represent the loss functions for the vehicle retrieval branch, camera view retrieval branch, and vehicle orientation retrieval branch, respectively. μ = 0.3 is the similarity constraint boundary. a represents the baseline sample, n represents the negative sample, p represents the positive sample, and V represents the vehicle orientation retrieval branch. a V n V p Let C represent the feature vectors of the baseline sample vehicles, the negative sample vehicles, and the positive sample vehicles, respectively. a C n C p Let O represent the feature vectors of the reference sample camera view, the negative sample camera view, and the positive sample camera view, respectively. a O n O p S represents the orientation vector of the reference sample vehicle, the orientation vector of the negative sample vehicle, and the orientation vector of the positive sample vehicle, respectively. v (V a V p ), S v (V a V n ), S C (C a C p ), SC (C a C n ), S O (O a O p ), S O (O a O n ) respectively represent V a and V p V a and V n C a and C p C a and C n O a and O p O a and O n Cosine similarity between them;

[0022] The overall loss function of the vehicle retrieval model is L = L triplet_V +αL triplet_C +βL triplet_O ;

[0023] Where α = 0.5 and β = 0.25 are predetermined hyperparameters, representing the loss function L, respectively. triplet_C L triplet_O The weighting percentage.

[0024] In some embodiments, the retrieval strategy includes:

[0025] Based on the vehicle feature vector The camera view feature vector The vehicle orientation feature vector Calculate the vehicle feature similarity matrix S respectively v Camera view similarity matrix S C Vehicle orientation similarity matrix S o The formula for calculating the similarity matrix is ​​as follows:

[0026]

[0027] Where f represents the image feature vector, i, j are positive integers, f i f j Let i and j represent the i-th and j-th elements of the image feature vector f, respectively, and S(i,j) represent the value of the element of the similarity matrix with coordinates (i,j).

[0028] The final similarity matrix is ​​obtained based on the feature similarity matrices.

[0029] wherein S represents the final similarity matrix, ω = 0.025, is a predetermined hyper-parameter, and respectively represent the camera view similarity matrix S C , the vehicle orientation similarity matrix S O , and the weight proportion of the camera view similarity matrix S

[0030] In some embodiments, the method further comprises the following steps:

[0031] inputting data in the test data set into the vehicle retrieval model to perform retrieval accuracy testing, obtaining a retrieval accuracy result, and determining whether to continue training the vehicle retrieval model according to the retrieval accuracy result.

[0032] In some embodiments, the vehicle image is input into the vehicle retrieval model to perform vehicle retrieval.

[0033] Another aspect of the embodiments of the present application also provides a vehicle retrieval device based on multi-task learning, the device comprising: an input module configured to obtain a training data set, a verification data set, and a test data set containing vehicle image annotation information, data in each data set being saved in a dictionary file with an image serial number as a query key and a tuple containing an image tensor, a vehicle ID, a vehicle orientation, and a camera view as a value;

[0034] a training module configured to input data in the training data set into an initial vehicle retrieval model based on multi-task learning according to the image serial number to obtain a vehicle retrieval model;

[0035] a retrieval module configured to input data in the verification data set into the trained vehicle retrieval model to obtain a preliminary ranking result, and to reorder the preliminary ranking result using a multi-task retrieval strategy to obtain a final ranking result.

[0036] Another aspect of the embodiments of the present application also provides a computer device comprising at least one processor, and a memory storing computer instructions executable on the processor, the instructions being executed by the processor to implement the steps of any of the above methods.

[0037] Another aspect of the embodiments of the present application also provides a computer readable storage medium storing a computer program executable by a processor to implement the steps of any of the above methods.

[0038] The application provides a vehicle retrieval method and device based on multi-task learning, an electronic device and a storage medium. The vehicle image feature extraction is guided by the multi-task learning mode, the camera perspective retrieval and vehicle orientation retrieval are designed as auxiliary tasks, the camera perspective information and vehicle orientation information in the image are mined, and the features of the auxiliary tasks are used to further eliminate the deviation caused by the background and perspective difference during retrieval in the inference stage, so that the problems caused by the camera perspective and vehicle orientation transformation are effectively solved, and the vehicle retrieval with strong explainability, high efficiency and high retrieval accuracy is realized. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other embodiments can be obtained without creative labor on the basis of these drawings.

[0040] Figure 1 The vehicle retrieval method based on multi-task learning in some embodiments of the present application is a flowchart.

[0041] Figure 2 The vehicle retrieval model based on multi-task learning in some embodiments of the present application is a flowchart.

[0042] Figure 3 The schematic diagram of the triple training data in some embodiments of the present application is a flowchart.

[0043] Figure 4 The model training flowchart of the vehicle retrieval model based on multi-task learning in some embodiments of the present application is a flowchart.

[0044] Figure 5 The schematic diagram of the vehicle retrieval device based on multi-task learning provided by the present application is a flowchart.

[0045] Figure 6 The schematic diagram of the computer device provided by the present application is a flowchart.

[0046] Figure 7 The schematic diagram of the computer readable storage medium provided by the present application is a flowchart. DETAILED DESCRIPTION

[0047] The embodiments of the present application are described below. However, it should be understood that the disclosed embodiments are only examples, and other embodiments can take various alternative forms.

[0048] In addition, it needs to be explained that all the terms "comprising", "containing" or any other variants thereof used in the embodiments of the present application are intended to cover non-exclusive inclusion, so that the process, method, article or device containing a series of elements not only includes those elements, but also can include elements not explicitly listed or inherent to these processes, methods, articles or devices.

[0049] Vehicle retrieval is a sub-problem of image retrieval, which refers to finding a vehicle with the same identity as a given vehicle image in a large-scale vehicle image library. With the acceleration of urbanization and the rapid growth of the number of vehicles, the increasingly severe traffic conditions and complex monitoring scenarios bring great challenges to vehicle retrieval. The vehicle retrieval algorithm for large-scale monitoring scenarios mainly faces two difficulties: first, the small inter-class difference, the vehicle appearance has similarity and diversity, and the vehicles with the same color, brand and style usually have very similar appearance; second, the large intra-class difference, the environment in the monitoring scene is complex and changeable, and the vehicle image will be affected by various objective factors, such as light change, view change, occlusion, video blur, etc., so there is a large difference between the same vehicle in different views and different backgrounds.

[0050] The research on vehicle retrieval is mainly divided into two categories: the first category is the vehicle retrieval algorithm based on local region, which focuses on the local region with strong distinguishability in the vehicle image, such as bumper, window sign, insurance sign, etc. However, most of such algorithms rely on complex network structure to extract local features, which greatly increases the model complexity. The second category is the vehicle retrieval algorithm based on vehicle meta information, which accurately mines the unique distinguishable appearance information of different vehicle images by using rich labeled information of the vehicle (such as vehicle body color, vehicle type, vehicle brand, space-time information, etc.), and extracts more robust vehicle features. However, the use of vehicle meta information in such methods stays at the level of vehicle appearance attributes such as vehicle body color and vehicle type, and it is difficult to solve the challenge of small inter-class difference caused by the appearance similarity of vehicles with the same color, brand and style.

[0051] In view of the problem that the related method only stays at the level of vehicle appearance attributes or local region in mining vehicle image features and the network model is complex, the present application provides a vehicle retrieval method based on multi-task learning to realize a vehicle retrieval technology with strong explainability, high efficiency and high retrieval precision.

[0052] One or more embodiments of the present application will be described below with reference to the accompanying drawings.

[0053] Figure 1 The flowchart of the vehicle retrieval method based on multi-task learning in some embodiments of the present application is shown in FIG. 1, which shows that the present application provides a vehicle retrieval method based on multi-task learning, including the following steps: Figure 1 ​

[0054] S1: Obtain a training data set, a verification data set and a test data set containing vehicle image annotation information, and save the data in each data set in a dictionary file with an image sequence number as a query key and a tuple containing an image tensor, a vehicle ID, a vehicle orientation and a camera perspective as a value.

[0055] Obtain a training data set T, a verification data set V and a test data set U containing vehicle image annotation information; specifically, some embodiments of the present application use a VeRi-776 data set (a large data set for vehicle re-identification) containing a total of 51035 vehicle images of 776 different identities, wherein the training data set T contains 37778 images, the verification data set V contains 560 query images and 3860 test images, and the test data set U contains 1118 query images and 7719 test images.

[0056] In combination with the actual vehicle image annotation information in the training data set T, write the data in the training data set T into a dictionary file with an image sequence number as a query key and a tuple containing an image tensor, a vehicle ID, a vehicle orientation and a camera perspective as a value. In some embodiments, the dictionary file is named according to a file naming rule R, which at least contains an image sequence number, and the image sequence number name is used as a query key and the tuple containing the image tensor, the vehicle ID, the vehicle orientation and the camera perspective is saved as a value. For example, a feasible video dictionary of a file naming rule R is image_id1 (id1 = 1, 2, …, 37778).

[0057] S2: Input the data of the training data set into an initial vehicle retrieval model based on multi-task learning according to the image sequence number to obtain a vehicle retrieval model.

[0058] S3: Input the data in the verification data set into the trained vehicle retrieval model to obtain a preliminary ranking result.

[0059] Figure 2 For some embodiments of the present application, the vehicle retrieval model based on multi-task learning; as shown in Figure 2 the vehicle retrieval model based on multi-task learning is constructed, and the parameter set of the vehicle retrieval model is set and adjusted, the training set data saved is imported into the vehicle retrieval model according to the image sequence number, and the vehicle retrieval model is iteratively trained and saved.

[0060] In some embodiments of the present application, the verification data set is read, the vehicle retrieval model is used for retrieval, and a preliminary ranking result is obtained.

[0061] In some embodiments of the present application, the present application further comprises: constructing a global feature extraction model, and obtaining an image feature vector using the global feature extraction model wherein n represents the number of images, i is a positive integer, and f i represents the i-th element of the image feature vector f.

[0062] Residual Network (ResNet) or Vision Transformer (ViT) can be used as a global feature extraction model, and an image feature vector is obtained using the global feature extraction model and an adaptive average pooling function wherein n represents the number of images, the feature dimension is 2048 or 768, corresponding to ResNet and ViT, respectively.

[0063] The present application also includes constructing a multi-task learning branch including a vehicle retrieval branch, a camera view retrieval branch, and a vehicle orientation retrieval branch based on a linear fully connected layer, and mapping the image feature vector to the corresponding vehicle feature vector camera view feature vector vehicle orientation feature vector Vi, Ci, and Oi are the i-th elements of the vehicle feature vector V, the camera view feature vector C, and the vehicle orientation feature vector O, respectively. It should be further noted that the elements Vi, Ci, and Oi themselves can also be vectors, depending on the dimensions of V, C, and O. The dimension of the vehicle feature vector V is 1024 or 768, corresponding to ResNet and ViT, respectively, the dimension of the camera view feature vector C is 96, and the dimension of the vehicle orientation feature vector O is 48. The similarity is calculated according to the feature vectors:

[0064]

[0065]

[0066]

[0067] wherein S v (V i ,V j ) represents the cosine similarity between the vehicle feature vectors V i ,V j , S C (C i ,C j ) represents the cosine similarity between the camera view feature vectors C i ,C j , and S O (O i , Oj ) represents the cosine similarity between the vehicle orientation vector O i j

[0068] The application further comprises constructing a triplet loss function for each multi-task learning branch and assigning different weights to obtain a final loss function and model training:

[0069] L triplet_V = max(μ - S v (V a , V p ) + S v (V a , V n ), 0),

[0070] L triplet_C = max(μ - S C (C a , C p ) + S C (C a , C n ), 0),

[0071] L triplet_O = max(μ - S O (O a , O p ) + S O (O a , O n ), 0),

[0072] wherein L triplet_V , L triplet_ , L triplet_ represent the loss functions of the vehicle retrieval branch, the camera view retrieval branch and the vehicle orientation retrieval branch respectively, μ = 0.3 is the similarity constraint boundary, a represents the reference sample, n represents the negative sample, and p represents the positive sample, V a , V n , V p represent the vehicle feature vectors of the reference sample, the negative sample and the positive sample respectively, C a , C n , C p represent the camera view feature vectors of the reference sample, the negative sample and the positive sample respectively, O a , O n , O p represent the vehicle orientation vectors of the reference sample, the negative sample and the positive sample respectively, S v (V a , V​​p ), S v (V a , V n ), S C (C a , C p ), S C (C a , C n ), S O (O a , O p ), SO(O a , O n ) respectively represent the cosine similarity between V a and V p , V a and V n , C a and C p , C a and C n , O a and O p , O a and O n .

[0073] L = L triplet_V + aL triplet_C + bL triplet_O ;

[0074] wherein L represents the loss function of the model ensemble, L triplet_V , L triplet_C , L triplet_O represent the loss functions of different branches respectively, a, b are predetermined hyperparameters, taking values of 0.5, 0.25 respectively, representing the weight proportion of the loss functions L triplet_C , L triplet_O , and m is the similarity constraint boundary, taking a value of 0.3, and the subscripts a, p, n are triplets. The triplet loss (also known as triple loss function) is composed of a triplet <a, p, n>, which requires three pictures as input, wherein a: anchor (anchor) represents the reference sample, p: positive (positive) represents the positive sample of the same category but different from the anchor, and n: negative (negative) represents the negative sample of different categories from the reference sample. Figure 3 is a schematic diagram of triplet training data in some embodiments of the present application; as shown in Figure 3 , using each generated triplet, the model can create a corresponding positive pair <a, p> and negative pair <a, n>.

[0075] For a set of triplets (Anchor, Positive, Negative) (Anchor and Positive are different samples of the same class, and Anchor and Negative are different samples of different classes), Triplet loss attempts to learn a feature space in which the anchor sample (Anchor) of the same class and the positive sample (Positive) are closer in distance, and the Anchor and the negative sample (Negative) of different classes are farther apart. Neural network models based on Triplet loss can well distinguish details, and Triplet loss can usually learn better subtle features during training. More specifically, Triplet loss can set a certain threshold according to the needs of model training. The network structure with Triplet loss generally sets a threshold margin when training. Designers can control the distance between positive and negative samples by changing the value of the threshold margin.

[0076] The vehicle retrieval method based on multi-task learning in the application realizes a general multi-task learning framework suitable for convolutional neural network and self-attention mechanism encoder, solves the problem that most of the existing algorithms are based on convolutional neural network and do not have universality. The vehicle retrieval based on multi-task learning uses multi-task learning to represent features, designs camera perspective retrieval and vehicle orientation retrieval as auxiliary tasks, mines training information related to auxiliary tasks, extracts differentiated features of vehicle images, and thus effectively improves vehicle retrieval accuracy without introducing too many network parameters (only about 1.1% network parameters are introduced compared to the backbone network).

[0077] S4: Reordering the preliminary sorting result by using a multi-task retrieval strategy to obtain a final sorting result.

[0078] Figure 4 The model training flowchart of the vehicle retrieval model based on multi-task learning in some embodiments of the application; as shown in Figure 4 , a multi-task retrieval strategy is constructed to reorder the obtained preliminary sorting result to obtain a final sorting result.

[0079] In some embodiments of the application, the application further comprises: constructing a feature similarity matrix according to the vehicle feature vector camera perspective feature vector vehicle orientation feature vector vehicle feature similarity matrix S v , camera perspective similarity matrix S C , and vehicle orientation similarity matrix S O are calculated respectively. The calculation formula of the similarity matrix is as follows:

[0080]

[0081] wherein f represents the image feature vector, f i , f j represent the i-th and j-th element of the image feature vector f respectively, S(i,j represents the value of the coordinate (i,j) in the similarity matrix.

[0082] According to each feature similarity matrix, a final similarity matrix is obtained, and the preliminary sorting result is reordered:

[0083]

[0084] wherein S represents the final similarity matrix, ω, is a predetermined hyperparameter, and the values are 0.025 and 0.035 respectively, representing the weight proportion of the camera view similarity matrix S C and the vehicle orientation similarity matrix S O .

[0085] In the multi-task retrieval strategy in the present application, the cosine similarity of the vehicle features extracted by the vehicle retrieval branch in the reasoning stage is calculated, the initial sorting result is obtained based on the measurement of the vehicle similarity, the similarity of the camera view and the vehicle orientation is measured based on the camera view features and the vehicle orientation features extracted by the auxiliary task, and the secondary sorting is performed based on this, the deviation caused by the background and the different views during the retrieval is further eliminated by using the features of the auxiliary task, thereby effectively solving the intra-class difference problem caused by the transformation of the camera view and the vehicle orientation, and improving the vehicle retrieval accuracy.

[0086] In some embodiments of the present application, the present application further comprises the following steps:

[0087] The step of repeating the training is performed to test the retrieval accuracy, the retrieval accuracy result is obtained, and it is determined whether the model performance continues to improve. If there is no performance gain, the model training is ended, otherwise the training process is returned to be executed.

[0088] The step of testing the training effect is to read the test data set in turn, load the vehicle retrieval model file, test the retrieval accuracy, and obtain the retrieval accuracy result.

[0089] The application guides the extraction of vehicle image features in a multi-task learning manner, designs camera view retrieval and vehicle orientation retrieval as auxiliary tasks, mines camera view information and vehicle orientation information in the image, and further eliminates the deviation caused by different backgrounds and views during retrieval by using the features of the auxiliary tasks in the inference stage, thereby effectively solving the problems caused by camera view and vehicle orientation transformation, and providing a vehicle retrieval method based on multi-task learning to realize strong explainability, high efficiency and high retrieval accuracy of vehicle retrieval.

[0090] To achieve the same purpose, a second aspect of the embodiments of the application provides a vehicle retrieval device based on multi-task learning. Figure 5 Figure 1 is a schematic diagram of an embodiment of a vehicle retrieval device based on multi-task learning in some embodiments of the application. As shown in Figure 1, the vehicle retrieval device based on multi-task learning in some embodiments of the application includes the following modules: Figure 5 The vehicle retrieval device based on multi-task learning in some embodiments of the application includes the following modules:

[0091] The input module is configured to obtain a training data set, a verification data set and a test data set containing vehicle image annotation information, and the data in each data set is saved in a dictionary file with an image serial number as a query key and a tuple containing an image tensor, a vehicle ID, a vehicle orientation and a camera view as a value.

[0092] The training module is configured to input the data of the training data set into an initial vehicle retrieval model based on multi-task learning according to the image serial number to train a vehicle retrieval model.

[0093] The retrieval module is configured to input the data in the verification data set into the trained vehicle retrieval model to obtain a preliminary ranking result, and reorder the preliminary ranking result by using a multi-task retrieval strategy to obtain a final ranking result.

[0094] To achieve the above purpose, a third aspect of the embodiments of the application provides a computer device, Figure 6 Figure 2 is a schematic diagram of an embodiment of a computer device in some embodiments of the application. As shown in Figure 2, the computer device in some embodiments of the application includes the following modules: Figure 6 The embodiment of the computer device in some embodiments of the application includes the following modules: at least one processor 021; and a memory 022, the memory 022 storing computer instructions 023 executable on the processor 021, and the computer instructions 023 are executed by the processor 021 to implement the steps of the method as described above.

[0095] The application also provides a computer readable storage medium. Figure 7 Figure 3 is a schematic diagram of an embodiment of a computer readable storage medium in some embodiments of the application. As shown in Figure 3, the computer readable storage medium in some embodiments of the application includes the following modules: Figure 7The computer readable storage medium 031 stores a computer program 032 which, when executed by the processor, performs the above method.

[0096] Finally, it should be noted that those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program to instruct related hardware to complete, and the program of the method of setting system parameters can be stored in a computer readable storage medium. When the program is executed, it can include the processes of the above-mentioned embodiments of the method. Among them, the storage medium of the program can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM) and the like. The above-mentioned computer program embodiments can achieve the same or similar effects as the corresponding any method embodiments.

[0097] In addition, the method disclosed in the embodiments of the present application can also be implemented as a computer program executed by a processor, which can be stored in a computer readable storage medium. When the computer program is executed by the processor, the above-mentioned functions defined in the method disclosed in the embodiments of the present application are executed.

[0098] In addition, the above-mentioned method steps and system units can also be implemented by using a controller and a computer readable storage medium for storing a computer program for enabling the controller to implement the above-mentioned steps or unit functions.

[0099] Those skilled in the art will also appreciate that the various illustrative logical blocks, modules, circuits, and method steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Skilled persons can implement the functions in various ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments disclosed herein.

[0100] In one or more exemplary designs, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another. A storage media can be any available media that can be accessed by a general purpose or special purpose computer. By way of example, and not limitation, such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code means in the form of instructions or data structures and that can be accessed by a general-purpose or special-purpose computer, or a general-purpose or special-purpose processor. Also, any connection is properly termed a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or other wire-based, fiber-based, or wireless technologies, then the coaxial cable, fiber optic cable, twisted pair, DSL, or other wire-based, fiber-based, or wireless technologies are included in the definition of medium. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0101] The foregoing is considered as illustrative of the principles of the application. However, further modifications and changes can occur to those skilled in the art without departing from the scope of the application as defined in the following claims. The singular number implementation of the method claims in the disclosure herein should not be construed as limiting of only a single recited step to be executed, unless explicitly so limited according to context. Further, elements of the application disclosed herein can be understood as being plural, unless explicitly limited to a single according to context.

[0102] It is to be understood that the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. It is to be understood that the term "and / or" as used herein encompasses all possible combinations of the items listed.

[0103] The above-described application embodiment serial numbers are merely for description, not representing advantages or disadvantages of the embodiments.

[0104] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed by programs to complete the related hardware, and the programs can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0105] Those skilled in the art shall understand that the above discussion of any embodiment is only exemplary, and is not intended to imply that the scope (including claims) of the embodiments disclosed in the present application is limited to these examples; the technical features in the above embodiments or different embodiments can also be combined, and there are many other changes of different aspects of the embodiments of the present application as above. In order to be brief, they are not provided in details. Therefore, any omission, modification, equivalent replacement, improvement, etc. made in the spirit and principles of the embodiments of the present application shall be included in the protection scope of the embodiments of the present application.

Claims

1. A vehicle retrieval method based on multi-task learning, characterized in that, The method comprises the following steps: obtain a training data set, a verification data set and a test data set containing vehicle image annotation information, the data in each data set is saved in a dictionary file with an image sequence number as a query key and a tuple containing an image tensor, a vehicle ID, a vehicle orientation and a camera perspective as a value; input the data of the training data set into an initial vehicle retrieval model based on multi-task learning according to the image sequence number to train a vehicle retrieval model; input the data in the verification data set into the trained vehicle retrieval model to obtain a preliminary ranking result; re-rank the preliminary ranking result using a multi-task retrieval strategy to obtain a final ranking result; wherein the file name of the dictionary file at least contains an image sequence number; wherein the vehicle retrieval model comprises a global feature extraction model and a multi-task learning branch; The global feature extraction model is used to obtain an image feature vector wherein n represents the number of images, i is a positive integer, f i represents the i-th element of the image feature vector f. the multi-task learning branch comprises a vehicle retrieval branch, a camera perspective retrieval branch and a vehicle orientation retrieval branch, The vehicle retrieval branch is configured to map the image feature vector into a corresponding vehicle feature vector and compute a similarity score S according to the vehicle feature vector V v (V i ,V j ) denotes the cosine similarity between vehicle feature vectors V i ,V j , j is a positive integer, and V i , V j denote the i-th and j-th elements of vehicle feature vector V, respectively. The camera view retrieval branch is configured to map the image feature vector into a camera view feature vector and compute a similarity according to the camera view feature vector C where C i , C j denote the i-th and j-th element of the camera view feature vector C, respectively, S C (C i , C j ) denotes the cosine similarity between the camera view feature vectors C i , C j . The vehicle orientation retrieval branch is configured to map the image feature vector to a vehicle orientation feature vector and compute a similarity score where O i , O j denote the i-th and j-th elements of the vehicle orientation vector O, respectively, and S O (O i , O j ) denotes the cosine similarity between the vehicle orientation feature vectors O i , O j . wherein the multi-task retrieval strategy comprises: According to the vehicle feature vector The camera view feature vector The vehicle orientation feature vector Respectively calculate the vehicle feature similarity matrix S v , the camera view similarity matrix S C , the vehicle orientation similarity matrix S O The calculation formula of the similarity matrix is as follows: wherein f denotes an image feature vector, i, j are positive integers, f i , f j denote the i-th and j-th element of the image feature vector f, respectively, and S(i, j) denotes the value of the element of the similarity matrix with coordinates (i, j). According to the feature similarity matrices, a final similarity matrix is obtained where S denotes the final similarity matrix, ω = 0.025, is a predetermined hyper-parameter, and respectively denote the weight proportions of the camera view similarity matrix S C , the vehicle orientation similarity matrix S O . 2.The vehicle retrieval method based on multi-task learning according to claim 1, wherein, each multi-task learning branch further comprises a triplet loss function: L triplet_V = max(μ - S v (V a , V p ) + S v (V a , V n ), 0), L triplet_C = max(μ - S C (C a , C p ) + S C (C a , C n ), 0), L triplet_O = max(μ - S O (O a , O p ) + S O (O a , O n ), 0), wherein, L triplet_V , L triplet_C , L triplet_O represent the loss function of the vehicle retrieval branch, the camera view retrieval branch and the vehicle orientation retrieval branch respectively, μ = 0.3 is the similarity constraint boundary, a represents the reference sample, n represents the negative sample, p represents the positive sample, V a , V n , V p represent the reference sample vehicle feature vector, the negative sample vehicle feature vector and the positive sample feature vector respectively, C a , C n , C p represent the reference sample camera view feature vector, the negative sample camera view feature vector and the positive sample camera view feature vector respectively, O a , O n , O p represent the reference sample vehicle orientation vector, the negative sample vehicle orientation vector and the positive sample vehicle orientation vector respectively, S v (V a ,V p ), S v (V a ,V n ), S C (C a ,C p ), S C (C a ,C n ), S O (O a ,O p ), S O (O a ,O n ) respectively represent the cosine similarity between V a and V p , V a and V n , C a and C p , C a and C n , O a and O p , O a and O n . The overall loss function L of the vehicle retrieval model = L triplet_V + aL triplet_C + bL triplet_O ; wherein a=0.5, b=0.25 are predetermined hyperparameters, respectively representing the weight proportion of the loss function L triplet_C , L triplet_O . 3.The vehicle retrieval method based on multi-task learning according to claim 1, characterized in that, further comprising the following steps: input the data in the test data set into the vehicle retrieval model for retrieval accuracy testing to obtain a retrieval accuracy result, and determine whether to continue training the vehicle retrieval model according to the retrieval accuracy result.

4. The vehicle retrieval method based on multi-task learning according to any one of claims 1-3, characterized in that, input a vehicle image into the vehicle retrieval model for vehicle retrieval. 5.A vehicle retrieval apparatus based on multi-task learning, characterized by comprise: an input module for obtaining a training data set, a verification data set and a test data set containing vehicle image annotation information, the data in each data set is saved in a dictionary file with an image sequence number as a query key and a tuple containing an image tensor, a vehicle ID, a vehicle orientation and a camera perspective as a value; a training module for inputting the data of the training data set into an initial vehicle retrieval model based on multi-task learning according to the image sequence number to train a vehicle retrieval model; a retrieval module for inputting the data in the verification data set into the trained vehicle retrieval model to obtain a preliminary ranking result, and re-ranking the preliminary ranking result using a multi-task retrieval strategy to obtain a final ranking result; wherein the file name of the dictionary file at least contains an image sequence number; wherein the vehicle retrieval model comprises a global feature extraction model and a multi-task learning branch; The global feature extraction model is used to obtain an image feature vector wherein n represents the number of images, i is a positive integer, f i denotes the i-th element of the image feature vector f. the multi-task learning branch comprises a vehicle retrieval branch, a camera perspective retrieval branch and a vehicle orientation retrieval branch, The vehicle retrieval branch is used to process the image feature vector. Mapped to the corresponding vehicle feature vector And calculate the similarity based on the vehicle feature vector V. Among them, S v (V i V j ) represents the vehicle feature vector V i V j The cosine similarity between them, where j is a positive integer, V i V j These represent the i-th and j-th elements of the vehicle feature vector V, respectively; The camera view retrieval branch is configured to map the image feature vector into a camera view feature vector and compute a similarity according to the camera view feature vector C where C i , C j denote the i-th and j-th element of the camera view feature vector C, respectively, S C (C i , C j ) denotes the cosine similarity between the camera view feature vectors C i , C j . The vehicle orientation retrieval branch is configured to map the image feature vector to a vehicle orientation feature vector and compute a similarity score where O i , O j are the i-th and j-th elements of the vehicle orientation vector O, respectively, and S O (O i , O j ) is the cosine similarity between the vehicle orientation feature vectors O i , O j . wherein the multi-task retrieval strategy comprises: According to the vehicle feature vector The camera view feature vector The vehicle orientation feature vector Respectively calculate the vehicle feature similarity matrix S v , the camera view similarity matrix S C , the vehicle orientation similarity matrix S O The calculation formula of the similarity matrix is as follows: wherein f denotes an image feature vector, i, j are positive integers, f i , f j denote the i-th and j-th elements of the image feature vector f, respectively, and S(i, j) denotes the value of the element of the similarity matrix with coordinates (i, j). According to the feature similarity matrices, a final similarity matrix is obtained wherein S denotes the final similarity matrix, ω = 0.025, is a predetermined hyper-parameter, and respectively denote the weight proportions of the camera view similarity matrix S C , the vehicle orientation similarity matrix S O .

6. A computer device, comprising: comprise: at least one processor; and a memory, the memory stores computer instructions executable on the processor, the instructions are executed by the processor to implement the steps of the method of any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program is executed by the processor to implement the steps of the method of any one of claims 1-4.

Images