Vehicle search method, device and electronic equipment
By extracting the local feature block sequence of vehicle pictures and using bitmask mapping relationship tables, the accuracy and efficiency problems in vehicle data retrieval are solved, and more efficient vehicle recognition and retrieval is achieved.
Patent Information
- Application Number
- CN202411223571.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-03
AI Technical Summary
In the existing vehicle data retrieval technology, the accuracy of license plate recognition is insufficient, especially in the search for vehicles of the same style and color, and the search process takes a long time and cannot meet the needs of large-scale data retrieval.
By extracting the local feature block sequence of vehicle pictures, a local feature validity identification is constructed, and the mapping relationship table between the pre-configured bitmask and the local feature information set is used to quickly judge the similarity between vehicle pictures and historical pictures, and reduce the search time consuming.
It improves the accuracy and efficiency of vehicle search results, can extract local features of the vehicle from multiple directions, fully express the differences in vehicle details, and reduces the time-consuming of the search process.
Smart Images

Figure CN118760779B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of intelligent transportation technology and computer technology, and in particular to a vehicle retrieval method, device and electronic equipment. Background Art
[0002] In the existing vehicle data retrieval field, license plate recognition technology is often used for rapid vehicle retrieval and identification. As the identity of the vehicle, accurate recognition of the license plate information is crucial for vehicle retrieval. However, due to a variety of factors, such as the limitation of camera resolution, unfavorable lighting conditions, and blur or damage of the license plate itself, accurate recognition of license plate information faces challenges.
[0003] In the related art, the global features of the vehicle, such as the license plate number and license plate color, are extracted, and the abstract features of the vehicle are extracted using a deep learning model, and then the similarity between the image to be retrieved and the vehicle features in the database is calculated to achieve vehicle retrieval. This global feature cannot fully represent the detailed differences of the vehicle, resulting in insufficient accuracy in the retrieval of vehicles of the same style and color, resulting in recognition errors, thereby reducing the accuracy of the retrieval results. At the same time, the retrieval process is time-consuming and cannot meet the needs of large-scale data retrieval. Summary of the invention
[0004] The embodiments of the present application provide a vehicle retrieval method, device and electronic device. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.
[0005] In a first aspect, an embodiment of the present application provides a vehicle retrieval method, which is applied to a server, and the method includes:
[0006] When the license plate information of the vehicle image to be retrieved is not extracted, the features of the local area are extracted from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences to obtain a target local feature block sequence;
[0007] Constructing a local feature validity flag of the vehicle image to be retrieved, the local feature validity flag is used to indicate whether each target local feature block in the target local feature block sequence contains valid information;
[0008] When the local feature validity flag indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, the similarity between the vehicle image to be retrieved and the first image is generated through a mapping relationship table of a pre-configured bit mask and a local feature information set according to the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image, the first image is each historical image, and the mapping relationship table is used to provide a similarity determination flag for determining whether to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image and the local feature block index that needs to be compared;
[0009] The first image whose similarity is greater than a preset similarity threshold is sent to the client as a search result.
[0010] In a second aspect, an embodiment of the present application provides a vehicle search device, the device comprising:
[0011] A feature extraction module is used to extract features of local areas from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences when the license plate information of the vehicle image to be retrieved is not extracted, so as to obtain a target local feature block sequence;
[0012] A local feature validity identification construction module is used to construct a local feature validity identification of the vehicle image to be retrieved, and the local feature validity identification is used to indicate whether each target local feature block in the target local feature block sequence contains valid information;
[0013] A similarity calculation module, used for generating the similarity between the vehicle image to be retrieved and the first image through a mapping relationship table of a pre-configured bit mask and a local feature information set when the local feature validity identifier indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, according to the local feature validity identifier of the vehicle image to be retrieved and a preset local feature validity identifier of the first image, wherein the first image is each historical image, and the mapping relationship table is used for providing a similarity determination identifier for determining whether to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image and an index of the local feature block to be compared;
[0014] The retrieval result sending module is used to send the first image whose similarity is greater than a preset similarity threshold as the retrieval result to the client.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, which may include: a processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the above-mentioned method steps.
[0016] The technical solutions provided by some embodiments of the present application may have the following beneficial effects:
[0017] In some embodiments of the present application, on the one hand, multiple information sequence representations are preset to sequentially extract information from multiple positions of the vehicle, so that local features of the vehicle can be extracted from multiple directions. The multiple local features can fully represent the detailed differences of the vehicle, thereby improving the accuracy of the retrieval results; on the other hand, the present application has a mapping relationship table between a pre-configured bit mask and a local feature information set, and the mapping relationship table is used to provide a similarity determination identifier for determining whether to calculate the similarity of the local feature vector between the vehicle image to be retrieved and the first image, and an index of the local feature block that needs to be compared. Therefore, the retrieval strategy of the vehicle image to be retrieved and the basic data for similarity calculation can be quickly determined by table lookup, and the table lookup process is less time-consuming, thereby reducing the retrieval time.
[0018] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0020] Figure 1 It is a schematic diagram of the relationship between a vehicle travel behavior and a behavior scenario provided in an embodiment of the present application;
[0021] Figure 2 This is a schematic diagram of image comparison during a search process provided by an embodiment of the present application;
[0022] Figure 3 It is a coding schematic diagram of a local feature block sequence provided in an embodiment of the present application;
[0023] Figure 4 It is a schematic diagram of a coding comparison process of a local feature block sequence in a retrieval process provided by an embodiment of the present application;
[0024] Figure 5 This is another schematic diagram of image comparison during the retrieval process provided by an embodiment of the present application;
[0025] Figure 6 This is another schematic diagram of image comparison during the retrieval process provided by an embodiment of the present application;
[0026] Figure 7 It is a schematic block diagram of a vehicle search process provided by an embodiment of the present application;
[0027] Figure 8 is a structural schematic diagram of a vehicle retrieval device provided in an embodiment of the present application;
[0028] Fig. 9It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0029] The following description and the accompanying drawings sufficiently illustrate specific embodiments of the present application to enable those skilled in the art to practice them.
[0030] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.
[0031] When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0032] In the description of the present application, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances. In addition, in the description of the present application, unless otherwise specified, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0033] The following will be combined with the attached Figure 1 -Attached Figure 7 , a vehicle search method provided in an embodiment of the present application is introduced in detail. The method can be implemented by a computer program and can be run on a vehicle search device based on the von Neumann system. The computer program can be integrated into an application or run as an independent tool application.
[0034] See also Figure 1 , which is a flow chart of a vehicle search method provided in an embodiment of the present application, and is applied to a server. Figure 1 As shown, the method of the embodiment of the present application may include the following steps:
[0035] S101, when the license plate information of the vehicle image to be retrieved is not extracted, extracting features of local areas from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences to obtain a target local feature block sequence;
[0036] Among them, the vehicle image to be retrieved is a vehicle image used for retrieval or matching in the database, and the image is the query object of the retrieval process. The license plate information refers to the registration license plate information of the vehicle, including the license plate number. The local area refers to a specific, relatively small area in the vehicle image, such as the headlights, taillights, license plate area, etc. of the vehicle. Feature extraction is the process of identifying and quantifying visual information from the image, with the purpose of converting visual data into a numerical representation that can be used for machine learning models or retrieval algorithms. Preset multiple information orders are a predefined priority for information extraction. Preset multiple information orders are, for example, headlights, car ornaments, car frame glass, annual inspection labels, taillights, car logos, and rear windshields. The system will extract local features of different information from the image in sequence according to the priority. The target local feature block sequence is a series of local features extracted according to the preset multiple information orders, and these features are organized into a block sequence.
[0037] It should be noted that presetting multiple information orders can ensure that the order of extracted features is consistent with the order of features of each historical image, thereby ensuring the uniformity of subsequent calculations.
[0038] In some embodiments, the server receives a vehicle image to be retrieved from a client or other smart device. When the server fails to extract the license plate information of the vehicle image to be retrieved, the server extracts the features of the local area from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences to obtain a target local feature block sequence. The extraction of the features of the local area can be performed based on a neural network.
[0039] In other embodiments, the server receives a vehicle image to be retrieved from a client or other smart device. When the server extracts the license plate information of the vehicle image to be retrieved, it obtains the first hash value corresponding to the license plate information of the vehicle image to be retrieved; determines whether the first hash value is equal to the second hash value corresponding to the first image; counts the number of first images with equal hash values; when the number of images is greater than or equal to a preset number threshold, sends the first images with equal hash values to the client as retrieval results; or, when the number of images is less than the preset number threshold, continues to perform the step of extracting features of local areas from the vehicle image to be retrieved in sequence according to a preset multiple information order. The preset number threshold is topk=10. In the present application, when the license plate information can be extracted, the first image that matches the vehicle image to be retrieved can be quickly determined by comparing the hash values. This retrieval method can significantly accelerate the retrieval of vehicle images in a checkpoint environment.
[0040] For example, the license plate information of the vehicle picture to be retrieved (Picture 0) extracted by the server is "Jing A10000". At this time, the first hash value corresponding to the license plate information is 577874364. The second hash values corresponding to each of the first pictures (Pictures 1, 2, and 3) are shown in Table 1. By comparison, it can be seen that the second hash value corresponding to Picture 3 is equal to
[0041] Table 1
[0042]
[0043] the first hash value corresponding to the license plate information of the vehicle picture to be retrieved. At this time, Picture 3 can be used as a result. Pictures 0, 1, 2, and 3 are as shown in Figure 2 When the final comparison result shows that there are 10+ results in the result set, the result set can be sent to the client as the retrieval result.
[0044] S102. Construct a local feature validity identifier for the vehicle picture to be retrieved. This local feature validity identifier is used to represent whether each target local feature block in the target local feature block sequence contains valid information;
[0045] In some embodiments of the present application, the specific process of constructing the local feature validity identifier for the vehicle picture to be retrieved includes: determining whether each target local feature block in the target local feature block sequence is a null value; taking the local feature blocks that are not null values as valid local feature blocks and marking the valid local feature blocks as 1; taking the local feature blocks that are null values as invalid local feature blocks and marking the invalid local feature blocks as 0; when the number of valid local feature blocks is greater than or equal to the preset quantity threshold, taking the marked value sequence as the local feature validity identifier for the vehicle picture to be retrieved.
[0046] For example, there are a total of eight target local feature blocks. The third target local feature block from right to left is a car pendant. If the features of the car pendant are successfully extracted, it is considered that the target local feature block corresponding to the car pendant is a valid local feature block. For example, there are a total of eight local feature blocks, and the features of the first local feature block (front headlight), the second local feature block (car pendant), and the fifth local feature block (annual inspection label) are successfully extracted. Then the number of valid local feature blocks is equal to 3.
[0047] For example Figure 3 As shown, the target local feature block sequence includes 8 target local feature blocks. The rightmost one is the first extracted target local feature block, and the leftmost one is the last extracted target local feature block. The target local feature blocks marked as 1 are valid local feature blocks, and the target local feature blocks marked as 0 are invalid local feature blocks. The target local feature block index positions from right to left are 0, 1, 2, 3, 4, 5, 6, and 7 in sequence. It can be seen that the vehicle picture to be retrieved has 4 valid local features, and the index positions are 0, 3, 5, and 6 respectively.
[0048] S103, when the local feature validity flag indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, based on the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image, a similarity between the vehicle image to be retrieved and the first image is generated through a mapping relationship table of a pre-configured bit mask and a local feature information set, the first image is each historical image, and the mapping relationship table is used to provide a similarity determination flag for determining whether to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image and an index of the local feature block to be compared;
[0049] In some embodiments of the present application, based on the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image, the specific process of generating the similarity between the vehicle image to be retrieved and the first image through a mapping relationship table of a pre-configured bit mask and a local feature information set includes: performing a bit operation on the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image to obtain a bit operation result; performing a decimal conversion on the bit operation result to obtain a bit mask of the vehicle image to be retrieved and the first image; obtaining a target local feature information set corresponding to the bit mask from the mapping relationship table of a pre-configured bit mask and a local feature information set; and generating the similarity between the vehicle image to be retrieved and the first image based on the target local feature information set.
[0050] For example, the local feature validity flag of the vehicle image to be retrieved is 01101001, and the preset local feature validity flag of the first image is 11111001. At this time, the bit operation of the two is 11111001, and the decimal conversion is 105, which is the bit mask of the vehicle image to be retrieved and the first image.
[0051] The target local feature information set includes a similarity determination identifier for determining whether to calculate the similarity of the local feature vector between the vehicle image to be retrieved and the first image, and an index of the local feature block to be compared.
[0052] In some embodiments of the present application, the specific process of generating the similarity between the vehicle picture to be retrieved and the first picture based on the target local feature information set includes: when the similarity judgment identifier indicates that the local feature vector similarity is calculated between the vehicle picture to be retrieved and the first picture, obtaining the local feature blocks corresponding to a group of identifiers corresponding to each index position in the local feature block index from the local feature validity identifier of the vehicle picture to be retrieved and the preset local feature validity identifier of the first picture; calculating the feature vector distance between the local feature blocks corresponding to a group of identifiers corresponding to each index position; and summing and averaging the calculated feature vector distances to obtain the similarity between the vehicle picture to be retrieved and the first picture.
[0053] For example, the bit mask of the vehicle image to be retrieved and the first image is 105, and the query in Table 2 can be
[0054] Table 2
[0055]
[0056] The similarity determination mark corresponding to 105 and the local feature block indexes to be compared are True and {0,3,5,6} respectively. At this time, it can be determined that 0,3,5,6 correspond to the headlights, annual inspection mark, taillights and car logo.
[0057] Therefore, the final similarity is: Similarity = [(headlight vector distance) + (annual inspection mark vector distance) + (taillight vector distance) + (car logo vector distance)] / 4.
[0058] The local feature validity identifier of the present application is stored in bits, which can effectively reduce the content usage rate; at the same time, bit operations and table lookups are used during comparison to quickly determine whether to perform local feature retrieval and select the local blocks that need to be compared.
[0059] In an embodiment of the present application, a mapping relationship table between a preconfigured bit mask and a local feature information set can be generated according to the following process, including: creating a first local feature validity identifier and a second local feature validity identifier with the same number of bits, arranging and combining different binary values in the first local feature validity identifier with different binary values in the second local feature validity identifier to obtain multiple pairs of binary values, for each pair of binary values, calculating its bit mask, counting the valid local feature index bits and their number, and all valid index bits and their number in each pair of binary values, the value of the valid local feature index bit is 1, and all valid index bits refer to the same index bit with a value of 1; storing the mapping relationship between the bit mask of each pair of binary values, all valid index bits and their number, marking the all valid index bits whose number in each mapping relationship is greater than or equal to a preset threshold with a similarity determination flag that requires calculation of the local feature vector similarity, and marking the all valid index bits whose number in each mapping relationship is less than the preset threshold with a similarity determination flag that does not require calculation of the local feature vector similarity, to obtain a mapping relationship table between the preconfigured bit mask and the local feature information set. Among them, the valid index bits can also be understood as the local feature blocks at the corresponding positions of the two data are valid. For example, A successfully extracts the first local feature block (headlights) and the second local feature block (car ornaments), and B successfully extracts the second local feature block (car ornaments) and the fifth local feature block (annual inspection label). Then, the index bit for the local features to be valid is {1} (corresponding to the second local feature being valid).
[0060] For example Figure 4As shown, multiple pairs of binary values can be 01101001 and 01110001, 01101001 and 00110001, and 01101001 and 11111001. At this time, the valid local feature index bits of 01101001 and 01110001 are 0, 4, 5, and 6, the number of valid local feature index bits is 4, and the local features are all valid index bits 0, 5, and 6. At this time, the local feature vector similarity needs to be calculated, so the similarity determination flag of 01101001 and 01110001 is marked as "true".
[0061] S104: Send the first image whose similarity is greater than a preset similarity threshold as a search result to the client.
[0062] In an embodiment of the present application, after the local feature retrieval through steps S101 to S103, the similarity between the first image (i.e., each historical image) and the vehicle image to be retrieved can be obtained, and the first image whose similarity is greater than a preset similarity threshold can be sent to the client as a retrieval result.
[0063] In an embodiment of the present application, for example, the vehicle image to be retrieved is a relatively blurry image of a vehicle that escaped from a car accident at night, and the only valid local feature blocks that can be extracted are the front lights, which does not meet the conditions for calculating the similarity of local feature retrieval. In this case, global feature retrieval is required.
[0064] In some embodiments, the specific process of performing global feature retrieval includes: when the local feature validity flag indicates that the valid local feature block of the vehicle image to be retrieved meets the preset conditions, obtaining the first global feature of the vehicle image to be retrieved; using the inverted compression indexing technology and the preset number of categories, according to the first global feature of the vehicle image to be retrieved and the preset second global feature of the first image, calculating the global similarity between the vehicle image to be retrieved and the first image. The global features mainly describe the shape, texture and other attributes of the vehicle.
[0065] Specifically, only two valid local features (frame glass, headlights) are extracted from the vehicle image to be retrieved. This does not meet the sufficient conditions for local feature retrieval (local feature retrieval requires at least three valid local feature blocks to be extracted). In this case, global feature comparison can be performed directly. Global features are stored and retrieved using inverted compressed indexes. When comparing, only the nearest nprobe categories need to be compared for retrieval, which can greatly speed up the retrieval speed in this scenario and save RAM usage. For example Figure 5 As shown, the similarities of the three retrieved historical images are 95%, 68%, and 31% respectively.
[0066] Among them, the inverted compressed index technology and the nprobe parameter are two methods used to improve the efficiency of large-scale data retrieval. The inverted index is an indexing method that maps data items (such as words, feature vectors, etc.) to a list of documents (or data points) containing these items. In feature vector retrieval, the inverted index can quickly locate data points with similar features. Compression technology is used to reduce the size of the index and improve storage efficiency. This is usually achieved through methods such as quantization, encoding, or clustering to map the original feature vector to a set of representative codewords or center points. nprobe is a parameter used to control the depth of the approximate nearest neighbor (ANN) search. It defines the number of candidate data points that need to be considered during the retrieval process. By setting nprobe, a trade-off can be made between retrieval speed and accuracy.
[0067] In the embodiment of the present application, when enough valid local feature blocks can be extracted from the vehicle image to be retrieved (i.e., three valid local feature blocks are extracted) and there are not enough local feature blocks in the historical images in the library to match it (i.e., the number of valid index bits does not reach the threshold), the global feature is selected for comparison. By introducing the attention mechanism, the attention weight of the global feature is set to a lower value, that is, the similarity calculated by this part of the global feature is weakened.
[0068] It should be noted that when the number of valid index bits does not reach the threshold, the corresponding similarity determination flag in Table 2 is false, that is, there is no need to calculate the similarity of the local feature vector between the vehicle image to be retrieved and the first image.
[0069] In other embodiments, the specific process of performing global feature retrieval includes: when the similarity determination flag indicates that there is no need to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image, obtaining the first global feature of the vehicle image to be retrieved; using the inverted compression indexing technology and the preset number of categories, according to the first global feature of the vehicle image to be retrieved and the preset second global feature of the first image, calculating the initial similarity between the vehicle image to be retrieved and the first image; multiplying the initial similarity with the preset attention weight to obtain the global similarity between the vehicle image to be retrieved and the first image. Among them, using this method to calculate the similarity makes the retrieval results more focused and more accurate.
[0070] For example Figure 6As shown in the figure, the vehicle image to be retrieved (Query) has extracted 4 valid local feature blocks (car logo, headlight, annual inspection mark, and car frame glass), and the historical image 2 has also extracted 4 valid local feature blocks (car logo, car frame glass, taillight, and rear windshield). The total number of feature blocks with valid local features in both image 0 and image 2 is 2 (car logo, car frame glass), which does not reach the threshold of 3 for local feature comparison. Therefore, the global feature similarity of the introduced attention is calculated. It can be seen that although the global feature similarity between image 0 and image 2 is still relatively high (67%), after the introduction of the attention mechanism, the similarity is greatly reduced (67%*0.3), that is, at this time, the similarity between image 2 and image 1 is not paid much attention.
[0071] Furthermore, the confidence of the search results compared using global features is not high (for example, two cars of the same brand may be similar in shape and texture). In this case, the pre-stored structured attributes such as vehicle body color, vehicle type, and vehicle brand can be used to give a certain weight when calculating the similarity. In some embodiments, each pre-stored vehicle structured attribute and its weight are obtained; based on the vehicle image to be retrieved and the first image, the score of each vehicle structured attribute is determined; the score of each vehicle structured attribute is multiplied and summed with the weight of each vehicle structured attribute to obtain a target value; the global similarity is multiplied with the preset percentage and then summed with the target value to obtain the final similarity between the vehicle image to be retrieved and the first image; the first image whose final similarity is greater than the preset similarity threshold is sent to the client as a search result.
[0072] For example, the final similarity of the retrieval based on global feature comparison = global feature comparison similarity * 0.7 + body color score * 0.1 + vehicle type score * 0.1 + vehicle brand score * 0.1.
[0073] It should be noted that the scoring mechanism for structured attributes is: if the query vehicle image has the same structured attribute as the base image, it is scored as 1, otherwise it is scored as 0.
[0074] For example Figure 7 As shown, Figure 7This is a schematic flowchart of a vehicle search process provided by the present application. After receiving the image of the vehicle to be searched, the server can extract information from the image of the vehicle to be searched. If the extracted information contains license plate information, the license plate information can be used for pre-search. When the result obtained by the license plate pre-matching search meets the preset number, the result is directly returned. When the result obtained by the license plate pre-matching search does not meet the preset number, a feature search is performed. When performing the feature search, it is determined whether the local feature search conditions are met. If so, a local feature search is performed and the results are output. If the local feature search conditions are not met, a global feature search is performed, and the global search results are output. The results are re-sorted and the sorted results are output.
[0075] It should be noted that there are many technical implementation methods for feature extraction, such as CNN, RNN, Transformer, etc. There are also many clustering algorithms, such as IVFFlat, IVFSQ, IVFPQ, HNSW. The feature extraction algorithm and clustering algorithm can be replaced as needed and are not limited here.
[0076] In some embodiments of the present application, on the one hand, multiple information sequence representations are preset to sequentially extract information from multiple positions of the vehicle, so that local features of the vehicle can be extracted from multiple directions. The multiple local features can fully represent the detailed differences of the vehicle, thereby improving the accuracy of the retrieval results; on the other hand, the present application has a mapping relationship table between a pre-configured bit mask and a local feature information set, and the mapping relationship table is used to provide a similarity determination identifier for determining whether to calculate the similarity of the local feature vector between the vehicle image to be retrieved and the first image, and an index of the local feature block that needs to be compared. Therefore, the retrieval strategy of the vehicle image to be retrieved and the basic data for similarity calculation can be quickly determined by table lookup, and the table lookup process is less time-consuming, thereby reducing the retrieval time.
[0077] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.
[0078] See also Figure 8 , which shows a schematic diagram of the structure of a vehicle search device provided by an exemplary embodiment of the present application. The vehicle search device can be implemented as all or part of an electronic device through software, hardware or a combination of both. The device 1 includes a feature extraction module 10, a local feature validity identification construction module 20, a similarity calculation module 30, and a search result sending module 40.
[0079] The feature extraction module 10 is used to extract the features of the local area from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences when the license plate information of the vehicle image to be retrieved is not extracted, so as to obtain a target local feature block sequence;
[0080] A local feature validity identification construction module 20 is used to construct a local feature validity identification of the vehicle image to be retrieved, and the local feature validity identification is used to indicate whether each target local feature block in the target local feature block sequence contains valid information;
[0081] A similarity calculation module 30 is used to generate the similarity between the vehicle image to be retrieved and the first image through a mapping relationship table of a pre-configured bit mask and a local feature information set when the local feature validity identifier indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, according to the local feature validity identifier of the vehicle image to be retrieved and the preset local feature validity identifier of the first image, wherein the first image is each historical image, and the mapping relationship table is used to provide a similarity determination identifier for determining whether to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image and the local feature block index to be compared;
[0082] The search result sending module 40 is used to send the first picture whose similarity is greater than a preset similarity threshold as the search result to the client.
[0083] It should be noted that the vehicle retrieval device provided in the above embodiment only uses the division of the above functional modules as an example when executing the vehicle retrieval method. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the vehicle retrieval device provided in the above embodiment and the vehicle retrieval method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.
[0084] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0085] In some embodiments of the present application, on the one hand, multiple information sequence representations are preset to sequentially extract information from multiple positions of the vehicle, so that local features of the vehicle can be extracted from multiple directions. The multiple local features can fully represent the detailed differences of the vehicle, thereby improving the accuracy of the retrieval results; on the other hand, the present application has a mapping relationship table between a pre-configured bit mask and a local feature information set, and the mapping relationship table is used to provide a similarity determination identifier for determining whether to calculate the similarity of the local feature vector between the vehicle image to be retrieved and the first image, and an index of the local feature block that needs to be compared. Therefore, the retrieval strategy of the vehicle image to be retrieved and the basic data for similarity calculation can be quickly determined by table lookup, and the table lookup process is less time-consuming, thereby reducing the retrieval time.
[0086] The present application also provides a computer-readable medium having program instructions stored thereon, and when the program instructions are executed by a processor, the vehicle retrieval method provided by the above-mentioned various method embodiments is implemented.
[0087] The present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the vehicle retrieval method of each of the above method embodiments.
[0088] See also Fig. 9 , is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 1000 may include: at least one processor 1001 , at least one network interface 1004 , a user interface 1003 , a memory 1005 , and at least one communication bus 1002 .
[0089] The communication bus 1002 is used to realize the connection and communication between these components.
[0090] The user interface 1003 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.
[0091] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).
[0092] Among them, the processor 1001 may include one or more processing cores. The processor 1001 uses various interfaces and lines to connect various parts of the entire electronic device 1000, and executes various functions and processes data of the electronic device 1000 by running or executing instructions, programs, code sets or instruction sets stored in the memory 1005, and calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 1001 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1001, and it can be implemented separately through a chip.
[0093] Among them, the memory 1005 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 1005 includes a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 1005 may also be optionally at least one storage system located away from the aforementioned processor 1001. As Fig. 9 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a vehicle retrieval application.
[0094] exist Fig. 9 In the electronic device 1000 shown, the user interface 1003 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 1001 can be used to call the vehicle retrieval application stored in the memory 1005 and specifically perform the following operations:
[0095] When the license plate information of the vehicle image to be retrieved is not extracted, the features of the local area are extracted from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences to obtain a target local feature block sequence;
[0096] Constructing a local feature validity flag of the vehicle image to be retrieved, the local feature validity flag is used to indicate whether each target local feature block in the target local feature block sequence contains valid information;
[0097] When the local feature validity flag indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, the similarity between the vehicle image to be retrieved and the first image is generated through a mapping relationship table of a pre-configured bit mask and a local feature information set according to the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image, the first image is each historical image, and the mapping relationship table is used to provide a similarity determination flag for determining whether to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image and the local feature block index that needs to be compared;
[0098] The first image whose similarity is greater than a preset similarity threshold is sent to the client as a search result.
[0099] In one embodiment, when executing the construction of the local feature validity identification of the vehicle image to be retrieved, the processor 1001 specifically performs the following operations:
[0100] Determine whether each target local feature block in the target local feature block sequence is a null value;
[0101] The local feature block that is not a null value is regarded as a valid local feature block, and the valid local feature block is marked as 1; the local feature block that is a null value is regarded as an invalid local feature block, and the invalid local feature block is marked as 0;
[0102] When the number of valid local feature blocks is greater than or equal to a preset number threshold, the marked value sequence is used as a local feature validity identifier of the vehicle image to be retrieved.
[0103] In one embodiment, when the processor 1001 generates the similarity between the vehicle image to be retrieved and the first image through a mapping relationship table of a pre-configured bit mask and a local feature information set according to the local feature validity identifier of the vehicle image to be retrieved and the preset local feature validity identifier of the first image, the following operations are specifically performed:
[0104] Performing a bitwise operation on the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image to obtain a bitwise operation result;
[0105] Convert the bit operation result into decimal to obtain the bit mask of the vehicle image to be retrieved and the first image;
[0106] Obtaining a target local feature information set corresponding to the bit mask from a pre-configured mapping relationship table between the bit mask and the local feature information set;
[0107] According to the target local feature information set, the similarity between the vehicle image to be retrieved and the first image is generated.
[0108] In one embodiment, when the processor 1001 generates the similarity between the to-be-retrieved vehicle image and the first image according to the target local feature information set, the following operations are specifically performed:
[0109] When the similarity determination flag indicates that the local feature vector similarity is calculated between the to-be-retrieved vehicle image and the first image, a local feature block corresponding to a group of flags corresponding to each index bit in the local feature block index is obtained from the local feature validity flag of the to-be-retrieved vehicle image and the preset local feature validity flag of the first image;
[0110] Calculate the feature vector distance between the local feature blocks corresponding to a set of identifiers corresponding to each index bit;
[0111] The calculated feature vector distances are summed and averaged to obtain the similarity between the vehicle image to be retrieved and the first image.
[0112] In one embodiment, the processor 1001 further performs the following operations:
[0113] When the local feature validity flag indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, obtaining a first global feature of the vehicle image to be retrieved;
[0114] By adopting the inverted compression indexing technology and the preset number of categories, the global similarity between the vehicle image to be retrieved and the first image is calculated according to the first global feature of the vehicle image to be retrieved and the preset second global feature of the first image.
[0115] In one embodiment, the processor 1001 further performs the following operations:
[0116] When the similarity determination flag indicates that there is no need to calculate the similarity of the local feature vector between the to-be-retrieved vehicle image and the first image, obtaining the first global feature of the to-be-retrieved vehicle image;
[0117] Using an inverted compressed indexing technique and a preset number of categories, the initial similarity between the vehicle image to be retrieved and the first image is calculated according to the first global feature of the vehicle image to be retrieved and the preset second global feature of the first image;
[0118] The initial similarity is multiplied by the preset attention weight to obtain the global similarity between the vehicle image to be retrieved and the first image.
[0119] In one embodiment, after obtaining the global similarity between the to-be-retrieved vehicle image and the first image, the processor 1001 further performs the following operations:
[0120] Obtain the pre-stored structured attributes and weights of each vehicle;
[0121] Determine a score of each vehicle structured attribute based on the vehicle image to be retrieved and the first image;
[0122] The score of each vehicle structured attribute is multiplied by the weight of each vehicle structured attribute and the sum is calculated to obtain the target value.
[0123] The global similarity is multiplied by the preset percentage and then summed with the target value to obtain the final similarity between the vehicle image to be retrieved and the first image;
[0124] The first image whose final similarity is greater than the preset similarity threshold is sent to the client as a retrieval result.
[0125] In one embodiment, the processor 1001 further performs the following operations:
[0126] When the license plate information of the vehicle image to be retrieved is extracted, a first hash value corresponding to the license plate information of the vehicle image to be retrieved is obtained;
[0127] Determining whether the first hash value is equal to a second hash value corresponding to the first image;
[0128] Count the number of pictures with the same hash value as the first picture;
[0129] When the number of pictures is greater than or equal to a preset number threshold, the first picture with the same hash value is sent to the client as a search result; or,
[0130] When the number of pictures is less than a preset number threshold, the step of extracting features of local areas from the vehicle pictures to be retrieved in sequence according to a preset plurality of information sequences is continued.
[0131] In some embodiments of the present application, on the one hand, multiple information sequence representations are preset to sequentially extract information from multiple positions of the vehicle, so that local features of the vehicle can be extracted from multiple directions. The multiple local features can fully represent the detailed differences of the vehicle, thereby improving the accuracy of the retrieval results; on the other hand, the present application has a mapping relationship table between a pre-configured bit mask and a local feature information set, and the mapping relationship table is used to provide a similarity determination identifier for determining whether to calculate the similarity of the local feature vector between the vehicle image to be retrieved and the first image, and an index of the local feature block that needs to be compared. Therefore, the retrieval strategy of the vehicle image to be retrieved and the basic data for similarity calculation can be quickly determined by table lookup, and the table lookup process is less time-consuming, thereby reducing the retrieval time.
[0132] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the vehicle search program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the above-mentioned embodiments of the methods. The storage medium of the vehicle search program can be a disk, an optical disk, a read-only storage memory, or a random access memory, etc.
[0133] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A vehicle search method, characterized in that: Applied to the server, the method includes: When the license plate information of the vehicle image to be retrieved is not extracted, the features of the local area are extracted from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences to obtain a target local feature block sequence; Constructing a local feature validity flag of the vehicle image to be retrieved, the local feature validity flag is used to indicate whether each target local feature block in the target local feature block sequence contains valid information; wherein, Constructing the local feature validity identifier of the vehicle image to be retrieved, including: Determine whether each target local feature block in the target local feature block sequence is a null value; The local feature block that is not a null value is regarded as a valid local feature block, and the valid local feature block is marked as 1; the local feature block that is a null value is regarded as an invalid local feature block, and the invalid local feature block is marked as 0; When the number of the valid local feature blocks is greater than or equal to a preset number threshold, the marked value sequence is used as a local feature validity identifier of the vehicle image to be retrieved; When the local feature validity flag indicates that the valid local feature block of the vehicle image to be retrieved meets the preset condition, the similarity between the vehicle image to be retrieved and the first image is generated according to the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image, through a mapping relationship table of a pre-configured bit mask and a local feature information set, the first image is each historical image, and the mapping relationship table is used to provide a similarity determination flag for determining whether to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image and the local feature block index that needs to be compared; wherein, The generating the similarity between the vehicle image to be retrieved and the first image through a mapping relationship table of a pre-configured bit mask and a local feature information set according to the local feature validity identifier of the vehicle image to be retrieved and the preset local feature validity identifier of the first image includes: Performing a bitwise operation on the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image to obtain a bitwise operation result; Convert the bit operation result into decimal to obtain a bit mask of the vehicle image to be retrieved and the first image; Obtaining a target local feature information set corresponding to the bit mask from a pre-configured mapping relationship table between the bit mask and the local feature information set; Generating a similarity between the to-be-retrieved vehicle image and the first image according to the target local feature information set; The first image whose similarity is greater than a preset similarity threshold is sent to the client as a search result.
2. The method according to claim 1, characterized in that The target local feature information set includes a similarity determination mark used to determine whether to calculate the local feature vector similarity between the to-be-retrieved vehicle image and the first image and an index of a local feature block to be compared; The generating, according to the target local feature information set, the similarity between the to-be-retrieved vehicle image and the first image comprises: When the similarity determination flag indicates that the local feature vector similarity is calculated between the vehicle image to be retrieved and the first image, a local feature block corresponding to a group of flags corresponding to each index bit in the local feature block index is obtained from the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image; Calculate the feature vector distance between the local feature blocks corresponding to a set of identifiers corresponding to each index bit; The calculated feature vector distances are summed and averaged to obtain the similarity between the vehicle image to be retrieved and the first image.
3. The method according to claim 1, characterized in that The method further comprises: When the local feature validity flag indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, obtaining a first global feature of the vehicle image to be retrieved; By adopting the inverted compression indexing technology and the preset number of categories, the global similarity between the vehicle image to be retrieved and the first image is calculated according to the first global feature of the vehicle image to be retrieved and the preset second global feature of the first image.
4. The method according to claim 1, characterized in that: The method further comprises: When the similarity determination flag indicates that there is no need to calculate the similarity of the local feature vector between the to-be-searched vehicle image and the first image, obtaining a first global feature of the to-be-searched vehicle image; Using an inverted compressed indexing technique and a preset number of categories, calculating an initial similarity between the vehicle image to be retrieved and the first image according to a first global feature of the vehicle image to be retrieved and a preset second global feature of the first image; The initial similarity is multiplied by a preset attention weight to obtain a global similarity between the vehicle image to be retrieved and the first image.
5. The method according to claim 3 or 4, characterized in that: After obtaining the global similarity between the to-be-searched vehicle image and the first image, the method further includes: Obtain the pre-stored structured attributes and weights of each vehicle; Determining a score of each vehicle structured attribute based on the to-be-retrieved vehicle image and the first image; The score of each vehicle structured attribute is multiplied by the weight of each vehicle structured attribute and the sum is calculated to obtain the target value. The final similarity between the to-be-retrieved vehicle image and the first image is obtained by multiplying the global similarity by a preset percentage and then summing the result with the target value; The first image whose final similarity is greater than a preset similarity threshold is sent to the client as a retrieval result.
6. The method according to claim 1, characterized in that The method comprises: When the license plate information of the vehicle image to be retrieved is extracted, a first hash value corresponding to the license plate information of the vehicle image to be retrieved is obtained; Determining whether the first hash value is equal to a second hash value corresponding to the first picture; Count the number of pictures with the same hash value as the first picture; When the number of pictures is greater than or equal to a preset number threshold, the first picture with the same hash value is sent to the client as a search result; or, When the number of the images is less than a preset number threshold, the step of extracting features of local areas from the vehicle images to be retrieved in sequence according to a preset plurality of information sequences is continued.
7. A vehicle search device, characterized in that: The device comprises: A feature extraction module is used to extract features of a local area from the vehicle image to be retrieved in sequence according to a preset plurality of information sequences when the license plate information of the vehicle image to be retrieved is not extracted, so as to obtain a target local feature block sequence; A local feature validity identification construction module is used to construct a local feature validity identification of the vehicle image to be retrieved, and the local feature validity identification is used to indicate whether each target local feature block in the target local feature block sequence contains valid information; wherein, Constructing the local feature validity identifier of the vehicle image to be retrieved, including: Determine whether each target local feature block in the target local feature block sequence is a null value; The local feature block that is not a null value is regarded as a valid local feature block, and the valid local feature block is marked as 1; the local feature block that is a null value is regarded as an invalid local feature block, and the invalid local feature block is marked as 0; When the number of the valid local feature blocks is greater than or equal to a preset number threshold, the marked value sequence is used as a local feature validity identifier of the vehicle image to be retrieved; A similarity calculation module is used to generate the similarity between the vehicle image to be retrieved and the first image through a mapping relationship table of a pre-configured bit mask and a local feature information set when the local feature validity identifier indicates that the valid local feature block of the vehicle image to be retrieved meets a preset condition, according to the local feature validity identifier of the vehicle image to be retrieved and the preset local feature validity identifier of the first image, wherein the first image is each historical image, and the mapping relationship table is used to provide a similarity determination identifier for determining whether to calculate the local feature vector similarity between the vehicle image to be retrieved and the first image and the local feature block index that needs to be compared; wherein, The generating the similarity between the vehicle image to be retrieved and the first image through a mapping relationship table of a pre-configured bit mask and a local feature information set according to the local feature validity identifier of the vehicle image to be retrieved and the preset local feature validity identifier of the first image includes: Performing a bitwise operation on the local feature validity flag of the vehicle image to be retrieved and the preset local feature validity flag of the first image to obtain a bitwise operation result; Convert the bit operation result into decimal to obtain a bit mask of the vehicle image to be retrieved and the first image; Obtaining a target local feature information set corresponding to the bit mask from a pre-configured mapping relationship table between the bit mask and the local feature information set; Generating a similarity between the to-be-retrieved vehicle image and the first image according to the target local feature information set; The retrieval result sending module is used to send the first image whose similarity is greater than a preset similarity threshold as a retrieval result to the client.
8. An electronic device, characterized in that: include: A processor and a memory; wherein the memory stores a computer program, and the computer program is suitable for being loaded by the processor and executing the method according to any one of claims 1 to 6.
