Method, device, equipment and storage medium for retrieval of difficult-example images for autonomous driving

Abstract

The present invention discloses a method, device, equipment and storage medium for retrieving difficult-case images for autonomous driving. The method comprises: obtaining an autonomous driving image data set, and constructing an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model comprises a feature extraction module and a hash quantization module; training the initial autonomous driving image retrieval model based on the autonomous driving image data set to obtain an autonomous driving image retrieval model; based on a driving image database, inputting autonomous driving difficult-case sample images into the autonomous driving image retrieval model, and outputting retrieval results corresponding to autonomous driving difficult-case sample images in the driving image database. Since the present invention inputs autonomous driving difficult-case sample images into an autonomous driving image retrieval model comprising a feature extraction module and a hash quantization module, and outputs retrieval results corresponding to autonomous driving difficult-case sample images in the driving image database, the efficiency and accuracy of autonomous driving difficult-case image retrieval are improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for retrieving difficult-example images for autonomous driving. Background Art

[0002] In the field of autonomous driving, there is a large amount of image data captured from on-board cameras, including complex traffic conditions, extreme weather conditions, road construction, traffic accidents and other difficult examples. These scenarios are crucial to improving the robustness and safety of autonomous driving systems. How to mine these difficult examples, quickly and accurately find images that match the query conditions from the massive autonomous driving image data through the provided difficult sample images, so as to build an autonomous driving difficult sample database, and make targeted improvements and optimizations to the autonomous driving algorithm, thereby improving the overall performance of the system, has always been the direction of continuous improvement in the industry. However, the efficiency and accuracy of autonomous driving difficult image retrieval in the existing technology are low.

[0003] Therefore, there is an urgent need for a difficult image retrieval method for autonomous driving that can effectively improve the efficiency and accuracy of difficult image retrieval for autonomous driving. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, equipment and storage medium for retrieval of difficult-example images for autonomous driving, aiming to solve the technical problems of low efficiency and accuracy of retrieval of difficult-example images for autonomous driving in the prior art.

[0005] To achieve the above object, the present invention provides a method for retrieval of difficult-example images for autonomous driving, the method comprising the following steps:

[0006] Acquire an autonomous driving image dataset and construct an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module;

[0007] Performing model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model;

[0008] Based on a driving image database, the autonomous driving difficult example sample images are input into the autonomous driving image retrieval model, and the retrieval results corresponding to the autonomous driving difficult example sample images in the driving image database are output.

[0009] Optionally, the step of performing model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain the autonomous driving image retrieval model includes:

[0010] Dividing the autonomous driving image dataset into a training set, a validation set, and a test set according to a preset ratio;

[0011] Inputting the training set in the autonomous driving image data set into the initial autonomous driving image retrieval model, and performing model training on the initial autonomous driving image retrieval model using a preset loss function to obtain a training result;

[0012] Based on the training results, the initial autonomous driving image retrieval model is updated through a back propagation algorithm to obtain an autonomous driving image retrieval model.

[0013] Optionally, after the step of updating the initial autonomous driving image retrieval model by a back propagation algorithm based on the training result to obtain the autonomous driving image retrieval model, the step further includes:

[0014] Using the validation set and the test set in the autonomous driving image dataset to perform model evaluation on the autonomous driving image retrieval model to obtain a model evaluation result;

[0015] Based on the model evaluation result, the autonomous driving image retrieval model is optimized and adjusted to obtain a new autonomous driving image retrieval model.

[0016] Optionally, the preset loss function includes a similarity information loss function and a hash quantization loss function;

[0017] The similarity information loss function is:

[0018] ;

[0019] In the formula, , u and v Represents a hash code, Represents a hash code u and v The similarity between Represents a hash code u and v The similarity of the labels corresponding to the two images. Indicates that two images are similar. Indicates that the two images are not similar;

[0020] The hash quantization loss function is:

[0021] ;

[0022] Where N is the length of the hash code, i is the number of the hash code length, is the original hash code, is the binary hash code;

[0023] The preset loss function is:

[0024] ;

[0025] In the formula, is the weight parameter of the hash quantization loss, is similar information loss, Quantify the loss for hashing.

[0026] Optionally, the step of inputting the autonomous driving difficult sample image into the autonomous driving image retrieval model based on the driving image database, and outputting the retrieval result corresponding to the autonomous driving difficult sample image in the driving image database includes:

[0027] Using the feature extraction module of the autonomous driving image retrieval model, respectively extract features from the images in the driving image database and the autonomous driving difficult sample images to obtain dataset image features and sample image features;

[0028] Using the hash quantization module of the autonomous driving image retrieval model, the image features of the data set and the sample image features are converted into a data set image hash code and a sample image hash code respectively;

[0029] Determining hash code similarity between the sample image hash code and the dataset image hash code;

[0030] The retrieval result corresponding to the autonomous driving difficult sample image in the driving image database is output according to the hash code similarity.

[0031] Optionally, the step of determining the hash code similarity between the sample image hash code and the dataset image hash code comprises:

[0032] Calculating the Hamming distance between the sample image hash code and the dataset image hash code;

[0033] Using the Hamming distance as the hash code similarity between the sample image hash code and the dataset image hash code;

[0034] The calculation formula of the Hamming distance is:

[0035] ;

[0036] In the formula, N is the length of the hash code, i is the number of the hash code length, and Represents the difficult sample images of autonomous driving qand images from the Driving Image Database p In the i The hash value on bits.

[0037] Optionally, after the step of inputting the autonomous driving difficult sample image into the autonomous driving image retrieval model based on the driving image database and outputting the retrieval result corresponding to the autonomous driving difficult sample image in the driving image database, the step further includes:

[0038] Arrange the search results in descending order according to the Hamming distance to obtain an arrangement result;

[0039] Based on the arrangement results, a preset number of autonomous driving difficult example images are selected to construct an autonomous driving difficult example training sample library.

[0040] In addition, to achieve the above-mentioned purpose, the present invention also proposes a difficult-example image retrieval device for autonomous driving, the device comprising:

[0041] A model building module, used to obtain an autonomous driving image dataset and build an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module;

[0042] A model training module, used to perform model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model;

[0043] The result output module is used to input the difficult sample images of autonomous driving into the autonomous driving image retrieval model based on the driving image database, and output the retrieval results corresponding to the difficult sample images of autonomous driving in the driving image database.

[0044] In addition, to achieve the above-mentioned purpose, the present invention also proposes an autonomous driving difficult example image retrieval device, which includes: a memory, a processor, and an autonomous driving difficult example image retrieval program stored on the memory and executable on the processor, wherein the autonomous driving difficult example image retrieval program is configured to implement the steps of the autonomous driving difficult example image retrieval method as described above.

[0045] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a difficult image retrieval program for autonomous driving is stored. When the difficult image retrieval program for autonomous driving is executed by a processor, the steps of the difficult image retrieval method for autonomous driving as described above are implemented.

[0046] The present invention discloses obtaining an autonomous driving image dataset and constructing an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module; performing model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model; based on a driving image database, inputting autonomous driving difficult sample images into the autonomous driving image retrieval model, and outputting retrieval results corresponding to the autonomous driving difficult sample images in the driving image database. Since the present invention inputs autonomous driving difficult sample images into an autonomous driving image retrieval model including a feature extraction module and a hash quantization module, and outputs retrieval results corresponding to autonomous driving difficult sample images in the driving image database. Compared with the prior art, the present invention effectively improves the efficiency and accuracy of autonomous driving difficult image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a flow chart of a first embodiment of a difficult-to-retrieve image retrieval method for autonomous driving according to the present invention;

[0048] Figure 2 Schematic diagram of the model structure of the autonomous driving image retrieval model in the autonomous driving difficult example image retrieval method of the present invention;

[0049] Figure 3 It is a schematic diagram of the image retrieval process in the difficult-example image retrieval method for autonomous driving of the present invention;

[0050] Figure 4 This is a flow chart of a second embodiment of a difficult-to-retrieve image method for autonomous driving according to the present invention;

[0051] Figure 5 It is a flowchart of a third embodiment of a difficult-example image retrieval method for autonomous driving according to the present invention;

[0052] Figure 6 This is a structural block diagram of a first embodiment of a difficult-example image retrieval device for autonomous driving according to the present invention;

[0053] Figure 7 It is a structural diagram of an automatic driving difficult-example image retrieval device in a hardware operating environment involved in an embodiment of the present invention.

[0054] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0055] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0056] The embodiment of the present invention provides a method for retrieving difficult-example images for autonomous driving, referring to Figure 1 , Figure 1 Schematic diagram of the process of the first embodiment of the difficult-example image retrieval method for autonomous driving of the present invention.

[0057] In this embodiment, the method for retrieving difficult-example images for autonomous driving includes steps S10 to S30:

[0058] Step S10: Acquire an autonomous driving image dataset and construct an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module.

[0059] It should be noted that the execution subject of this embodiment can be a computer server device with data processing, network communication and program running functions used in the scenario of mining difficult examples for autonomous driving, such as a server, a tablet computer, a smart phone, an image retrieval device for difficult examples for autonomous driving, etc., or an electronic device capable of realizing the above functions, an image retrieval device for difficult examples for autonomous driving, etc. The following takes a vehicle positioning device as an example to illustrate this embodiment and the following embodiments.

[0060] It should be understood that the autonomous driving image dataset can be a collection of image datasets used for the development and evaluation of autonomous driving technology. The images in the autonomous driving image dataset are usually from cameras on autonomous driving vehicles. These cameras may be located at different positions of the vehicle to capture images from different angles and fields of view. The images may cover a variety of road types and traffic environments.

[0061] It is understandable that the autonomous driving image retrieval model is an important component of autonomous driving technology. It can quickly and accurately retrieve images that match the query conditions from a large amount of image data, providing strong support for the decision-making and planning of the autonomous driving system. The initial autonomous driving image retrieval model can be an autonomous driving image retrieval model that has not been trained.

[0062] It should be noted that the initial autonomous driving image retrieval model in this embodiment includes a feature extraction module and a hash quantization module. The feature extraction module is used to extract image features. The feature extraction module may include a deep convolutional neural network and a fully connected layer. The feature extraction module may adopt AlexNet, ResNet, etc. The input image is passed through the convolutional neural network to generate a feature map; the hash quantization module is used to transform the high-dimensional feature vector (i.e., image feature) obtained by the feature extraction module into a low-dimensional hash code (i.e., pseudo-hash code) of a certain code length.

[0063] Step S20: Performing model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model.

[0064] It should be noted that the autonomous driving image dataset can be input into the initial autonomous driving image retrieval model, the initial autonomous driving image retrieval model can be trained using a preset loss function, and then the model can be updated through the back propagation algorithm to obtain the autonomous driving image retrieval model.

[0065] It should be explained that the preset loss function in this embodiment may include a similar information loss function and a hash quantization loss function. The similar information loss function and the hash quantization loss function are used to update the parameters of the model during the model training process.

[0066] For example, refer to Figure 2 , Figure 2 The schematic diagram of the model structure of the autonomous driving image retrieval model in the autonomous driving difficult case image retrieval method of the present invention is shown in the figure. In the figure, the image features are extracted by the feature extraction module, the feature extraction module may include a deep convolutional neural network and a fully connected layer, the feature extraction module may adopt AlexNet, ResNet, etc., and the input image generates a feature map after passing through the convolutional neural network; the hash quantization module in the figure transforms the high-dimensional feature vector (i.e., image feature) obtained by the feature extraction module into a low-dimensional hash code (i.e., pseudo-hash code) of a certain code length; then the pseudo-hash code is predicted by the classification layer, and the similarity information loss is calculated based on the predicted label using the similarity information loss function, and the new hash code is obtained by binarizing the pseudo-hash code, and then the hash quantization loss is calculated using the hash quantization loss function, and finally the model parameters of the initial autonomous driving image retrieval model are optimized based on the similarity information loss and the hash quantization loss.

[0067] Step S30: Based on the driving image database, the difficult sample images of autonomous driving are input into the autonomous driving image retrieval model, and the retrieval results corresponding to the difficult sample images of autonomous driving in the driving image database are output.

[0068] It should be noted that the driving image database may be a car center image database. The search result may be an image in the driving image database that matches the difficult sample image for autonomous driving.

[0069] In a specific implementation, the feature extraction module of the autonomous driving image retrieval model can be used to perform feature extraction on the images of the driving image database and the autonomous driving difficult sample images, respectively, to obtain data set image features and sample image features; the hash quantization module of the autonomous driving image retrieval model can be used to convert the data set image features and sample image features into data set image hash codes and sample image hash codes, respectively; the hash code similarity between the sample image hash code and the data set image hash code is determined; and the retrieval result corresponding to the autonomous driving difficult sample image in the driving image database is output according to the hash code similarity.

[0070] It should be noted that the step of determining the hash code similarity between the sample image hash code and the data set image hash code includes: calculating the Hamming distance between the sample image hash code and the data set image hash code; and using the Hamming distance as the hash code similarity between the sample image hash code and the data set image hash code.

[0071] The calculation formula of the Hamming distance is:

[0072] ;

[0073] In the formula, N is the length of the hash code, i is the number of the hash code length, and Represents the difficult sample images of autonomous driving q and images from the Driving Image Database p In the i The hash value on bits.

[0074] For example, refer to Figure 3 , Figure 3 The figure is a flow chart of image retrieval in the method for retrieval of difficult examples of autonomous driving in the present invention. In the figure, the sample image features and the dataset image features are obtained by extracting features of the sample image (i.e., the difficult example sample image of autonomous driving) and the dataset image (i.e., the image of the driving image database), and then the sample image hash code and the dataset image hash code are obtained by the hash function, and finally, the Hamming distance is calculated according to the sample image hash code and the dataset image hash code, and the Hamming distance is sorted, and the retrieval result is output.

[0075] In a specific implementation, corresponding binary hash codes can be generated for all images in the driving image database and stored; then, the query image uploaded by the user (i.e., the difficult sample image of autonomous driving) is obtained, and the hash code of the query image is obtained; then, the Hamming distance is calculated based on the hash code of the query image and the hash code of the image in the driving image database; finally, the retrieval results are sorted in descending order according to the Hamming distance, and the retrieved difficult example images of autonomous driving are returned.

[0076] This embodiment discloses obtaining an autonomous driving image dataset and constructing an initial autonomous driving image retrieval model, the initial autonomous driving image retrieval model including a feature extraction module and a hash quantization module; training the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model; based on a driving image database, inputting autonomous driving difficult sample images into the autonomous driving image retrieval model, and outputting retrieval results corresponding to the autonomous driving difficult sample images in the driving image database. Since this embodiment inputs autonomous driving difficult sample images into an autonomous driving image retrieval model including a feature extraction module and a hash quantization module, it outputs retrieval results corresponding to autonomous driving difficult sample images in the driving image database. Compared with the prior art, this embodiment effectively improves the efficiency and accuracy of autonomous driving difficult image retrieval.

[0077] refer to Figure 4 , Figure 4 Schematic diagram of the flow chart of the second embodiment of the difficult-example image retrieval method for autonomous driving of the present invention.

[0078] Based on the above first embodiment, in this embodiment, step S20 includes steps S201 to S203:

[0079] Step S201: Divide the autonomous driving image dataset into a training set, a validation set and a test set according to a preset ratio.

[0080] Step S202: input the training set in the autonomous driving image data set into the initial autonomous driving image retrieval model, perform model training on the initial autonomous driving image retrieval model using a preset loss function, and obtain a training result.

[0081] Step S203: Based on the training result, the initial autonomous driving image retrieval model is updated by a back propagation algorithm to obtain an autonomous driving image retrieval model.

[0082] It should be understood that the above preset ratio can be a custom setting, and this embodiment does not limit this.

[0083] In a specific implementation, in order to further improve the accuracy of difficult image retrieval in autonomous driving, after step S203, it also includes: using the verification set and the test set in the autonomous driving image dataset to perform model evaluation on the autonomous driving image retrieval model to obtain a model evaluation result; based on the model evaluation result, optimizing and adjusting the autonomous driving image retrieval model to obtain a new autonomous driving image retrieval model.

[0084] It should be noted that the above preset loss function includes a similarity information loss function and a hash quantization loss function. The similarity information loss function is:

[0085] ;

[0086] In the formula, , u and v Represents a hash code, Represents a hash code u and v The similarity between Represents a hash code u and v The similarity of the labels corresponding to the two images. Indicates that two images are similar. Indicates that the two images are not similar.

[0087] The hash quantization loss function is:

[0088] ;

[0089] Where N is the length of the hash code, i is the number of the hash code length, is the original hash code, is the binary hash code.

[0090] The preset loss function is:

[0091] ;

[0092] In the formula, is the weight parameter of the hash quantization loss, is similar information loss, Quantify the loss for hashing.

[0093] This embodiment discloses that the autonomous driving image data set is divided into a training set, a validation set and a test set according to a preset ratio; the training set in the autonomous driving image data set is input into the initial autonomous driving image retrieval model, and the initial autonomous driving image retrieval model is trained using a preset loss function to obtain a training result; based on the training result, the initial autonomous driving image retrieval model is updated by a back propagation algorithm to obtain an autonomous driving image retrieval model; the autonomous driving image retrieval model is evaluated by using the validation set and the test set in the autonomous driving image data set to obtain a model evaluation result; based on the model evaluation result, the autonomous driving image retrieval model is optimized and adjusted to obtain a new autonomous driving image retrieval model. Compared with the prior art, since this embodiment uses a preset loss function to train the initial autonomous driving image retrieval model, and based on the training result, the initial autonomous driving image retrieval model is updated by a back propagation algorithm, the accuracy of the autonomous driving image retrieval model for retrieval of difficult autonomous driving images is further improved.

[0094] refer to Figure 5 , Figure 5 Schematic diagram of the flow chart of the third embodiment of the difficult-example image retrieval method for autonomous driving of the present invention.

[0095] Based on the above embodiments, in this embodiment, after step S30, steps S40 to S50 are further included:

[0096] Step S40: Arrange the search results in descending order according to the Hamming distance to obtain an arrangement result.

[0097] Step S50: Based on the arrangement result, a preset number of autonomous driving difficult example images are selected to construct an autonomous driving difficult example training sample library.

[0098] It should be noted that by providing difficult sample images, images matching the query conditions can be quickly and accurately found from a massive driving image database, thereby building a difficult driving training sample library, and making targeted improvements and optimizations to the autonomous driving algorithm, which is conducive to optimizing the autonomous driving algorithm and improving the intelligent driving system's ability to respond to complex environments.

[0099] It should be understood that the Hamming distance is a metric used to measure the number of different characters (or elements) between two strings (or vectors) of equal length. The smaller the Hamming distance, the more similar the two strings (or vectors) are because they have fewer differences.

[0100] This embodiment discloses obtaining an autonomous driving image dataset and constructing an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module; training the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model; inputting autonomous driving difficult sample images into the autonomous driving image retrieval model based on a driving image database, and outputting retrieval results corresponding to the autonomous driving difficult sample images in the driving image database; arranging the retrieval results in descending order according to the Hamming distance to obtain an arrangement result; and selecting a preset number of autonomous driving difficult images based on the arrangement result to construct an autonomous driving difficult training sample library. Compared with the prior art, this embodiment constructs a driving difficult training sample library, which improves the diversity of autonomous driving image training samples, thereby facilitating the optimization of autonomous driving algorithms and improving the coping capabilities of intelligent driving systems in complex environments.

[0101] In addition, an embodiment of the present invention also proposes a storage medium, on which a difficult-to-retrieve image retrieval program for autonomous driving is stored. When the difficult-to-retrieve image retrieval program for autonomous driving is executed by a processor, the steps of the difficult-to-retrieve image retrieval method for autonomous driving as described above are implemented.

[0102] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the difficult-example image retrieval device for autonomous driving of the present invention.

[0103] like Figure 6 As shown, the difficult-example image retrieval device for autonomous driving proposed in an embodiment of the present invention includes: a model building module 601, a model training module 602 and a result output module 603.

[0104] The model building module 601 is used to obtain an autonomous driving image dataset and build an initial autonomous driving image retrieval model, which includes a feature extraction module and a hash quantization module.

[0105] The model training module 602 is used to perform model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model.

[0106] The result output module 603 is used to input the autonomous driving difficult example sample images into the autonomous driving image retrieval model based on the driving image database, and output the retrieval results corresponding to the autonomous driving difficult example sample images in the driving image database.

[0107] The result output module 603 is also used to use the feature extraction module of the autonomous driving image retrieval model to perform feature extraction on the images of the driving image database and the autonomous driving difficult example sample images, respectively, to obtain data set image features and sample image features; use the hash quantization module of the autonomous driving image retrieval model to convert the data set image features and sample image features into data set image hash codes and sample image hash codes, respectively; determine the hash code similarity between the sample image hash code and the data set image hash code; and output the retrieval result corresponding to the autonomous driving difficult example sample image in the driving image database according to the hash code similarity.

[0108] The result output module 603 is further used to calculate the Hamming distance between the sample image hash code and the dataset image hash code; and use the Hamming distance as the hash code similarity between the sample image hash code and the dataset image hash code.

[0109] The embodiment of the present device discloses obtaining an autonomous driving image dataset and constructing an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module; performing model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model; based on a driving image database, inputting autonomous driving difficult sample images into the autonomous driving image retrieval model, and outputting retrieval results corresponding to the autonomous driving difficult sample images in the driving image database. Since the embodiment of the present device inputs autonomous driving difficult sample images into an autonomous driving image retrieval model including a feature extraction module and a hash quantization module, and outputs retrieval results corresponding to autonomous driving difficult sample images in the driving image database. Compared with the prior art, the embodiment of the present device effectively improves the efficiency and accuracy of autonomous driving difficult image retrieval.

[0110] Based on the above-mentioned first embodiment of the difficult image retrieval device for autonomous driving of the present invention, a second embodiment of the difficult image retrieval device for autonomous driving of the present invention is proposed.

[0111] In this embodiment, the model training module 602 is also used to divide the autonomous driving image data set into a training set, a validation set and a test set according to a preset ratio; input the training set in the autonomous driving image data set into the initial autonomous driving image retrieval model, and use a preset loss function to perform model training on the initial autonomous driving image retrieval model to obtain a training result; based on the training result, the initial autonomous driving image retrieval model is updated by a back propagation algorithm to obtain an autonomous driving image retrieval model.

[0112] Other embodiments or specific implementation methods of the difficult-example image retrieval device for autonomous driving of the present invention may refer to the above-mentioned method embodiments and will not be described in detail here.

[0113] The present application provides an autonomous driving difficult image retrieval device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the autonomous driving difficult image retrieval method in the above-mentioned embodiment one.

[0114] Reference below Figure 7 , which shows a schematic diagram of the structure of an automatic driving difficult example image retrieval device suitable for implementing the embodiment of the present application. The automatic driving difficult example image retrieval device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The difficult-to-image retrieval device for autonomous driving shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Figure 7As shown, the difficult-case image retrieval device for autonomous driving may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the difficult-case image retrieval device for autonomous driving are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the autonomous driving difficult example image retrieval device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an autonomous driving difficult example image retrieval device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0116] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0117] The difficult-case image retrieval device for autonomous driving provided by the present application adopts the difficult-case image retrieval method for autonomous driving in the above-mentioned embodiment, which can solve the technical problem of low efficiency and accuracy of difficult-case image retrieval for autonomous driving in the prior art. Compared with the prior art, the beneficial effects of the difficult-case image retrieval device for autonomous driving provided by the present application are the same as the beneficial effects of the difficult-case image retrieval method for autonomous driving provided by the above-mentioned embodiment, and the other technical features of the difficult-case image retrieval device for autonomous driving are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0118] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0120] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0121] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0123] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for retrieval of difficult-example images for autonomous driving, characterized in that: The method comprises: Acquire an autonomous driving image dataset and construct an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module; Performing model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model; Based on a driving image database, inputting the autonomous driving difficult sample image into the autonomous driving image retrieval model, and outputting a retrieval result corresponding to the autonomous driving difficult sample image in the driving image database; The step of performing model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain the autonomous driving image retrieval model comprises: Dividing the autonomous driving image dataset into a training set, a validation set, and a test set according to a preset ratio; Inputting the training set in the autonomous driving image data set into the initial autonomous driving image retrieval model, and performing model training on the initial autonomous driving image retrieval model using a preset loss function to obtain a training result; Based on the training result, updating the initial autonomous driving image retrieval model through a back propagation algorithm to obtain an autonomous driving image retrieval model; The preset loss function includes a similarity information loss function and a hash quantization loss function; The similarity information loss function is: ； In the formula, , u and v Represents a hash code, Represents a hash code u and v The similarity between Represents a hash code u and v The similarity of the labels corresponding to the two images. Indicates that two images are similar. Indicates that the two images are not similar; The hash quantization loss function is: ； In the formula, N is the length of the hash code, i is the number of the hash code length, is the original hash code, is the binary hash code; The preset loss function is: ； In the formula, is the weight parameter of the hash quantization loss, is similar information loss, Quantify the loss for hashing.

2. The method for retrieving difficult-example images for autonomous driving according to claim 1, wherein: After the step of updating the initial autonomous driving image retrieval model based on the training result by a back propagation algorithm to obtain the autonomous driving image retrieval model, the method further includes: Using the validation set and the test set in the autonomous driving image dataset to perform model evaluation on the autonomous driving image retrieval model to obtain a model evaluation result; Based on the model evaluation result, the autonomous driving image retrieval model is optimized and adjusted to obtain a new autonomous driving image retrieval model.

3. The method for retrieving difficult-example images for autonomous driving according to claim 1, wherein: The step of inputting the autonomous driving difficult sample image into the autonomous driving image retrieval model based on the driving image database and outputting the retrieval result corresponding to the autonomous driving difficult sample image in the driving image database comprises: Using the feature extraction module of the autonomous driving image retrieval model, respectively extract features from the images in the driving image database and the autonomous driving difficult sample images to obtain dataset image features and sample image features; Using the hash quantization module of the autonomous driving image retrieval model, the dataset image features and the sample image features are converted into a dataset image hash code and a sample image hash code respectively; Determining hash code similarity between the sample image hash code and the dataset image hash code; The retrieval result corresponding to the autonomous driving difficult sample image in the driving image database is output according to the hash code similarity.

4. The method for retrieving difficult-example images for autonomous driving according to claim 3, wherein: The step of determining the hash code similarity between the sample image hash code and the dataset image hash code comprises: Calculating the Hamming distance between the sample image hash code and the dataset image hash code; Using the Hamming distance as the hash code similarity between the sample image hash code and the dataset image hash code; The calculation formula of the Hamming distance is: ； In the formula, N is the length of the hash code, i is the number of the hash code length, and Represents the difficult sample images of autonomous driving q and images from the Driving Image Database p In the i The hash value on bits.

5. The method for retrieving difficult-example images for autonomous driving according to claim 4, wherein: After the step of inputting the autonomous driving difficult sample image into the autonomous driving image retrieval model based on the driving image database and outputting the retrieval result corresponding to the autonomous driving difficult sample image in the driving image database, the method further includes: Arrange the search results in descending order according to the Hamming distance to obtain an arrangement result; Based on the arrangement results, a preset number of autonomous driving difficult example images are selected to construct an autonomous driving difficult example training sample library.

6. An automatic driving difficult example image retrieval device, characterized in that: The device comprises: A model building module, used to obtain an autonomous driving image dataset and build an initial autonomous driving image retrieval model, wherein the initial autonomous driving image retrieval model includes a feature extraction module and a hash quantization module; A model training module, used to perform model training on the initial autonomous driving image retrieval model based on the autonomous driving image dataset to obtain an autonomous driving image retrieval model; A result output module is used to input the autonomous driving difficult sample image into the autonomous driving image retrieval model based on the driving image database, and output the retrieval result corresponding to the autonomous driving difficult sample image in the driving image database; The model training module is further used to divide the autonomous driving image data set into a training set, a validation set and a test set according to a preset ratio; input the training set in the autonomous driving image data set into the initial autonomous driving image retrieval model, and perform model training on the initial autonomous driving image retrieval model using a preset loss function to obtain a training result; based on the training result, the initial autonomous driving image retrieval model is updated by a back propagation algorithm to obtain an autonomous driving image retrieval model; The preset loss function includes a similarity information loss function and a hash quantization loss function; The similarity information loss function is: ； In the formula, , u and v Represents a hash code, Represents a hash code u and v The similarity between Represents a hash code u and v The similarity of the labels corresponding to the two images. Indicates that two images are similar. Indicates that the two images are not similar; The hash quantization loss function is: ； In the formula, N is the length of the hash code, i is the number of the hash code length, is the original hash code, is the binary hash code; The preset loss function is: ； In the formula, is the weight parameter of the hash quantization loss, is similar information loss, Quantify the loss for hashing.

7. An automatic driving difficult example image retrieval device, characterized in that: The device includes: a memory, a processor, and an autonomous driving difficult image retrieval program stored in the memory and executable on the processor, wherein the autonomous driving difficult image retrieval program is configured to implement the steps of the autonomous driving difficult image retrieval method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium stores an autonomous driving difficult example image retrieval program, and when the autonomous driving difficult example image retrieval program is executed by the processor, the steps of the autonomous driving difficult example image retrieval method as described in any one of claims 1 to 5 are implemented.

Images