Image processing method and device

Abstract

The present application discloses an image processing method and device, and relates to the field of computer vision and deep learning technology. The specific implementation method includes: determining the feature vector of the target image through a trained neural network; using the feature vector of the target image to perform image retrieval in a database to obtain retrieval results, wherein each image in the retrieval results has a label; sorting the retrieval results based on the label to obtain a sorted result; and outputting the sorted result as the result corresponding to the target image. The embodiment of the present application obtains the retrieval result by calculating the similarity of the feature vectors of the target image and the image in the database, and sorting the images in the retrieval result based on the retrieval result, thereby improving the retrieval accuracy and making the retrieval result more in line with the user's expectations.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, specifically to the field of computer vision and deep learning technology, and in particular to image processing methods and devices. Background Art

[0002] Artificial intelligence is an emerging science and technology that is currently being studied and developed to simulate, extend and expand human intelligence. Nowadays, artificial intelligence technology has been widely used in many scenarios, such as image retrieval scenarios. Simply put, image retrieval is to retrieve other images similar or identical to a certain image from a large number of images, that is, to find images by images, allowing users to enter an image to find other similar or identical images.

[0003] In image retrieval, most of the features are designed manually, or an artificial intelligence neural network model is directly trained as the basis for image retrieval, resulting in poor retrieval accuracy. Summary of the invention

[0004] Provided are an image processing method, an image processing device, an electronic device, and a storage medium.

[0005] According to a first aspect, an image processing method is provided, comprising: determining a feature vector of a target image through a trained neural network; performing image retrieval in a database using the feature vector of the target image to obtain retrieval results, wherein each image in the retrieval results has a label; sorting the retrieval results based on the labels to obtain sorted results; and outputting the sorted results as results corresponding to the target image.

[0006] According to the second aspect, an image processing device is provided, comprising: a determination module, configured to determine a feature vector of a target image through a trained neural network; a retrieval module, configured to perform image retrieval in a database using the feature vector of the target image to obtain retrieval results, wherein each image in the retrieval results has a label; a sorting module, configured to sort the retrieval results based on the labels to obtain sorted results; and an output module, configured to output the sorted results as results corresponding to the target image.

[0007] According to a third aspect, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a method as in any embodiment of the image processing method.

[0008] According to a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, a method as in any embodiment of the image processing method is implemented.

[0009] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method according to any embodiment of the image processing method.

[0010] According to the scheme of the present application, the retrieval results are obtained by calculating the similarity of the feature vectors between the target image and the images in the database, and the images in the retrieval results are sorted based on the labels of the images in the retrieval results, thereby improving the retrieval accuracy and making the retrieval results more in line with the user's expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0012] Figure 1 is an exemplary system architecture diagram to which some embodiments of the present application may be applied;

[0013] Figure 2 is a flow chart of an embodiment of an image processing method according to the present application;

[0014] Figure 3 is a schematic diagram of an application scenario of the image processing method according to the present application;

[0015] Figure 4 is a flowchart of another embodiment of the image processing method according to the present application;

[0016] Figure 5 is a flowchart of another embodiment of the image processing method according to the present application;

[0017] Figure 6 is a structural schematic diagram of an embodiment of an image processing device according to the present application;

[0018] Figure 7 It is a block diagram of an electronic device used to implement the image processing method of an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0020] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] Figure 1 An exemplary system architecture 100 is shown to which an embodiment of the image processing method or image processing apparatus of the present application can be applied.

[0022] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0023] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as browser applications, video applications, live broadcast applications, instant messaging tools, email clients, social platform software, etc.

[0024] The terminal devices 101, 102, 103 here can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers and desktop computers, etc. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented as multiple software or software modules (for example, multiple software or software modules for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0025] The server 105 may be a server that provides various services, such as a background server that provides support for the terminal devices 101, 102, and 103. The background server may analyze and process the received target image and other data, and feed back the processing result (such as the sorting result) to the terminal device.

[0026] It should be noted that the image processing method provided in the embodiment of the present application can be executed by the server 105 or the terminal devices 101, 102, 103, and accordingly, the image processing device can be set in the server 105 or the terminal devices 101, 102, 103.

[0027] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0028] Continue to refer Figure 2 , shows a process 200 of an embodiment of an image processing method according to the present application. The image processing method comprises the following steps:

[0029] Step 201, determining a feature vector of a target image through a trained neural network.

[0030] In this embodiment, the execution subject (eg Figure 1 The server or terminal device shown in the figure can extract feature vectors from a target image actively or passively acquired, or can extract feature vectors from multiple received target images respectively. Specifically, the neural network here can refer to a deep neural network.

[0031] It should be understood that there are many ways for the above-mentioned execution subject to obtain the target image. For example, the user may send it to the server through the terminal, and the server may obtain it; or the processor of the terminal device may obtain it after the user issues an instruction. The embodiments of the present application do not impose any special restrictions on this.

[0032] In this embodiment, the neural network is trained through metric learning, which can obtain features with stronger discrimination between categories, thereby significantly improving retrieval accuracy.

[0033] Step 202, using the feature vector of the target image to perform image retrieval in the database to obtain retrieval results, wherein each image in the retrieval results has a label.

[0034] In this embodiment, the image data in the database includes a feature vector and a label of each image. The feature vector and the label of each image may be determined in advance by a trained neural network.

[0035] The implementation of this embodiment can be implemented based on a variety of methods. For example, in one implementation, the above-mentioned execution subject can use the feature vector of the target image to perform image retrieval in the database, and the specific process of obtaining the retrieval result may include: the above-mentioned execution subject first determines the cosine similarity between the feature vectors of the target image and all images in the database, and then sorts the images in order from high to low according to the cosine similarity, and selects the first K results as the retrieval results, where K is greater than zero. Alternatively, the above-mentioned execution subject first determines the cosine similarity between the feature vectors of the target image and all images in the database, and then determines the images whose cosine similarity is higher than a preset threshold, and uses the images whose cosine similarity is higher than the preset threshold as the retrieval results.

[0036] Step 203: sort the search results based on the tags to obtain sorted results.

[0037] In the prior art, only the search results based on image features are used as the final results. However, if only the similarity of image features is compared, the image sorting results may not be reasonable, and the required correct search results may be ranked behind other erroneous search results.

[0038] In this embodiment, the execution entity may use multiple images in the search results as preliminary results, and adjust the ranking results of the multiple images in the search results by labels as output results, so as to further improve the accuracy of similarity search performed only by feature vectors.

[0039] It should be understood that the above-mentioned execution entity can use various methods to sort the above-mentioned search results based on labels. For example, the images in the above-mentioned search results can be re-sorted according to the labels of the images in the search results, or the image sorting in the search results can be adjusted according to the labels of the images in the search results.

[0040] It should also be understood that the embodiments of the present application can also sort the search results based on the weight and label of each image in the search results, and the weight is determined according to the difference level between the feature vector of each image in the search and the above-mentioned preset threshold. The specific implementation of sorting the search results based on the above-mentioned weights and labels can be implemented according to any algorithm in the prior art, and the embodiments of the present application do not impose any particular limitation on this.

[0041] Step 204: output the sorting result as the result corresponding to the target image.

[0042] In this embodiment, there are many ways to implement the output of the results by the execution subject. For example, the sorting results may be sent or displayed through a terminal device, or sent by a server to a terminal device and displayed by the terminal device.

[0043] The method provided by the above-mentioned embodiment of the present application obtains retrieval results by calculating the similarity of feature vectors between the target image and the images in the database, and sorts the images in the retrieval results based on the labels of the images in the retrieval results, thereby improving the retrieval accuracy and making the retrieval results more in line with user expectations.

[0044] Continue to see Figure 3 , Figure 3 FIG. 1 is a schematic diagram of an application scenario of the image processing method according to this embodiment. Figure 3In the application scenario, the terminal device 301 sends the target image to be retrieved to the execution subject server. After receiving the image, the server extracts features through the neural network 302 to obtain the feature vector 303 corresponding to the target image, and then retrieves the image through the cosine similarity between the feature vector 303 and the feature vector of the image in the database 304, and sorts the images from high to low using the cosine similarity, and uses the obtained result as the retrieval result 305. Then, the retrieval result is sorted according to the label, and the obtained result is used as the sorting result 306. Finally, the sorting result 306 is output to the terminal device 301 as the result corresponding to the target image.

[0045] Further references Figure 4 , which shows a process 400 of another embodiment of the image processing method. Process 400 determines the confidence label by the label of the image in the search result, and sorts the image based on the confidence label, which can verify the search result and further improve the search accuracy. Moreover, by selecting a preset number of results with higher similarity in the search result, the number of samples is reduced, noise interference is avoided to a certain extent, and the confidence of the confidence label is improved. The process 400 includes the following steps:

[0046] Step 401, determining a feature vector of a target image through a trained neural network.

[0047] Step 402, using the feature vector of the target image to perform image retrieval in the database to obtain retrieval results, wherein each image in the retrieval results has a label, and the retrieval results are image sequences arranged according to feature vector similarity.

[0048] Step 401 and step 402 are basically the same as step 201 and step 202, and thus will not be described in detail.

[0049] Step 403, selecting a preset number of images from the search results as reference images in descending order of feature vector similarity of the images in the image sequence;

[0050] It should be understood that the order of the image sequence can be from large to small according to the similarity of the feature vectors, or from small to large according to the similarity of the feature vectors, and the present application does not impose any particular limitation on this.

[0051] In this embodiment, the execution subject (eg Figure 1 The server or terminal device shown in the figure further selects a preset number of images as reference images based on the image sequence arranged from large to small in terms of feature vector similarity. For example, based on the K images corresponding to the image sequence, the first N images are further selected as reference images, where N is greater than zero.

[0052] It should be understood that the method of further selecting the first N images as reference images may be to select the first N images continuously, or to select the first N images at a certain interval, or to select the first N images by sampling.

[0053] Step 404: Based on the frequencies of the labels corresponding to the reference image, determine the label with the highest frequency in the reference image as a confidence label.

[0054] In this embodiment, taking stamp and coin data as an example, when the execution subject selects the first ten images of the search results (N=10), there are five one-cent coins, three two-cent coins, and two five-cent coins. At this time, the one-cent coin has the highest frequency and number of occurrences, so the one-cent coin is used as the confidence label. When some special cases occur, such as N=1, the execution subject can use the label of the image with the highest cosine similarity in the search results as the confidence label.

[0055] Step 405: adjust the image sequence using the confidence labels to obtain a ranking result.

[0056] In this embodiment, the execution subject adjusts the image sequence (K images) based on the confidence label, which can be implemented in a variety of ways. For example, the execution subject determines whether the label of each image in the retrieval result is consistent with the confidence label; if not, the ranking of the image corresponding to the inconsistent label is moved backward in the retrieval result; if consistent, the ranking of the image corresponding to the consistent label remains unchanged. Alternatively, the execution subject determines whether the label of each image in the retrieval result is consistent with the confidence label; if not, the ranking of the image corresponding to the inconsistent label remains unchanged in the retrieval result; if consistent, the ranking of the image corresponding to the consistent label is moved forward.

[0057] Finally, the execution subject can make the images inconsistent with the confidence labels in the adjusted image sorting (sorting result) be arranged behind the images consistent with the confidence labels, and the images with confidence labels are arranged from high to low with the cosine similarity maintained.

[0058] The embodiment of the present application adjusts the image sequence of the retrieval result based on the confidence label, ensuring that images inconsistent with the confidence label are sorted behind images consistent with the confidence label, thereby ensuring the rationality of the image sorting in the sorting result.

[0059] Step 406: Output the sorting result as the result corresponding to the target image. Step 406 is basically the same as step 204, so it will not be described in detail.

[0060] Further references Figure 5, which shows a process 500 of another embodiment of an image processing method. The neural network includes: a backbone network, a batch normalization (BN) layer, a global average pooling (GAP) layer, a fully connected (FC) layer and a classification head.

[0061] In the prior art, a classification model is usually trained directly. For example, there are 100 categories of stamp and coin data. The model outputs 100-dimensional classification accuracy results, and the category with the highest accuracy is regarded as the recognition result. A feature is obtained through the feature output layer before the model classification layer as a retrieval basis. Alternatively, features such as SIFT (Scale-Invariant Feature Transform) and LBP (Local Binary Pattern) are artificially designed to directly calculate the features of stamp and coin images as a retrieval basis.

[0062] However, the above operation mode of the prior art has the following problems:

[0063] First, the features obtained by the neural network classification model are suitable for classification tasks, but not for retrieval. The main reason is that this classification method does not constrain the features between different categories to be discriminative, and only needs to output the probability that conforms to the data distribution. For retrieval, discriminative features are more conducive to retrieving similar images.

[0064] Second, the accuracy of manually designed features is generally weaker than that of features trained on labeled data sets using neural network models, such as convolutional neural networks (CNNs). The main reason is that data in real retrieval scenarios usually have various conditions such as lighting, angles, complex backgrounds, and occlusions. Manual features cannot cope with all situations, while neural network models can learn various complex situations in data sets through deep learning methods, thus avoiding the problem of weak features. In addition, manual feature design is time-consuming and labor-intensive, which is not conducive to the rapid iteration of subsequent processes.

[0065] Taking stamp and coin data as an example, stamp and coin data can include coins, ancient coins (copper coins, knife coins, etc.), silver dollars, foreign coins, commemorative coins and other coins. The characteristics of this type of data are that the categories are complex and complicated, and the shapes are mostly round and similar in size, with high similarity. The same type of coins are only distinguished in value, such as a two-cent coin and a one-cent coin, which have only slight differences, but cannot be classified into the same category. At present, it is impossible to obtain highly discriminative feature vectors through simple classification CNN models, and thus it is impossible to achieve high-quality retrieval results.

[0066] Process 500 reorganizes the structure of the neural network classification model and uses metric learning to obtain a more discriminative feature vector, which can better improve the retrieval accuracy. The process 500 includes the following steps:

[0067] Step 501, extracting the feature vector of the target image through the backbone network, batch normalization layer, global average pooling layer, and fully connected layer.

[0068] In this embodiment, the backbone network (Backbone) can be a neural network classification model used for classification, such as resnet50, resnet101, etc., such as a part of CNN used for classification, or a part of other models used for classification. This application does not impose any special restrictions on this.

[0069] In some embodiments, if the penultimate layer of the neural network has a pooling layer, the last layer for classification of the penultimate layer of the neural network can be removed. The layer used for classification is usually an FC layer, so that the input image is processed by the spine network and the output is N*C*H*W, where N represents N feature matrices, C represents the number of channels, H represents the feature map height, and W represents the feature map width.

[0070] Furthermore, based on the backbone network, the above-mentioned execution body is also provided with a BN layer, a GAP layer, and a FC layer, that is, the feature map is normalized through the BN layer in turn, the H*W (feature map height, feature map width) is eliminated through the GAP layer, and the feature vector of a preset length is obtained through the FC layer. For example, the feature vector of the preset length can be 512 dimensions, 1024 dimensions, etc.

[0071] It should be understood that the above-mentioned execution entity can also pre-process the images in the database through the trained neural network to obtain the feature vector and label of each image. The specific steps may include: extracting the feature vector of the image in the retrieval result through the backbone network, batch normalization layer, global average pooling layer, and fully connected layer; inputting the feature vector of the image in the retrieval result into the classification head, and determining the label vector of the image in the retrieval result through the classification head; generating a label through the label vector.

[0072] In the above embodiment, a more discriminative feature vector has been obtained. Therefore, the label obtained according to the more discriminative feature vector is also more discriminative, thereby achieving better sorting.

[0073] It should be understood that generating labels through label vectors can be implemented through various existing technologies, such as through softmax, and the embodiments of the present application do not impose any particular limitation on this.

[0074] It should also be understood that, since the label vector is generated by a certain specific method, in some other specific embodiments of the present application, the label vector can be directly used as a label, that is, the label vector is directly used as the basis for sorting the search results.

[0075] In this embodiment, the implementation principle of the classification head (ArcMargin) is similar to that of ArcFace, which will not be repeated here.

[0076] Step 502, using the feature vector of the target image to perform image retrieval in the database to obtain retrieval results, wherein each image in the retrieval results has a label.

[0077] Step 503: sort the search results based on the tags to obtain sorted results.

[0078] Step 504: output the sorting result as the result corresponding to the target image.

[0079] Steps 502, 503, and 504 are basically the same as steps 202, 203, and 204, and thus will not be described in detail.

[0080] In some optional embodiments, the neural network can be trained by various loss functions, such as mean square error loss function, cross entropy loss function (softmax loss), etc.

[0081] In an exemplary practical application scenario, the embodiment of the present application trains a neural network through a labeled stamp and coin dataset. The labels of different categories of datasets only contain image data of the category. For example, the data under the one-cent coin label can be one-cent coin images from different angles, different lighting conditions, and different backgrounds. The trained neural network can output a corresponding feature vector and label vector for each input image.

[0082] Further references Figure 6 As an implementation of the methods shown in the above figures, the present application provides an embodiment of an image processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown in FIG. 1 , in addition to the features described below, the device embodiment may also include Figure 2 The device can be applied to various electronic devices.

[0083] like Figure 6As shown, the image processing device 600 of this embodiment includes: a determination module 601, configured to determine the feature vector of the target image through a trained neural network; a retrieval module 602, configured to use the feature vector of the target image to perform image retrieval in a database to obtain retrieval results, wherein each image in the retrieval results has a label; a sorting module 603, configured to sort the retrieval results based on the labels to obtain sorting results; and an output module 604, configured to output the sorting results as the results corresponding to the target image.

[0084] In this embodiment, the specific processing of the image processing device 600 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of step 201, step 202, step 203 and step 204 in the corresponding embodiment are not repeated here.

[0085] In some optional implementations of the present embodiment, the retrieval result is an image sequence arranged according to feature vector similarity; the sorting module 603 includes: a selection unit, configured to select a preset number of images in the retrieval result as reference images in descending order of feature vector similarity of the images in the image sequence; a determination unit, configured to determine, based on the frequency of the labels corresponding to the reference images, the label with the highest frequency in the reference image as a confidence label; and an adjustment unit, adjusting the image sequence through the confidence labels to obtain a sorting result.

[0086] In some optional implementations of this embodiment, the adjustment unit is further configured to determine whether the label of each image in the retrieval result is consistent with the confidence label; if inconsistent, the ranking of the image corresponding to the inconsistent label is moved backward in the retrieval result.

[0087] In some optional implementations of this embodiment, the neural network includes, in sequence: a backbone network, a batch normalization layer, a global average pooling layer, and a fully connected layer; the determination module 601 is further configured to extract a feature vector of the target image through the backbone network, the batch normalization layer, the global average pooling layer, and the fully connected layer.

[0088] In some optional implementations of this embodiment, the neural network also includes a classification head located after the fully connected layer; the determination module 601 is further configured to extract the feature vector of the image in the retrieval result through the backbone network, the batch normalization layer, the global average pooling layer, and the fully connected layer; input the feature vector of the image in the retrieval result into the classification head, and determine the label vector of the image in the retrieval result through the classification head; generate a label through the label vector.

[0089] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0090] like Figure 7 , is a block diagram of an electronic device according to an image processing method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0091] like Figure 7 As shown, the electronic device includes: one or more processors 701, a memory 702, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 701 is taken as an example.

[0092] The memory 702 is a non-transitory computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable at least one processor to perform the image processing method provided in the present application. The non-transitory computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the image processing method provided in the present application.

[0093] The memory 702 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the image processing method in the embodiment of the present application (for example, the attached Figure 6 The processor 701 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 702, that is, the image processing method in the above method embodiment is implemented.

[0094] The memory 702 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the image retrieval electronic device, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 702 may optionally include a memory remotely arranged relative to the processor 701, and these remote memories may be connected to the image processing electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0095] The electronic device of the image processing method may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703 and the output device 704 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0096] The input device 703 can receive input digital or character information, and generate key signal input related to user settings and function control of the image retrieval electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 704 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0097] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0098] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0100] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0101] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0102] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0103] The modules or units involved in the embodiments described in the present application may be implemented by software or hardware. The modules or units described may also be set in a processor. For example, they may be described as: a processor includes a determination module, a retrieval module, a sorting module, and an output module. The names of these modules do not, in some cases, constitute limitations on the modules themselves. For example, the determination module may also be described as a "module for determining a feature vector of a target image."

[0104] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the device, the device: determines the feature vector of the target image through the trained neural network; uses the feature vector of the target image to perform image retrieval in the database to obtain retrieval results, wherein each image in the retrieval results has a label; sorts the retrieval results based on the label to obtain a sorted result; and outputs the sorted result as the result corresponding to the target image.

[0105] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above invention concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. An image processing method, the method comprising: Determining a feature vector of a target image by means of a trained neural network, wherein the neural network is trained by metric learning, wherein the metric learning is used to enhance inter-category discrimination of feature vectors of images of different categories; Performing image retrieval in a database using the feature vector of the target image to obtain retrieval results, wherein each image in the retrieval results has a label representing the image category, and the retrieval results are image sequences arranged according to feature vector similarity; Selecting a preset number of images from the search results as reference images in descending order of feature vector similarities of the images in the image sequence; Based on the frequency of the labels corresponding to the reference image, determining the label with the highest frequency in the reference image as a confident label; By determining whether the label of each image in the retrieval result is consistent with the confidence label, the retrieval result is sorted to obtain a sorted result; and the sorted result is output as a result corresponding to the target image.

2. The method according to claim 1, wherein: The step of determining whether the label of each image in the search result is consistent with the confidence label and sorting the search result to obtain the sorting result includes: If they are inconsistent, the ranking of the image corresponding to the inconsistent label is moved backward in the image sequence.

3. The method according to claim 1, wherein: The neural network includes, in sequence: a backbone network, a batch normalization layer, a global average pooling layer, and a fully connected layer; The step of determining the feature vector of the target image by using the trained neural network includes: The feature vector of the target image is extracted through the backbone network, the batch normalization layer, the global average pooling layer, and the fully connected layer.

4. The method according to claim 3, wherein: The neural network also includes a classification head located after the fully connected layer; The step of generating labels for images in the database comprises: Extracting feature vectors of images in the retrieval results through the backbone network, the batch normalization layer, the global average pooling layer, and the fully connected layer; Inputting the feature vector of the image in the retrieval result into the classification head, and determining the label vector of the image in the retrieval result through the classification head; The labels are generated by a label vector.

5. An image processing device, comprising: A determination module is configured to determine a feature vector of a target image through a trained neural network, wherein the neural network is trained through metric learning, and the metric learning is used to enhance the inter-category discrimination of feature vectors of images of different categories; A retrieval module is configured to perform image retrieval in a database using the feature vector of the target image to obtain a retrieval result, wherein each image in the retrieval result has a label representing the image category, and the retrieval result is an image sequence arranged according to the similarity of the feature vector; A sorting module is configured to select a preset number of images from the search results as reference images in descending order of feature vector similarities of the images in the image sequence; Based on the frequency of the labels corresponding to the reference image, determining the label with the highest frequency in the reference image as a confident label; By determining whether the label of each image in the retrieval result is consistent with the confidence label, the retrieval result is sorted to obtain a sorting result; The output module is configured to output the sorting result as a result corresponding to the target image.

6. The device according to claim 5, wherein: The step of judging whether the label of each image in the search result is consistent with the confidence label in the sorting module includes: If they are inconsistent, the ranking of the image corresponding to the inconsistent label is moved backward in the image sequence.

7. The device according to claim 5, wherein: The neural network includes, in sequence: a backbone network, a batch normalization layer, a global average pooling layer, and a fully connected layer; The determination module is further configured to extract the feature vector of the target image through the backbone network, the batch normalization layer, the global average pooling layer, and the fully connected layer.

8. The device according to claim 7, wherein: The neural network also includes a classification head located after the fully connected layer; The determination module is further configured to extract the feature vector of the image in the retrieval result through the backbone network, the batch normalization layer, the global average pooling layer, and the fully connected layer; input the feature vector of the image in the retrieval result into the classification head, and determine the label vector of the image in the retrieval result through the classification head; and generate the label through the label vector.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 4.

Images