An image search method, device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide an image search method and device, electronic equipment and storage medium, and relate to the technical field of image processing. The method is applied to an electronic equipment, the electronic equipment is configured with a persistent memory, the persistent memory is used to store image features of first candidate images, and the method comprises the following steps: obtaining a target image feature of a to-be-searched image; obtaining a search range; if there is an image in the search range in the first candidate images, directly accessing the persistent memory to obtain the image features of each first candidate image in the persistent memory, calculating the similarity between the target image feature and the image features of each first candidate image; and determining, according to the calculated similarity, an image similar to the to-be-searched image from the first candidate images. The scheme provided by the embodiments of the present application can improve the image search efficiency.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image search method, apparatus, electronic device and storage medium. Background Technology

[0002] In image processing, it is often necessary to search for images similar to the image being searched from an existing image library. For example, in a face recognition scenario, after acquiring a user's face image, it is necessary to search for images similar to that face image from an existing face image library.

[0003] In related technologies, electronic devices typically store the image features of each image in an image library on a hard disk beforehand. When an image search is needed, to facilitate the electronic device reading these image features, the image features from the hard disk need to be transferred to volatile memory. The electronic device can obtain the image features of the image to be searched, perform similarity calculations between the image features of the image to be searched and the image features in volatile memory, and determine the images in the image library that are similar to the image to be searched based on the calculation results.

[0004] While the above approach can achieve image search, it requires transferring image features from the hard drive to volatile memory, which increases the time consumption and results in low image search efficiency. Summary of the Invention

[0005] The purpose of this application is to provide an image search method, apparatus, electronic device, and storage medium to improve image search efficiency. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide an image search method applied to an electronic device, the electronic device being configured with persistent memory for storing image features of a first candidate image, the method comprising:

[0007] Obtain the target image features of the image to be searched;

[0008] The search range is obtained. If there is an image in the first candidate image whose search popularity is within the search range, the persistent memory is directly accessed to obtain the image features of each first candidate image in the persistent memory, and the similarity between the target image features and the image features of each first candidate image is calculated.

[0009] Based on the calculated similarity, images similar to the image to be searched are determined from the first candidate images.

[0010] In one embodiment of this application, the electronic device is further configured with volatile memory, which is used to store image features of a second candidate image;

[0011] The method further includes:

[0012] If there is an image in the second candidate image whose search popularity is within the search range, directly access the volatile memory, obtain the image features of each second candidate image in the volatile memory, and calculate the similarity between the target image features and the image features of each second candidate image;

[0013] Based on the calculated similarity, images similar to the image to be searched are determined from the second candidate images.

[0014] In one embodiment of this application, the search popularity of the second candidate image is higher than that of the first candidate image. The search popularity is described by at least one of the following: the time of image acquisition, the image content, the search hit frequency of the image, the data source of the image, and the application scenario of the image.

[0015] The process of obtaining the search range includes:

[0016] Obtain the search scope based on search popularity.

[0017] In one embodiment of this application, the persistent memory operates in AD mode, wherein the persistent memory stores data with persistent non-volatility in AD mode.

[0018] In one embodiment of this application, the electronic device is further configured with a hard disk for storing the first candidate image, and there is a correspondence between each first candidate image in the hard disk and each image feature in the persistent memory;

[0019] The step of determining images similar to the image to be searched from the first candidate images based on the calculated similarity includes:

[0020] Based on the calculated similarity, image features whose similarity to the target image features meets preset similarity conditions are determined and used as reference image features;

[0021] According to the correspondence, the image corresponding to the reference image feature is searched from the first candidate images stored on the hard disk.

[0022] Secondly, embodiments of this application provide an image search device, the device being disposed in an electronic device, the electronic device being configured with persistent memory, the persistent memory being used to store image features of a first candidate image, the device comprising:

[0023] The feature acquisition module is used to obtain the target image features of the image to be searched.

[0024] The range acquisition module is used to obtain the search range;

[0025] The first similarity calculation module is used to directly access the persistent memory if there is an image in the first candidate image whose search popularity is within the search range, obtain the image features of each first candidate image in the persistent memory, and calculate the similarity between the target image features and the image features of each first candidate image;

[0026] The first image search module is used to determine images similar to the image to be searched from the first candidate images based on the calculated similarity.

[0027] In one embodiment of this application, the electronic device is further configured with volatile memory, which is used to store image features of a second candidate image;

[0028] The device further includes:

[0029] The second similarity calculation module is used to directly access the volatile memory if there is an image in the second candidate image whose search popularity is within the search range, obtain the image features of each second candidate image in the volatile memory, and calculate the similarity between the target image features and the image features of each second candidate image.

[0030] The second image search module is used to determine images similar to the image to be searched from the second candidate images based on the calculated similarity.

[0031] In one embodiment of this application, the search popularity of the second candidate image is higher than that of the first candidate image. The search popularity is described by at least one of the following: the time of image acquisition, the image content, the search hit frequency of the image, the data source of the image, and the application scenario of the image.

[0032] The range acquisition module is specifically used for:

[0033] Obtain the search scope based on search popularity.

[0034] In one embodiment of this application, the persistent memory operates in AD mode, wherein the persistent memory stores data with persistent non-volatility in AD mode.

[0035] In one embodiment of this application, the electronic device is further configured with a hard disk for storing the first candidate image, and there is a correspondence between each first candidate image in the hard disk and each image feature in the persistent memory;

[0036] The first image search module is specifically used for:

[0037] Based on the calculated similarity, image features whose similarity to the target image features meets preset similarity conditions are determined and used as reference image features;

[0038] According to the correspondence, the image corresponding to the reference image feature is searched from the first candidate images stored on the hard disk.

[0039] Thirdly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0040] Memory, used to store computer programs;

[0041] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0042] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0043] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform any of the image search methods described above.

[0044] Beneficial effects of the embodiments in this application:

[0045] The image search scheme provided in this application can be applied to electronic devices. The electronic device is equipped with persistent memory, which stores image features of first candidate images. The electronic device can obtain the target image features of the image to be searched; determine the search range; if there are images in the first candidate images whose search popularity falls within the search range, it directly accesses the persistent memory to obtain the image features of each first candidate image in the persistent memory, calculates the similarity between the target image features and the image features of each first candidate image; and, based on the calculated similarity, determines images similar to the image to be searched from the first candidate images. This allows searching for images similar to the image to be searched from the first candidate images based on the similarity between the image features of the first candidate images and the image features of the image to be searched. Since persistent memory supports byte-level addressing, image features can be directly read from persistent memory without copying them back to memory, thus saving resource consumption associated with data copying and improving image search efficiency. Therefore, applying the scheme provided in this application can improve image search efficiency. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0047] Figure 1 A flowchart illustrating an image search method provided in this application embodiment;

[0048] Figure 2 A flowchart illustrating another image search method provided in an embodiment of this application;

[0049] Figure 3 This application provides a schematic diagram of the structure of a distributed system according to an embodiment of the present application.

[0050] Figure 4 This is a schematic diagram of the structure of an image search device provided in an embodiment of this application;

[0051] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0053] To improve image search efficiency, embodiments of this application provide an image search method, apparatus, electronic device, and storage medium, which will be described in detail below.

[0054] See Figure 1 , Figure 1 This application provides a flowchart of an image search method, which can be applied to electronic devices such as computers, servers, mobile phones, NVRs (Network Video Recorders), and DVRs (Digital Video Recorders).

[0055] The aforementioned electronic device is equipped with persistent memory, which is used to store the image features of the first candidate image.

[0056] The aforementioned persistent memory refers to persistent memory, which supports byte-level addressing. Processors such as the CPU (Central Processing Unit) and GPU (Graphics Processing Unit) in electronic devices can directly access persistent memory and read data directly from it. For ease of description, it will be referred to hereafter as electronic devices being able to directly access memory and read data directly from it.

[0057] The first candidate image mentioned above is an image that has been obtained and is to be used for searching. It can be an image that has been obtained directly or captured by an image acquisition device, or it can be an image obtained from a publicly available image data platform. This application embodiment does not limit this.

[0058] The image features of each image can include its color features, edge features, content features, semantic features, etc. These image features can be represented in the form of vectors, matrices, etc.

[0059] In one embodiment of this application, when obtaining the image features of the first image, a preset algorithm can be used to extract the image features. This algorithm can be the Sobel algorithm, the Canney algorithm, the Laplace algorithm, etc.

[0060] Alternatively, the image can be input into a pre-trained feature extraction model to obtain the image features output by the model. This feature extraction model can be a convolutional neural network model, a deep neural network model, a recurrent neural network model, etc.

[0061] In addition, if the first candidate image is an image obtained from an image data platform, the features of each image can be obtained from the platform and used as the image features of the first candidate image.

[0062] like Figure 1 As shown, the above image search method includes the following steps S101-S103:

[0063] S101, Obtain the target image features of the image to be searched.

[0064] Specifically, image features of the image to be searched, such as color features, edge features, content features, and semantic features, can be obtained as target image features.

[0065] In one embodiment of this application, an image acquired by an image acquisition device can be obtained as a search image, and the image features of the search image can be extracted as target image features.

[0066] The aforementioned image acquisition devices can be cameras, video recorders, or mobile phones, tablets, laptops, etc., with image acquisition capabilities.

[0067] In addition, images sent by image forwarding devices can also be obtained as images to be searched. These forwarding devices can be NVRs, DVRs, computers, etc.

[0068] In one embodiment of this application, image features sent by the feature forwarding device can also be directly obtained as target image features of the image to be searched. The aforementioned feature forwarding device can be an NVR, DVR, computer, mobile phone, etc.

[0069] Specifically, the aforementioned feature forwarding device can obtain the image to be searched, extract the image features of the image to be searched, and forward the image features to the electronic device, so that the electronic device can obtain the target image features.

[0070] S102, obtain the search range. If there are images in the first candidate images whose search popularity is within the search range, directly access the persistent memory, obtain the image features of each first candidate image in the persistent memory, and calculate the similarity between the target image features and the image features of each first candidate image.

[0071] The search scope can be a time range, a source range, a content range, etc. For example, the search scope could be: images obtained in the past week, images captured by checkpoint cameras, images containing vehicles, etc.

[0072] Specifically, the search range can be obtained. If there are images among the first candidate images whose search popularity falls within the aforementioned search range, it indicates that similarity calculation needs to be performed on the image features of the first candidate images stored in persistent memory. In this case, the electronic device can directly access persistent memory to obtain the image features of each first candidate image stored in persistent memory. Then, for each first candidate image's image features, the similarity between that image feature and the aforementioned target image features can be calculated. This similarity reflects the similarity between the first candidate image and the image to be searched.

[0073] In one embodiment of this application, the Pearson coefficient, Euclidean distance, cosine similarity, Manhattan distance, etc., between the image features of each first candidate image and the target image features can be calculated as similarity representation values, and then the similarity between the image features of the first candidate image and the target image features can be determined based on the above similarity representation values.

[0074] Alternatively, the image features of each first candidate image and the target image features can be input into a pre-trained similarity model to obtain the similarity between the image features output by the model.

[0075] In one embodiment of this application, when obtaining the search range, the range input by the user through an external input device can be received as the search range. The external input device can be a mouse, keyboard, touchpad, voice acquisition device, etc.

[0076] In addition, preset search range options can be displayed to the user in the display interface to determine the user's selected option and thus obtain the search range.

[0077] S103, Based on the calculated similarity, determine the image that is similar to the image to be searched from the first candidate images.

[0078] Specifically, the similarity between the target image features calculated in S102 and the image features of each first candidate image can reflect the similarity between each first candidate image and the image to be searched. Based on the above similarity, images similar to the image to be searched can be searched from the first candidate images.

[0079] In one embodiment of this application, based on the calculated similarity, an image whose corresponding image features and target image features reach a preset similarity threshold can be selected from the first candidate images as an image similar to the image to be searched.

[0080] Alternatively, a predetermined number of frames with the highest similarity between the corresponding image features and the target image features can be selected from the first candidate images as images similar to the image to be searched. The predetermined number can be 1, 3, 5, etc., and this embodiment does not limit this.

[0081] In one embodiment of this application, the electronic device is further configured with a hard disk for storing first candidate images. Each first candidate image in the hard disk corresponds to a feature of an image in persistent memory. The hard disk can be a solid-state drive (SSD) or a hard disk drive (HDD), etc. The correspondence can be stored on the hard disk or in the memory of the electronic device.

[0082] In this way, when searching for images similar to the image to be searched, the image features whose similarity to the target image features meets the preset similarity conditions can be determined based on the calculated similarity and used as reference image features; according to the correspondence, the image corresponding to the reference image features is searched from the first candidate images stored on the hard disk.

[0083] The aforementioned similarity conditions can be either reaching a preset similarity threshold or the highest preset number of similarities.

[0084] Specifically, after calculating the similarity between the image features of the image to be searched and each first candidate image, image features that meet the above similarity conditions with the target image features can be selected from the image features of each first candidate image as reference image features. Then, using the above correspondence, the image corresponding to the above reference image features can be found in the hard disk as an image similar to the image to be searched.

[0085] The image search scheme provided in the above embodiments can be applied to electronic devices. The electronic device is equipped with persistent memory, which stores image features of first candidate images. The electronic device can obtain the target image features of the image to be searched; determine the search range; if there are images in the first candidate images whose search popularity falls within the search range, it directly accesses the persistent memory to obtain the image features of each first candidate image in the persistent memory, calculates the similarity between the target image features and the image features of each first candidate image; and, based on the calculated similarity, determines images similar to the image to be searched from the first candidate images. This allows searching for images similar to the image to be searched from the first candidate images based on the similarity between the image features of the first candidate images and the image features of the image to be searched. Since persistent memory supports byte-level addressing, image features can be directly read from persistent memory without copying them back to memory, thus saving resource consumption associated with data copying and improving image search efficiency. Therefore, applying the scheme provided in the above embodiments can improve image search efficiency.

[0086] In one embodiment of this application, the electronic device is further configured with volatile memory for storing image features of a second candidate image.

[0087] The aforementioned volatile memory allows electronic devices to directly read data, but it is difficult to retain data permanently. Data will be lost when the electronic device is powered off. The aforementioned volatile memory can be DRAM (Dynamic Random Access Memory), CPU Caches, CPU Registers, etc.

[0088] The aforementioned second candidate image is an image that has been obtained, different from the first candidate image, and is to be used for searching. It can be an image that has been directly obtained or acquired by an image acquisition device, or an image obtained from a publicly available image data platform. This application embodiment does not limit this.

[0089] Based on the above scheme, the image search scheme provided in this application embodiment may further include:

[0090] If there are images in the second candidate images whose search popularity is within the search range, directly access the volatile memory to obtain the image features of each second candidate image in the volatile memory, calculate the similarity between the target image features and the image features of each second candidate image; based on the calculated similarity, determine the image similar to the image to be searched from the second candidate images.

[0091] Specifically, after obtaining the search range, if there are images among the second candidate images whose search popularity falls within the aforementioned search range, it indicates that a similarity calculation needs to be performed on the image features of the second candidate images stored in volatile memory. In this case, the electronic device can directly access the volatile memory to obtain the image features of each second candidate image stored in the volatile memory. Then, for each image feature of the second candidate image, the similarity between that image feature and the aforementioned target image feature can be calculated. This similarity reflects the similarity between the second candidate image and the image to be searched.

[0092] The similarity between the target image features and the image features of each second candidate image can reflect the similarity between each second candidate image and the image to be searched. Based on the above similarity, images similar to the image to be searched can be searched from the second candidate images.

[0093] In one embodiment of this application, based on the calculated similarity, an image whose corresponding image features and target image features reach a preset similarity threshold can be selected from the second candidate images as an image similar to the image to be searched.

[0094] Alternatively, a preset number of frames can be selected from the second candidate images, showing the highest similarity between the corresponding image features and the target image features, as images similar to the image to be searched.

[0095] In one embodiment of this application, the Pearson coefficient, Euclidean distance, cosine similarity, Manhattan distance, etc., between the image features of each second candidate image and the target image features can be calculated as similarity representation values, and then the similarity between the image features of the second candidate image and the target image features can be determined based on the above similarity representation values.

[0096] Alternatively, the image features of each second candidate image and the target image features can be input into a pre-trained similarity model to obtain the similarity between the image features output by the model.

[0097] In one embodiment of this application, the search popularity of the second candidate image is higher than that of the first candidate image. The search popularity of each image reflects the likelihood that the image will be searched. The higher the search popularity of each image, the greater the likelihood that the image will be searched; otherwise, the lower the likelihood that the image will be searched.

[0098] In one embodiment of this application, the aforementioned search popularity can be described by at least one of the following information:

[0099] Information 1: The time when the image was acquired.

[0100] Specifically, the image acquisition time period can be determined based on the time the image was acquired. The acquisition time period for each image is defined as the time period in which the image was acquired. This time period can be a day, week, month, etc. Then, the search popularity of an image can be determined based on its acquisition time period. Images with more recent acquisition time periods have higher search popularity, while images with earlier acquisition time periods have lower search popularity. In this case, the image with the most recent acquisition time period can be used as a second candidate image, and the image features of this second candidate image can be stored in volatile memory.

[0101] For example, assuming the time period is a week, the image obtained in the previous week can be used as the second candidate image, and the image features of the second candidate image can be stored in volatile memory.

[0102] Correspondingly, images other than the most recent image in the time period can be used as first candidate images, and the image features of the first candidate images can be stored in persistent memory.

[0103] Information 2: The image content of the image.

[0104] Specifically, images containing preset image content can be used as second candidate images with high search popularity. The preset image content can be target personnel, target vehicles, target objects, etc. Target personnel can be security personnel, delivery personnel, food delivery workers, etc., who have pre-registered their identity information; target vehicles can be private cars with pre-registered vehicle identification; and target objects can be dangerous goods, such as knives, sticks, or other sharp objects.

[0105] For example, assuming the target image content is security personnel, the image containing security personnel can be used as the second candidate image, and the image features of the second candidate image can be stored in volatile memory.

[0106] Correspondingly, an image that does not contain the aforementioned preset image content can be used as a first candidate image, and the image features of the first candidate image can be stored in persistent memory.

[0107] Information 3: Image search hit frequency.

[0108] The search hit frequency of each image can be used to characterize how frequently the image is searched.

[0109] Specifically, the number of times each image is matched by a search can be counted, and the search frequency of that image can be obtained from this count. The higher the search frequency of an image, the higher its search popularity is considered; otherwise, the image is considered to have lower search popularity.

[0110] In one embodiment of this application, a preset number of frames with the highest search hit frequency can be used as second candidate images, and the image features of the second candidate images can be stored in volatile memory. Alternatively, images with a search hit frequency higher than a preset frequency threshold can be used as second candidate images, and the image features of the second candidate images can be stored in volatile memory.

[0111] Correspondingly, images whose search hit frequency is outside the aforementioned preset number can be used as first candidate images, or images whose search hit frequency is lower than a preset frequency threshold can be used as first candidate images, and the image features of the aforementioned first candidate images can be stored in persistent memory.

[0112] Information 4: Data source for the image.

[0113] Specifically, images whose data source is a preset source can be used as images with high search popularity. The preset source can be: a preset image data platform, image acquisition equipment deployed at the checkpoint, etc.

[0114] For example, assuming the preset source is an image acquisition device deployed at a traffic intersection, the image acquired by the image acquisition device deployed at the traffic intersection can be used as a second candidate image with high search popularity, and the image features of the second candidate image can be stored in volatile memory.

[0115] Correspondingly, images whose data source is other than the aforementioned preset source can be used as the first candidate images with low search popularity, and the image features of the aforementioned first candidate images can be stored in persistent memory.

[0116] Information 5: Application scenarios of images.

[0117] The application scenario for each image reflects its intended use. This use could include facial recognition, vehicle recognition, hazardous materials identification, license plate detection, etc.

[0118] Specifically, images from preset application scenarios can be used as images with high search popularity. These preset application scenarios can include: face recognition, license plate detection, etc.

[0119] For example, assuming the above-mentioned preset application scenario is face recognition, the image used for face recognition can be used as a second candidate image with high search popularity, and the image features of the second candidate image can be stored in volatile memory.

[0120] Correspondingly, images outside the aforementioned preset application scenarios can be used as the first candidate images with low search popularity, and the image features of the first candidate images can be stored in persistent memory.

[0121] Based on the above scheme, this application provides another image search method, which will be described in detail below.

[0122] See Figure 2 , Figure 2 A flowchart illustrating another image search method provided in an embodiment of this application, the method further comprising the following steps:

[0123] S201, Obtain the target image features of the image to be searched.

[0124] S202, obtain the search scope based on search popularity.

[0125] The search range mentioned above is a pre-defined range that is related to the search popularity of the image.

[0126] For example, assuming that the search popularity can be described by the time the image was acquired, the search scope can be: the range of the time the image was acquired, such as: images acquired within one month.

[0127] Assuming that the search popularity can be described by the image content, the search scope can be the range of content obtained for the image. For example, the search scope can be images whose content contains the target vehicle.

[0128] In one embodiment of this application, the execution order of step S104 may be before S101 obtaining the target image features, or after S101, or may be executed synchronously with S101. This embodiment of the application does not limit this.

[0129] S203, if there are images in the second candidate images whose search popularity is within the search range, directly access the volatile memory, obtain the image features of each second candidate image in the volatile memory, and calculate the similarity between the target image features and the image features of each second candidate image.

[0130] Specifically, if there are images in the second candidate images whose search popularity falls within the above search range, it means that it is necessary to search for images similar to the image to be searched from the second candidate images. Since the image features of the second candidate images are stored in volatile memory, the volatile memory can be accessed directly to obtain the image features of each second candidate image in the volatile memory, and the similarity between the target image features and the image features of each second candidate image can be calculated as the similarity between the image to be searched and each second candidate image.

[0131] For example, suppose the search popularity can be described by the time the image was acquired, the second candidate image is an image acquired within the previous week, and the search range is an image acquired within the previous month. In this case, there are images in the second candidate image whose search popularity falls within the above search range. Therefore, it is necessary to directly access the above volatile memory to obtain the image features of the second candidate image.

[0132] S204, Based on the calculated similarity, determine the image that is similar to the image to be searched from the second candidate images.

[0133] Specifically, the similarity between the target image features calculated in S105 and the image features of each second candidate image can reflect the similarity between each second candidate image and the image to be searched. Based on the above similarity, images similar to the image to be searched can be searched from the second candidate images.

[0134] S205, if there are images in the first candidate images whose search popularity is within the search range, directly access persistent memory, obtain the image features of each first candidate image in persistent memory, and calculate the similarity between the target image features and the image features of each first candidate image.

[0135] Specifically, if there are images in the first candidate images whose search popularity falls within the above search range, it means that it is necessary to search for images similar to the image to be searched from the first candidate images. Since the image features of the first candidate images are stored in persistent memory, the persistent memory can be accessed directly to obtain the image features of each first candidate image in the persistent memory. The similarity between the target image features and the image features of each first candidate image is calculated as the similarity between the image to be searched and each first candidate image.

[0136] For example, suppose the search popularity can be described by the image content of an image. The first candidate image is an image containing the target vehicle or dangerous goods, and the search range is an image containing dangerous goods. In this case, there are images in the first candidate image whose search popularity is within the search range. Therefore, it is necessary to directly access the persistent memory to obtain the image features of the first candidate image.

[0137] S206, Based on the calculated similarity, determine the image that is similar to the image to be searched from the first candidate images.

[0138] In one embodiment of this application, if both the first candidate image and the second candidate image contain images whose search popularity is within the search range, the persistent memory and volatile memory are directly accessed to obtain the image features of each first candidate image and the second candidate image in the persistent memory and the volatile memory. Then, the similarity between each obtained image feature and the target image feature is calculated. Finally, based on the calculated similarity, the image similar to the image to be searched is determined from the first candidate image and the second candidate image.

[0139] In the solution provided by the above embodiments, images with high search popularity can be used as second candidate images. The number of second candidate images with high search popularity is usually small, so the data volume of the image features of the second candidate images is also small. Therefore, the image features of the second candidate images can be stored in volatile memory.

[0140] Furthermore, images with low search popularity can be used as first candidate images. Since there are usually many first candidate images with low search popularity, the data volume of their image features is also large. Therefore, the image features of these first candidate images can be stored in volatile memory. In this way, when obtaining image features, they can be retrieved from the aforementioned volatile memory and / or persistent memory according to a preset search range.

[0141] Both the aforementioned volatile and persistent memory support direct data reading by electronic devices. This eliminates the need to copy image features from volatile and / or persistent memory to main memory, saving resource consumption associated with data copying and improving image search efficiency. Furthermore, persistent memory is typically less expensive, allowing electronic devices to deploy larger amounts of persistent memory and store the image features of the largest candidate image in it, thus reducing costs.

[0142] In one embodiment of this application, the persistent memory operates in AD (APP Direct) mode, wherein persistent memory has persistent non-volatile properties when storing data in AD mode.

[0143] Specifically, persistent memory can operate in two modes: Memory mode and AD mode. In the Memory mode, persistent memory is volatile and cannot retain data permanently. In this mode, persistent memory will lose data when the electronic device is powered off.

[0144] In AD mode, persistent memory can persistently store data, so the data in persistent memory will not be lost even if the electronic device is powered off.

[0145] In the above scheme, the working mode of the persistent memory can be set to AD mode, which can persistently store the image features of the first candidate image in the persistent memory, thereby improving the reliability of data storage.

[0146] The image search scheme described above will be described again below using specific embodiments.

[0147] In one embodiment of this application, the aforementioned electronic device may be a computing node in a distributed system, for example, it may be a master node or a slave node.

[0148] See Figure 3 , Figure 3 This is a schematic diagram of the structure of a distributed system provided in an embodiment of this application. Figure 3 As shown, the system includes one master node and three slave nodes. Each computing node is configured with volatile memory (DRAM) and persistent memory (PM). The DRAM can be used to store image features of the second candidate image, and the PM can be used to store image features of the first candidate image. Each computing node is used for image feature storage and similarity calculation, specifically:

[0149] Each computing node is used to obtain the target image features of the image to be searched and to obtain the search range based on the search popularity. It determines whether there are images in the second candidate images whose search popularity is within the search range. If so, it directly accesses the DRAM to obtain the image features of each second candidate image in the DRAM and calculates the similarity between the target image features and the image features of each second candidate image.

[0150] Furthermore, it also determines whether there are images in the first candidate images whose search popularity is within the search range. If so, it directly accesses the PM to obtain the image features of each first candidate image in the PM and calculates the similarity between the target image features and the image features of each first candidate image.

[0151] After calculating the similarity, images similar to the image to be searched can be determined from the second candidate image and / or the first candidate image based on the calculated similarity.

[0152] The image search scheme provided in the above embodiments can be applied to electronic devices. The electronic device is equipped with persistent memory, which stores image features of first candidate images. The electronic device can obtain the target image features of the image to be searched; determine the search range; if there are images in the first candidate images whose search popularity falls within the search range, it directly accesses the persistent memory to obtain the image features of each first candidate image in the persistent memory, calculates the similarity between the target image features and the image features of each first candidate image; and, based on the calculated similarity, determines images similar to the image to be searched from the first candidate images. This allows searching for images similar to the image to be searched from the first candidate images based on the similarity between the image features of the first candidate images and the image features of the image to be searched. Since persistent memory supports byte-level addressing, image features can be directly read from persistent memory without copying them back to memory, thus saving resource consumption associated with data copying and improving image search efficiency. Therefore, applying the scheme provided in the above embodiments can improve image search efficiency.

[0153] See Figure 4 , Figure 4 This is a schematic diagram of an image search device provided in an embodiment of this application. The device is disposed in an electronic device, which is equipped with persistent memory for storing image features of a first candidate image. The device includes:

[0154] The feature acquisition module 401 is used to acquire the target image features of the image to be searched.

[0155] Range acquisition module 402 is used to obtain the search range;

[0156] The first similarity calculation module 403 is used to directly access the persistent memory if there is an image in the first candidate image whose search popularity is within the search range, obtain the image features of each first candidate image in the persistent memory, and calculate the similarity between the target image features and the image features of each first candidate image.

[0157] The first image search module 404 is used to determine, based on the calculated similarity, an image similar to the image to be searched from the first candidate images.

[0158] In one embodiment of this application, the electronic device is further configured with volatile memory, which is used to store image features of a second candidate image;

[0159] The device further includes:

[0160] The second similarity calculation module is used to directly access the volatile memory if there is an image in the second candidate image whose search popularity is within the search range, obtain the image features of each second candidate image in the volatile memory, and calculate the similarity between the target image features and the image features of each second candidate image.

[0161] The second image search module is used to determine images similar to the image to be searched from the second candidate images based on the calculated similarity.

[0162] In one embodiment of this application, the search popularity of the second candidate image is higher than that of the first candidate image, and the search popularity is described by at least one of the following:

[0163] Image acquisition time, image content, image search hit frequency, image data source, and image application scenario;

[0164] The range acquisition module 402 includes:

[0165] Obtain the search scope based on search popularity.

[0166] In one embodiment of this application, the persistent memory operates in AD mode, wherein the persistent memory stores data with persistent non-volatility in AD mode.

[0167] In one embodiment of this application, the electronic device is further configured with a hard disk for storing the first candidate image, and there is a correspondence between each first candidate image in the hard disk and each image feature in the persistent memory;

[0168] The first image search module 404 is specifically used for:

[0169] Based on the calculated similarity, image features whose similarity to the target image features meets preset similarity conditions are determined and used as reference image features;

[0170] According to the correspondence, the image corresponding to the reference image feature is searched from the first candidate images stored on the hard disk.

[0171] The image search scheme provided in the above embodiments can be applied to electronic devices. The electronic device is equipped with persistent memory, which stores image features of first candidate images. The electronic device can obtain the target image features of the image to be searched; obtain the search range; if there are images in the first candidate images whose search popularity falls within the search range, it directly accesses the persistent memory to obtain the image features of each first candidate image in the persistent memory, calculates the similarity between the target image features and the image features of each first candidate image; and, based on the calculated similarity, determines images similar to the image to be searched from the first candidate images. This allows searching for images similar to the image to be searched from the first candidate images based on the similarity between the image features of the first candidate images and the image features of the image to be searched. Since persistent memory supports byte-level addressing, image features can be directly read from persistent memory without copying them back to memory, thus saving resource consumption associated with data copying and improving image search efficiency. Therefore, applying the scheme provided in the above embodiments can improve image search efficiency.

[0172] This application also provides an electronic device, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0173] Memory 503 is used to store computer programs;

[0174] The processor 501 is used to implement the steps of the image search method when executing the program stored in the memory 503.

[0175] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0176] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0177] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0178] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0179] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described image search methods.

[0180] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the image search methods described above.

[0181] The image search scheme provided in the above embodiments can be applied to electronic devices. The electronic device is equipped with persistent memory, which stores image features of first candidate images. The electronic device can obtain the target image features of the image to be searched; obtain the search range; if there are images in the first candidate images whose search popularity falls within the search range, it directly accesses the persistent memory to obtain the image features of each first candidate image in the persistent memory, calculates the similarity between the target image features and the image features of each first candidate image; and, based on the calculated similarity, determines images similar to the image to be searched from the first candidate images. This allows searching for images similar to the image to be searched from the first candidate images based on the similarity between the image features of the first candidate images and the image features of the image to be searched. Since persistent memory supports byte-level addressing, image features can be directly read from persistent memory without copying them back to memory, thus saving resource consumption associated with data copying and improving image search efficiency. Therefore, applying the scheme provided in the above embodiments can improve image search efficiency.

[0182] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0183] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0184] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments, electronic device embodiments, computer-readable storage medium embodiments, and computer program product embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0185] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An image search method, characterized in that, The method is applied to an electronic device, which is a computing node in a distributed system. The computing nodes in the distributed system include master nodes and slave nodes. Each node is configured with persistent memory and volatile memory. The persistent memory supports the electronic device in directly reading data in a byte-addressable manner and is used to store image features of a first candidate image. The persistent memory operates in AD mode, where the data stored in the persistent memory is persistent and non-volatile. The volatile memory is used to store image features of a second candidate image, where the search popularity of the second candidate image is higher than that of the first candidate image. The search popularity of each image reflects the likelihood of the image being searched. The search popularity is described by at least one of the following: image acquisition time, image content, image search hit frequency, image data source, and image application scenario. The method includes: Obtain the target image features of the image to be searched; wherein, the target image features include: color features, edge features, content features and / or semantic features in the image; A search range based on search popularity is obtained. If there is an image among the first candidate images whose search popularity falls within the search range, the persistent memory is accessed directly using byte-addressing to obtain the image features of each first candidate image in the persistent memory. The similarity between the target image features and the image features of each first candidate image is calculated. The search range is obtained by: receiving a range input by the user through an external input device as the search range; or, displaying preset search range options to the user in the display interface, determining the option selected by the user, and obtaining the search range. Based on the calculated similarity, images similar to the image to be searched are determined from the first candidate images; If there is an image in the second candidate image whose search popularity is within the search range, directly access the volatile memory, obtain the image features of each second candidate image in the volatile memory, and calculate the similarity between the target image features and the image features of each second candidate image; Based on the calculated similarity, images similar to the image to be searched are determined from the second candidate images.

2. The method according to claim 1, characterized in that, The electronic device is also equipped with a hard disk for storing the first candidate images, and there is a correspondence between each first candidate image in the hard disk and the image features belonging to that first candidate image in the persistent memory; The step of determining images similar to the image to be searched from the first candidate images based on the calculated similarity includes: Based on the calculated similarity, image features whose similarity to the target image features meets preset similarity conditions are determined and used as reference image features; According to the correspondence, the image corresponding to the reference image feature is searched from the first candidate images stored on the hard disk.

3. An image search device, characterized in that, The device is installed in an electronic device, which is a computing node in a distributed system. The computing nodes in the distributed system include master nodes and slave nodes. Each node is configured with persistent memory and volatile memory. The persistent memory supports the electronic device in directly reading data in a byte-addressable manner and is used to store image features of a first candidate image. The persistent memory operates in AD mode, where the data stored in the persistent memory is persistent and non-volatile. The volatile memory is used to store image features of a second candidate image. The search popularity of the second candidate image is higher than that of the first candidate image. The search popularity of each image reflects the likelihood of the image being searched. The search popularity is described by at least one of the following: image acquisition time, image content, image search hit frequency, image data source, and image application scenario. The device includes: The feature acquisition module is used to obtain target image features of the image to be searched; wherein, the target image features include: color features, edge features, content features and / or semantic features in the image; The range acquisition module is used to obtain the search range based on search popularity; The first similarity calculation module is used to, if there are images in the first candidate images whose search popularity falls within the search range, directly access the persistent memory in a byte-addressing manner to obtain the image features of each first candidate image in the persistent memory, and calculate the similarity between the target image features and the image features of each first candidate image; wherein, the search range is obtained by: receiving a range input by the user through an external input device as the search range; or, displaying preset search range options to the user in the display interface, determining the option selected by the user, and obtaining the search range; The first image search module is used to determine images similar to the image to be searched from the first candidate images based on the calculated similarity. The second similarity calculation module is used to directly access the volatile memory if there is an image in the second candidate image whose search popularity is within the search range, obtain the image features of each second candidate image in the volatile memory, and calculate the similarity between the target image features and the image features of each second candidate image. The second image search module is used to determine images similar to the image to be searched from the second candidate images based on the calculated similarity.

4. The apparatus according to claim 3, characterized in that, The electronic device is also equipped with a hard disk for storing the first candidate images, and there is a correspondence between each first candidate image in the hard disk and the image features belonging to that first candidate image in the persistent memory; The first image search module is specifically used for: Based on the calculated similarity, image features whose similarity to the target image features meets preset similarity conditions are determined and used as reference image features; According to the correspondence, the image corresponding to the reference image feature is searched from the first candidate images stored on the hard disk.

5. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-2.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-2.

Images