Image retrieval method and system

Abstract

The invention provides an image retrieval method and system, and the method comprises the steps: obtaining a to-be-retrieved image, carrying out the feature similarity comparison of the to-be-retrieved image and an image in a pre-constructed database, and obtaining and outputting a retrieval result corresponding to the to-be-retrieved image. Wherein the images in the database comprise real shot images and rendered images corresponding to the plurality of objects, and the rendered images are image variants generated by performing data enhancement processing on the real shot images. Data enhancement processing is carried out on the real shot image in the database to enrich the content of the database, so that the accuracy and reliability of image retrieval are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image retrieval method and system. Background Art

[0002] Image retrieval is an important research direction in the field of computer vision and information retrieval, which aims to find images similar or related to the query image from a large number of image databases. With the rapid development of the Internet and digital media, image retrieval technology has been widely used in search engines, e-commerce, social media, medical image analysis and other fields.

[0003] In the related art, a database is usually constructed first, which includes real-shot images corresponding to a number of objects. Accordingly, when there is a retrieval requirement, the image retrieval system compares the image corresponding to the retrieval requirement with the real-shot images in the database to obtain and output the retrieval results.

[0004] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0005] This specification provides an image retrieval method and system to avoid at least one of the above technical problems.

[0006] In a first aspect, this specification provides an image retrieval method, comprising: Obtain the image to be retrieved; Compare the image to be retrieved with images in a pre-built database for feature similarity, and obtain and output a retrieval result corresponding to the image to be retrieved; The images in the database include real-shot images and rendered images corresponding to a number of objects, and the rendered images are image variants generated by performing data enhancement processing on the real-shot images.

[0007] In a second aspect, this specification provides an image retrieval system, comprising: at least one storage medium storing at least one instruction set to perform image retrieval; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, it reads the at least one instruction set and executes the method as described in the first aspect according to the instructions of the at least one instruction set.

[0008] In a third aspect, the present specification provides a computer-readable non-temporary storage medium, wherein the computer-readable non-temporary storage medium stores at least one instruction set, and the at least one instruction set is executed by at least one processor to implement the method described in the first aspect.

[0009] As can be seen from the above technical solutions, the image retrieval method and system provided in this specification enrich the content of the database by performing data enhancement processing on the real-shot images in the database. Accordingly, when the image to be retrieved transmitted by the user is obtained, the image retrieval system will compare the feature similarity of the image to be retrieved with the image in the enriched database to obtain and output the retrieval results. This can significantly improve the performance of the image retrieval system when dealing with different angles, lighting and posture changes. This allows users to find the required retrieval results more quickly and accurately.

[0010] Other functions of the image retrieval method and system provided in this specification will be partially listed in the following description. The creative aspects of the image retrieval method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram of the principle of an image retrieval method provided in one embodiment of this specification; Figure 2 A schematic diagram of the principle of an image retrieval method provided by another embodiment of this specification; Figure 3 A schematic diagram of an application scenario of the image retrieval method provided in the embodiments of this specification; Figure 4 A schematic diagram of the structure of an image retrieval system provided in an embodiment of this specification; Figure 5 A flowchart of an image retrieval method provided in an embodiment of this specification; Figure 6 A schematic diagram of the principle of an image retrieval method provided by another embodiment of this specification; Figure 7 A schematic diagram of the principle of an image retrieval method provided by another embodiment of this specification; Figure 8 A schematic diagram of the principle of obtaining a rendered image provided in an embodiment of this specification. DETAILED DESCRIPTION

[0013] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0014] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0015] In the embodiments of this specification, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0016] The term "plurality" in the embodiments of the present specification refers to two or more than two, and other quantifiers are similar thereto.

[0017] The terms "first", "second", "third", etc. in this specification are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this specification.

[0018] The term "unit / module" as used in this specification refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0019] To facilitate readers' understanding of this specification, the application scenarios of this specification are now introduced.

[0020] The technical solution provided in this specification is applicable to scenarios where images need to be retrieved and the retrieval results need to be fed back. For example, the technical solution provided in this specification can be applied to image search scenarios, image recognition scenarios (such as face recognition, etc.), content recommendation scenarios, etc., which will not be listed here one by one.

[0021] Take the above image search scenario as an example: In one example, the image retrieval system may be a server, etc. When a user needs to retrieve a target object, such as when the user wants to learn relevant information such as the name of a plant, the user may use a user device (such as a mobile phone) to shoot and send the image to be retrieved to the image retrieval system. The image retrieval system may execute the image retrieval method provided in this specification, determine and feedback information such as the name of the plant in the image to be retrieved to the user.

[0022] In another example, the image retrieval system may be a shopping platform, such as a server of the shopping platform. The user transmits the image to be retrieved of the item that the user wishes to purchase to the image retrieval system through the user device. The image retrieval system may execute the image retrieval method provided in this specification, determine and display to the user through the user device a page including the item that the user wishes to purchase.

[0023] It is worth noting that the above examples are only used to illustrate the application scenarios to which the technical solutions of this specification can be applied, and in different application scenarios, the execution subjects of the technical solutions provided in this specification may execute, and they cannot be understood as limitations on application scenarios and execution subjects.

[0024] In practical applications, a database is usually constructed first, which includes real-shot images corresponding to several objects. The number can be one or more, and is usually more than one. The objects may be different for different application scenarios. For example, in combination with the above example, the object may be a plant or a commodity.

[0025] When there is a retrieval demand, such as when the image retrieval system obtains the image to be retrieved transmitted by the user, the image retrieval system will compare the feature similarity of the image to be retrieved with the real-shot images in the database. If the image retrieval system can match a real-shot image in the database that is sufficiently similar to the image to be retrieved, the name and related information of the real-shot image in the database that is sufficiently similar will be returned as the retrieval result. For example, Figure 1 As shown: The database includes three real-shot images, namely, real-shot image A of object a, real-shot image B of object b, and real-shot image C of object c. The images uploaded by users to the image retrieval system are images to be retrieved.

[0026] The image retrieval system compares the feature similarity of the image to be retrieved with the real-shot image A, the real-shot image B, and the real-shot image C respectively to obtain the corresponding similarities, and can determine the similarity score based on the similarities. Figure 1 As shown, the similarity score between the image to be retrieved and the real image A is 85 points; the similarity score between the image to be retrieved and the real image B is 60 points; and the similarity score between the image to be retrieved and the real image C is 65 points.

[0027] The image retrieval system determines the highest similarity score from the three similarity scores, that is, a similarity score of 85. The image retrieval system returns the name a of the real-shot image A having the highest similarity score of 85 as a retrieval result.

[0028] However, the above solution is easily affected by the angle, lighting, etc. when the user takes the photo, resulting in inaccurate feature similarity comparison, which in turn leads to retrieval failure or retrieval error.

[0029] For example, combining the above examples and Figure 2 Due to the angle at which the user takes the photo, the image to be retrieved transmitted by the user to the image retrieval system is as follows: Figure 2 The image retrieval system compares the feature similarity between the image to be retrieved and the real-shot image A, real-shot image B, and real-shot image C in the database, and the similarity scores obtained are 60 points, 53 points, and 45 points respectively.

[0030] Since the highest similarity score is only 60 points, which is too low, the image retrieval system determines and feeds back the failure result to the user as "unable to retrieve". That is, the image retrieval system determines through retrieval that there is no real image in the database that actually corresponds to the image to be retrieved.

[0031] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification.

[0032] Therefore, in order to avoid the above technical problems, this specification proposes a technical concept through creative labor: data enhancement processing is performed on the real-shot images in the database to enrich the content in the database. For example, data enhancement processing of multi-view, multi-illumination, multi-pose, etc. is performed on the real-shot images in the database to generate a series of new image variants, and the image variants are added to the database for feature similarity comparison with the images to be retrieved transmitted by the user, so as to finally obtain and output the retrieval results.

[0033] Based on the above analysis, it can be seen that the above technical solution enriches the content of the database by performing data enhancement processing on the real-shot images in the database. Accordingly, when obtaining the image to be retrieved transmitted by the user, the image retrieval system will compare the feature similarity of the image to be retrieved with the image in the enriched database, obtain and output the retrieval result. This can significantly improve the performance of the image retrieval system when dealing with different angles, lighting and posture changes. This allows users to find the required retrieval results more quickly and accurately.

[0034] Figure 3FIG. 1 is a schematic diagram of an application scenario of the image retrieval method of an embodiment of the present specification. The image retrieval method of the present specification can be applied to Figure 3 The application scenario 300 shown. Figure 3 As shown, the application scenario 300 may include a target user 301 , a client 302 , a server 303 , and a network 304 .

[0035] The target user 301 may be a user who triggers the retrieval of the image to be retrieved. For example, the target user 301 may perform a target operation on the client 302 to trigger the retrieval of the image to be retrieved. In some embodiments, the target user 301 may upload the image to be retrieved on the client 302 to trigger the image retrieval system to retrieve the image to be retrieved.

[0036] The client 302 may be an electronic device that provides interactive functions to the target user 301. For example, the client 302 may provide an interactive interface to the target user 301, and the target user 301 may perform interactive operations in the interactive interface. In some embodiments, the client 302 executes the image retrieval method described in this specification in response to detecting a retrieval operation of an image to be retrieved triggered by the target user 301. At this time, the client 302 may store data or instructions for executing the image retrieval method described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the client 302 may include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work, so as to execute the image retrieval method described in this specification.

[0037] In some embodiments, the client 302 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, or the like, or any combination thereof. In some embodiments, the built-in device in a motor vehicle may include an onboard computer, an onboard TV, or the like. In some embodiments, the client 302 may include a capture device for capturing images to be retrieved.

[0038] In some embodiments, the client 302 may be installed with one or more applications (Application, APP). APP can provide the target user 301 with the ability and interface to interact with the outside world through the network 304. APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc. In some embodiments, the client 302 may be installed with a target APP. The target APP can collect images to be retrieved for the client 302.

[0039] like Figure 3 As shown, the client 302 can be connected to the server 303 for communication. The server 303 can be connected to one client 302 for communication, or can be connected to multiple clients 302 for communication. In some embodiments, the client 302 can interact with the server 303 through the network 304 to receive or send messages, etc. For example, the client 302 can interact with the server 303 through the network 304 to send a target logistics order to the server 303.

[0040] The server 303 may be a server that provides various services. For example, the server 303 may be a cloud server or a local server. The server 303 may be connected to a client 302 for communication and receive data sent by the client 302, or may be connected to multiple clients 302 for communication and receive data sent by each client 302.

[0041] In some embodiments, the image retrieval method described in this specification can be executed on the server 303. In this case, the server 303 can store data or instructions for executing the image retrieval method described in this specification, and can execute or be used to execute the data or instructions. The server 303 can include a hardware device with a data information processing function and the necessary programs required to drive the hardware device to work.

[0042] The network 304 is a medium for providing a communication connection between the client 302 and the server 303. The network 304 can facilitate the exchange of information or data. Figure 3 As shown, the client 302 and the server 303 can be connected to the network 304 respectively, and transmit information or data to each other through the network 304.

[0043] In some embodiments, the network 304 may be any type of wired or wireless network, or a combination thereof. For example, the network 304 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network TM, a short-range wireless network (ZigBee TM), a near field communication (NFC) network, or a similar network.

[0044] In some embodiments, network 304 may include one or more network access points. For example, network 304 may include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of client 302 and server 303 may connect to network 304 to exchange data or information.

[0045] It is worth mentioning that Figure 3 The number of the client 302, server 303 and network 304 is only illustrative. According to the implementation requirements, there can be any number of clients 302, servers 303 and networks 304. The image retrieval method provided in this specification can be executed completely on the client 302, can be executed completely on the server 303, or can be executed partially on the client 302 and partially on the server 303.

[0046] That is to say, Figure 3 and for Figure 3 The above description is only used to illustrate the application scenarios to which the image retrieval method of this specification may be applicable, and should not be understood as a limitation on the application scenarios.

[0047] Figure 4The hardware structure diagram of an image retrieval system 400 provided according to an embodiment of the present specification is shown. The image retrieval system 400 can execute the image retrieval method described in the present specification. The image retrieval method is introduced in other parts of the present specification. When the image retrieval method is executed on the client 302, the image retrieval system 400 can be the client 302. When the image retrieval method is executed on the server 303, the image retrieval system 400 can be the server 303. When the image retrieval method is partially executed on the client 302 and partially executed on the server 303, the image retrieval system 400 can be a system including the client 302 and the server 303.

[0048] like Figure 4 As shown, the image retrieval system 400 may include at least one storage medium 403 and at least one processor 402. In some embodiments, the image retrieval system 400 may further include a communication port 404 and an internal communication bus 401. The image retrieval system 400 may further include an I / O component 405.

[0049] The internal communication bus 401 can connect different system components. For example, the internal communication bus 401 can connect the storage medium 403, the processor 402, the communication port 404 and the I / O component 405.

[0050] The I / O component 405 supports input / output between the image retrieval system 400 and other components.

[0051] The communication port 404 is used for data communication between the image retrieval system 400 and the outside world. For example, the communication port 404 can be used for data communication between the image retrieval system 400 and the network 304. The communication port 404 can be a wired communication port or a wireless communication port.

[0052] The storage medium 403 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 4031, a read-only storage medium (ROM) 4032, or a random access storage medium (RAM) 4033. The storage medium 403 also includes at least one instruction set stored in the data storage device. The instruction set includes a computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the image retrieval method provided in this specification.

[0053] At least one processor 402 may be communicatively connected to at least one storage medium 403. At least one processor 402 is used to execute the above-mentioned at least one instruction set. When the image retrieval system 400 is running, at least one processor 402 reads the at least one instruction set and executes the image retrieval method provided in this specification according to the instructions of the at least one instruction set. The processor 402 may execute all the steps included in the image retrieval method. The processor 402 may be in the form of one or more processors. In some embodiments, the processor 402 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0054] For illustration purposes only, only one processor 402 is shown in the image retrieval system 400 in the accompanying drawings. However, it should be noted that the image retrieval system 400 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or by multiple processors in combination. For example, if the processor 402 of the image retrieval system 400 is described in this specification as performing step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 402 (for example, the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0055] See also Figure 5 , Figure 5 The following is a flow chart of the image retrieval method provided in the embodiment of this specification. Figure 5 As shown, the method includes the following S501 and S502: S501: Obtain an image to be retrieved.

[0056] As can be seen from the above examples, the image to be retrieved can be understood as an image that the image retrieval system needs to retrieve in order to obtain a retrieval result.

[0057] For example, for different application scenarios, the content of the images to be retrieved is different. For the plant retrieval scenario, the images to be retrieved are images of the plants to be retrieved; for the shopping scenario, the images to be retrieved are images of the goods to be purchased.

[0058] This embodiment does not limit the manner in which the image retrieval system obtains the image to be retrieved.

[0059] In one example, the image retrieval system may be connected to an image acquisition device and receive images sent by the image acquisition device. For example, the image acquisition device may be a camera of a user device. The image retrieval system may be a server or a processor.

[0060] In another example, the image retrieval system may provide a tool for loading images to be retrieved, and the user may transfer the images to be retrieved to the image retrieval system through the tool.

[0061] Among them, the tool for loading the image to be retrieved can be an interface for connecting to an external device, such as an interface for connecting to other storage devices, through which the image to be retrieved transmitted by the external device is obtained; the tool for loading the image to be retrieved can also be a display device, such as the image retrieval system can output an interface for the function of loading the image to be retrieved on the display device, and the user can import the image to be retrieved into the image retrieval system through this interface.

[0062] S502: performing feature similarity comparison between the image to be retrieved and the images in the pre-built database, and obtaining and outputting a retrieval result corresponding to the image to be retrieved.

[0063] The images in the database include real-shot images and rendered images corresponding to a number of objects, and the rendered images are image variants generated by performing data enhancement processing on the real-shot images.

[0064] Exemplarily, in this embodiment, the images in the database can be divided into two categories: real-shot images obtained by actual shooting, and rendered images obtained by performing data enhancement processing on the real-shot images.

[0065] Combination Figure 1 and Figure 6 It can be seen that in the related art, the database only includes real-shot images. In this embodiment, the database includes not only real-shot images, but also rendered images obtained by transforming the real-shot images.

[0066] like Figure 6 As shown, the database includes not only the real-shot image A of the object a, but also the rendered images A1, A2, and A3 obtained by performing data enhancement processing on the real-shot image.

[0067] Correspondingly, the image retrieval system compares the feature similarity of the image to be retrieved with the images in the database (including real-shot images and rendered images) to obtain and output the retrieval results.

[0068] Based on the above analysis of S501 and S502, it can be seen that in this embodiment, the database is a database after data enhancement processing, which includes not only real-shot images, but also rendered images obtained by transforming real-shot images. That is, the database of this embodiment is an enriched database. Therefore, when performing feature similarity comparison based on images in the enriched database to determine the retrieval results, the reliability, effectiveness, and comprehensiveness of the feature similarity comparison can be improved, thereby improving the accuracy and reliability of image retrieval.

[0069] This embodiment does not limit the method of data enhancement processing. Exemplarily, the rendered image is obtained by performing data enhancement processing on the real-shot image using multimedia generation technology; the multimedia generation technology includes one of the following technologies: 3D Artificial Intelligence Generated Content (3D AIGC) technology, Variational Autoencoder (VAE) technology, and Generative Adversarial Network (GAN) technology.

[0070] Among them, three-dimensional artificial intelligence content generation technology can be understood as the technology of using artificial intelligence technology to generate three-dimensional content, such as 3D models, scenes and animations.

[0071] The variational autoencoder technology can be understood as a technology that generates new data samples by learning the potential distribution of data. For example, based on the variational autoencoder technology, images with different styles from real images can be generated to increase the diversity of data in the database.

[0072] Generative adversarial network technology can be understood as a technology that generates new data samples through adversarial training. For example, based on generative adversarial network technology, images that are very close to real images but have different styles can be generated to increase the diversity of data in the database.

[0073] In this embodiment, by using the above multimedia generation technology to perform data enhancement processing on the real-shot images, high-quality and diversified rendering images can be generated, thereby improving the accuracy and efficiency of the image retrieval system and improving the user experience.

[0074] In some embodiments, the data enhancement processing includes transforming the real-shot image based on at least one of different viewing angles, different lighting conditions, and different postures to obtain a rendered image.

[0075] Combined with the above analysis, the database includes both real-shot images and rendered images. Specifically, the content of the database is enriched with rendered images. The rendered images are image variants of the real-shot images, that is, images obtained by transforming the real-shot images.

[0076] In one example, the rendered image may be obtained by transforming the real image based on different viewing angles. Figure 1 As shown in , the real shot image is a front shot of object a. Figure 6 As shown, the rendered image A1 is a side rendering of the object a.

[0077] In another example, the rendered image may be obtained by transforming the real-shot image based on different lighting conditions, for example, by adjusting the brightness, contrast, color saturation, etc. of the real-shot image.

[0078] In another example, the rendered image may be obtained by transforming the real-shot image based on different postures, for example, by adjusting the posture of the object in the real-shot image by rotation, flipping, affine transformation, etc. to obtain the rendered image.

[0079] It is worth noting that the database may be constructed by adopting one of the above data enhancement processing methods, or may be constructed by adopting multiple of the above data enhancement processing methods, which is not limited in this embodiment.

[0080] In this embodiment, by determining the rendered image in a data enhancement processing manner from different viewing angles, different lighting conditions, and different postures, the retrieval effect under different angles, different lighting conditions, and different postures can be improved, that is, the accuracy and reliability of image retrieval can be improved.

[0081] In some embodiments, generating a rendered image includes the following steps 1 and 2: Step 1: Convert the two-dimensional (2D) real-life image into three-dimensional (3D) structural information.

[0082] Exemplarily, the real-shot image is a 2D image. The 2D image can be converted into three-dimensional structure information. The three-dimensional structure information can constitute a 3D model and a texture (if any).

[0083] This embodiment does not limit the method of converting the 2D image into a 3D model. For example, the 3D model can be reconstructed based on 3D artificial intelligence (AI).

[0084] For example, to facilitate distinction, Figure 7 As shown in Figure 2, the database before data enhancement is the original database, and the database after data enhancement is the enhanced database. Figure 1 and Figure 2 It can be seen that the original database includes real images of different objects. Figure 7 As shown, 3D AI reconstruction can be performed one by one for the real-shot images in the original database.

[0085] In some embodiments, step 1 may include: performing subject recognition, segmentation and cutout, and three-dimensional reconstruction on the real-shot image in sequence to obtain three-dimensional structure information.

[0086] Among them, subject recognition can be understood as identifying the object or subject of interest from the image. Segmentation and cutout can be understood as extracting the precise outline of the subject from the image, removing the background, and generating a clean subject image. 3D reconstruction can be understood as restoring the 3D structural information of the object from the image to generate a 3D model.

[0087] Accordingly, in this embodiment, if Figure 8 As shown in FIG, after subject recognition is performed on the 2D real-shot image, segmentation and cutout can be performed, and then 3D AI reconstruction can be performed to obtain three-dimensional structure information. Figure 8 3D models and textures shown in .

[0088] In this embodiment, by sequentially performing subject recognition, segmentation and cutout, and 3D reconstruction on the real-shot image, effective and reliable 3D structural information can be obtained, thereby improving the accuracy and reliability of the subsequent rendered image.

[0089] Step 2: Rendering the three-dimensional structure information into a rendered image in a simulated environment, wherein the simulated environment includes at least one of simulating different viewing angles, simulating different lighting conditions, and simulating different postures.

[0090] Exemplarily, the three-dimensional structure information can be rendered into a new 2D image (i.e., a rendered image) based on a 3D rendering engine. Figure 6 , Figure 7 , Figure 8 As shown, the 3D rendering engine can simulate different viewing angles, different lighting conditions, and different postures to generate diverse 2D rendering images.

[0091] Combining the above analysis of step 1 and step 2, it can be seen that in this embodiment, by converting the 2D real-life image into 3D structural information, such as a 3D model and a texture, the rendered images in different simulation environments are rendered based on the 3D structural information. The reliability and diversity of the rendered images can be improved, so that the rendered images cover a wider range of application scenarios. Thus, the effectiveness and accuracy of image retrieval can be improved.

[0092] In some embodiments, step 2 may include the following steps 21 and 22: Step 21: Render the three-dimensional structure information as an image to be screened in a simulated environment.

[0093] The images to be screened can be understood as all images obtained by rendering, and are used to select images with relatively high quality to be added to the database.

[0094] Step 22: Evaluate the quality information of the images to be screened, and screen the rendered images from the images to be screened based on the quality information, wherein the quality information includes at least one of light intensity, picture quality, and similarity, and the quality of the rendered image is better than other images in the images to be screened.

[0095] For example, there is a close relationship between image quality and light intensity, which can be understood from multiple perspectives: exposure, color reproduction, noise, dynamic range, contrast, detail retention, etc. Take exposure as an example: Correct exposure is the basis for ensuring image quality. Appropriate light intensity can fully display the details in the image, with moderate color saturation and good contrast. For example, when the light intensity is too high, the image may be overexposed, that is, the highlights lose details and become white. Overexposure not only causes the loss of image information, but also reduces the overall quality of the image. For another example, when the light intensity is insufficient, the image may be underexposed, that is, the dark areas become too dark and the details cannot be recognized. Underexposure also causes the loss of image information and affects image quality.

[0096] The picture quality of an image can be understood as the visual performance of the image, which usually involves multiple aspects, including but not limited to clarity, color accuracy, detail richness, noise level, contrast and brightness.

[0097] Similarity can be understood as the similarity between the image to be screened and the real image. Relatively speaking, the higher the similarity, that is, the closer the image to be screened is to the real image, the better the image to be screened can fit the real image and accurately represent the object in the real image.

[0098] It is worth noting that the images to be screened may be evaluated based on one of light intensity, picture quality, and similarity, or may be evaluated based on multiple of light intensity, picture quality, and similarity, which is not limited in this embodiment.

[0099] Relatively speaking, based on the evaluation result, an image with a relatively high quality is selected from the images to be screened as a rendered image. Exemplarily, the evaluation result can be compared with a preset quality score to select an image with an evaluation result greater than the preset quality score from the images to be screened as a rendered image. The preset quality score can be determined based on demand, historical records, experiments, etc., and is not limited in this embodiment.

[0100] Combined with the above analysis of step 21 and step 22, it can be seen that in this embodiment, if Figure 7 and Figure 8 As shown in the figure, after rendering under multiple viewpoints, multiple illumination conditions, and multiple simulated postures based on three-dimensional structure information, the rendered images (i.e., images to be screened) can be screened for quality, so as to select images with relatively better image quality from the images to be screened as rendered images. On the one hand, a larger number of feature similarity comparisons can be avoided to reduce the resource consumption of feature similarity comparisons, improve the efficiency of feature similarity comparisons, and thus improve the efficiency of image retrieval; on the other hand, the accuracy and reliability of retrieval can be improved.

[0101] In some embodiments, when the quality information includes similarity, the above step 22 of "filtering the rendered image from the images to be filtered according to the quality information" includes: from the images to be filtered, determining the images with a similarity greater than a preset threshold as the rendered image.

[0102] Similarly, this embodiment does not limit the preset threshold value, which can be determined based on demand, historical records, experiments, etc.

[0103] Exemplarily, the similarity between the image to be screened and the real-shot image is calculated, and the calculated similarity is compared with a preset threshold, so as to select an image whose comparison result indicates that the similarity is greater than the preset threshold as the rendered image.

[0104] In this embodiment, by setting a preset threshold and using it for comparison, images with low similarity can be effectively filtered out, thereby improving the accuracy of screening. In addition, by using the preset threshold and similarity calculation, it can be ensured that the returned image is highly similar to the image to be retrieved, thereby improving the user experience.

[0105] In some embodiments, S502 may include the following steps 1 and 2: Step 1: From the images in the database, determine the image with the highest feature similarity to the image to be retrieved.

[0106] For example, Figure 6 As shown, the images in the database include real-shot images and rendered images. The image retrieval system compares the feature similarity of the real-shot images and the rendered images with the images to be retrieved to obtain their corresponding feature similarities. Figure 6 The similarity scores shown in .

[0107] Image retrieval systems use the similarity of each feature, such as Figure 6 Among the similarity scores shown in , the highest similarity score of 86 is determined, and the image corresponding to the score of 86, namely, the rendered image A1, is determined.

[0108] Step 2: Determine the information of the image with the highest feature similarity and output it as the search result.

[0109] Continuing with the above example and Figure 6 , the image retrieval system determines the name of the rendered image A1 and outputs it as a retrieval result.

[0110] Combining the above analysis of step 1 and step 2, it can be seen that in this embodiment, by determining the image with the highest feature similarity from the database and feeding back the information of the determined image as the search result, the effectiveness and reliability of the search can be improved, and the user experience can also be improved.

[0111] It is worth noting that the above examples are only used to illustrate possible implementations of the image retrieval method of this specification, and cannot be understood as limiting the implementation of the image retrieval method of this specification. Exemplarily, based on the above technical concept, some of the above technical features can be combined to obtain a new embodiment; new technical features can be added on the basis of the above examples to obtain a new embodiment; some technical features can be reduced on the basis of the above examples to obtain a new embodiment; some technical features in the above examples can be replaced with other technical features; some technical features and their order in the above examples can be adjusted to obtain a new embodiment, etc., which will not be listed here one by one.

[0112] According to the above technical concept, this specification also provides a computer-readable non-transitory storage medium, in which at least one instruction set is stored. When the at least one instruction set is executed by a processor, the steps of the image retrieval method described in this specification are implemented.

[0113] In some possible implementations, various aspects of this specification may also be implemented in the form of a program product, which includes a program code. When the program product is run on the image retrieval system 400, the program code is used to enable the image retrieval system 400 to perform the steps of the image retrieval method described in this specification. The program product for implementing the above method may include program code using a portable compact disk read-only memory (CD-ROM) and may be run on the image retrieval system 400. However, the program product of this specification is not limited to this. In this specification, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system. The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the image retrieval system 400, partially on the image retrieval system 400, as a stand-alone software package, partially on the image retrieval system 400 and partially on a remote image retrieval system, or entirely on a remote image retrieval system 400.

[0114] It should be noted that in the technical solution of this specification, the collection, storage, use, processing, transmission, provision and disclosure of relevant user information (such as images to be retrieved transmitted by users, etc.) are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0115] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0116] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0117] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0118] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0119] Each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, documents, literature, etc., cited herein (excluding any historical review documents related thereto) is hereby incorporated by reference for all purposes relevant to this document, such as in the specification and claims herein. However, if there is any inconsistency or conflict between the descriptions, definitions, and / or terminology of the aforementioned materials and the descriptions, definitions, and / or terminology used herein, the descriptions, definitions, and / or terminology used herein shall prevail.

[0120] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. An image retrieval method, comprising: Obtain the image to be retrieved; Compare the image to be retrieved with images in a pre-built database for feature similarity, and obtain and output a retrieval result corresponding to the image to be retrieved; The images in the database include real-shot images and rendered images corresponding to a number of objects, and the rendered images are image variants generated by performing data enhancement processing on the real-shot images.

2. The method according to claim 1, wherein: The data enhancement processing includes transforming the real-shot image based on at least one of different viewing angles, different lighting conditions, and different postures to obtain the rendered image.

3. The method according to claim 2, wherein: Generating the rendered image includes: Converting the two-dimensional real-shot image into three-dimensional structural information; The three-dimensional structure information is rendered as the rendered image in a simulated environment to perform the data enhancement processing on the real-shot image, wherein the simulated environment includes at least one of simulating the different viewing angles, simulating the different lighting conditions, and simulating the different postures.

4. The method according to claim 3, wherein: The step of converting the two-dimensional real-shot image into three-dimensional structural information includes: The real-shot image is sequentially subjected to subject recognition, segmentation and cutout, and three-dimensional reconstruction to obtain the three-dimensional structure information.

5. The method according to claim 3, wherein: The rendering of the three-dimensional structure information into the rendered image in the simulation environment includes: Rendering the three-dimensional structure information as an image to be screened in the simulation environment; Evaluate quality information of the image to be screened, and screen the rendered image from the image to be screened according to the quality information, wherein the quality information includes at least one of light intensity, picture quality, and similarity, and the quality of the rendered image is better than other images in the image to be screened.

6. The method according to claim 5, wherein: The quality information includes similarity; and the step of obtaining the rendered image from the image to be screened according to the quality information includes: From the images to be screened, an image whose similarity is greater than a preset threshold is determined as the rendered image.

7. The method according to claim 3, wherein: The three-dimensional structure information includes a three-dimensional model, or includes a three-dimensional model and a map.

8. The method according to claim 1, wherein: The rendered image is obtained by performing data enhancement processing on the real-shot image using multimedia generation technology; the multimedia generation technology includes one of three-dimensional artificial intelligence content generation technology, variational autoencoder technology, and generative adversarial network technology.

9. The method according to claim 1, wherein: The step of comparing the image to be retrieved with images in a pre-built database for feature similarity to obtain and output a retrieval result corresponding to the image to be retrieved includes: Determine, from the images in the database, an image having the highest feature similarity to the image to be retrieved; The information of the image with the highest feature similarity is determined and output as the search result.

10. An image retrieval system, comprising: at least one storage medium storing at least one instruction set to perform image retrieval; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method as described in any one of claims 1 to 9 is executed according to the instructions of the at least one instruction set.

Images