Image retrieval method, image retrieval device and electronic device

Abstract

The present disclosure provides an image retrieval method, an image retrieval device and an electronic device, which can be used in the field of artificial intelligence, finance or other fields. The method includes: obtaining an image retrieval instruction, the image retrieval instruction includes a keyword; in response to the image retrieval instruction, traversing a user tag set based on the keyword to obtain an identification set of image identifications matching the keyword, the user tag set including tags and a mapping relationship between the tags and the image identifications; and determining an image matching the identification set from the image set, and outputting an image that at least partially matches.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technology and finance, and more specifically, to an image retrieval method, an image retrieval device, and an electronic device. Background Art

[0002] In the related art, image albums mainly classify and store images stored by users based on dimensions such as time, photo location, and photo type, so that users can quickly find the desired images.

[0003] In the process of implementing the concept of the present disclosure, the applicant found that there are at least the following problems in the related art: as the images stored by users continue to grow, the time span becomes longer and longer, and the images involve more and more categories, users cannot easily find the images they need from the image album. Summary of the invention

[0004] In view of this, the present disclosure provides an image retrieval method, an image retrieval device and an electronic device, which at least partially solve the problem that users cannot easily find desired images from image albums, so as to improve user experience.

[0005] One aspect of the present disclosure provides an image retrieval method, an image retrieval device, and an electronic device to at least partially solve the network training difficulties caused by unlabeled fused images in image fusion tasks, and reduce the dependence on manually designed fusion rules to improve the fusion performance of the network.

[0006] One aspect of the present disclosure provides an image retrieval method, comprising: obtaining an image retrieval instruction, the image retrieval instruction including keywords; in response to the image retrieval instruction, traversing a user tag set based on the keywords to obtain an identification set of image identifications that match the keywords, the user tag set including tags and a mapping relationship between tags and image identifications; and determining images that match the identification set from the image set, and outputting images that at least partially match.

[0007] One aspect of the present disclosure provides an image retrieval device, comprising: a retrieval instruction acquisition module, an identification set determination module, and an image determination module. The retrieval instruction acquisition module is used to acquire an image retrieval instruction, the image retrieval instruction includes a keyword; the identification set determination module is used to respond to the image retrieval instruction, traverse a user tag set based on the keyword, and obtain an identification set of image identifications matching the keyword, the user tag set includes tags and a mapping relationship between tags and image identifications; and the image determination module is used to determine an image matching the identification set from the image set, and output an image that at least partially matches.

[0008] Another aspect of the present disclosure provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, and when the executable instructions are executed by the processor, the above method is implemented.

[0009] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the above-mentioned method for training an image retrieval model and / or an image retrieval method.

[0010] Another aspect of the present disclosure provides a computer program, which includes computer executable instructions, and when the instructions are executed, are used to implement the above-mentioned method for training an image retrieval model and / or an image retrieval method.

[0011] The image retrieval method, image retrieval device and electronic device provided by the embodiments of the present disclosure can, when performing image retrieval, describe key information of the required image, such as keywords, use the keywords to match existing tags in the user tag set, obtain image identifiers corresponding to the matching tags based on the mapping relationship, and then output images identified by the matching image identifiers, thereby improving the convenience for users to find the required images.

[0012] The image retrieval method, image retrieval device and electronic device provided by the embodiments of the present disclosure automatically generate descriptive text of an input image through a descriptive text acquisition model, then generate a label of the input image based on keywords extracted from the descriptive text and attribute information of the image, and then generate a user label set including a mapping relationship between image identifiers and labels. It is unnecessary for users to manually label each input image, thereby improving the convenience of using the album for users.

[0013] The image retrieval method, image retrieval device and electronic device provided by the embodiments of the present disclosure, combined with digital human technology, expand the application scenarios of digital humans for image retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1 The application scenarios of the image retrieval method, image retrieval device and electronic device according to the embodiments of the present disclosure are schematically shown;

[0016] Figure 2 Schematically illustrates an exemplary system architecture to which an image retrieval method and an image retrieval device according to an embodiment of the present disclosure can be applied;

[0017] Figure 3 A flowchart of an image retrieval method according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 4 A schematic diagram schematically shows a user tag set according to an embodiment of the present disclosure;

[0019] Figure 5 A flowchart of an image retrieval method according to another embodiment of the present disclosure is schematically shown;

[0020] Figure 6 The schematic diagram of the structure of the convolutional neural network according to the embodiment of the present disclosure is shown;

[0021] Figure 7 A schematic diagram of the structure of a neural network according to an embodiment of the present disclosure is shown;

[0022] Figure 8 A schematic diagram of the structure of a neural network according to another embodiment of the present disclosure is shown;

[0023] Fig. 9 The structure diagram of the self-attention mechanism network according to the embodiment of the present disclosure is schematically shown;

[0024] Fig.10 A block diagram schematically shows an image retrieval device according to an embodiment of the present disclosure; and

[0025] Fig.11 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0027] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0028] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification, and should not be interpreted in an idealized or overly rigid manner.

[0029] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.). The terms "first" and "second" are used only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features.

[0030] In the related art, image album software mainly divides user stored images based on time, photo location, photo type, etc. For example, images are divided based on the time when the user took the photo or uploaded the photo. Location information is annotated according to the location information when the user took the photo, such as Beijing, Jiangsu, etc. Images are divided based on deep learning to identify the content in the image, such as people, sky, buildings, flowers, etc.

[0031] At the same time, auxiliary face recognition and other methods can cluster faces in photos, and users can also search images based on identity information in photos. In addition, some software also provides users with manual photo tags to facilitate image search based on tags.

[0032] As the image data stored by users continues to grow, the time span is getting longer and longer, and more and more photos are taken in long-term residences, user preferences lead to a certain type of photo or a certain person's photos being more common. Under existing technical conditions, the time it takes for users to manually search is greatly extended with the number of photos. In addition, users generally upload a large number of photos in batches to albums or back them up to cloud space, making it difficult for users to maintain and implement manual labeling without compensation.

[0033] The embodiments of the present disclosure provide an image retrieval method, an image retrieval device, and an electronic device. The image retrieval method includes an identification set determination process and an image determination process. First, an image retrieval instruction is obtained, and the image retrieval instruction includes a keyword. Then, in response to the image retrieval instruction, a user tag set is traversed based on the keyword to obtain an identification set of image tags that match the keyword. The user tag set includes tags and a mapping relationship between tags and image tags. After completing the identification set determination process, the image determination process is entered to determine an image that matches the identification set from the image set, and output an image that at least partially matches.

[0034] Figure 1 The application scenarios of the image retrieval method, image retrieval device and electronic device according to the embodiments of the present disclosure are schematically illustrated.

[0035] like Figure 1 As shown, each image in the image set can have corresponding text description information, such as Figure 1 The description in can be: A man in a T-shirt is playing a motion game on a TV. In addition, the image can also have image attributes, such as the image was uploaded on XX / YY / XX, the image was taken on XX / YY / XX, the image was taken in Beijing, the image was taken at the Science and Technology Museum, etc. In order to facilitate image retrieval in the later stage, a label corresponding to the image can be generated based on the above description and image attributes, such as label 1, T-shirt, label 2, TV, label 3, game, label 4, motion game, label 5, XX / YY / XX, etc. Each label can be stored in the user label library, and then the label of each image identifier is determined by establishing a mapping relationship. This makes it easy to retrieve the required image by label matching.

[0036] Figure 2 The exemplary system architecture to which the image retrieval method and the image retrieval device according to the embodiments of the present disclosure can be applied is schematically shown. It should be noted that: Figure 2 What is shown is only an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. It should be noted that the image retrieval method, image retrieval device and electronic device provided by the embodiments of the present disclosure can be used in the field of artificial intelligence in image retrieval-related aspects, and can also be used in various fields other than the field of artificial intelligence, such as the financial field, etc. The application field of the image retrieval method, image retrieval device and electronic device provided by the embodiments of the present disclosure is not limited.

[0037] like Figure 2As shown, the system architecture 200 according to this embodiment may include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 may include multiple gateways, routers, hubs, network cables, etc., which are used to provide a medium for communication links between the terminal devices 201, 202, 203 and the server 205. The network 204 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0038] The user can use the terminal devices 201, 202, 203 to interact with other terminal devices and the server 205 through the network 204 to receive or send information, such as uploading images, sending image retrieval instructions, receiving images, etc. The terminal devices 201, 202, 203 can be installed with various communication client applications, such as photo album applications, banking applications, e-commerce applications, web browser applications, search applications, office applications, instant messaging tools, email clients, social platform software and other applications (only for example).

[0039] Terminal devices 201, 202, 203 include but are not limited to smart phones, desktop computers, augmented reality devices, tablet computers, laptop computers, and other electronic devices that can support functions such as surfing the Internet and voice collection. A descriptive text acquisition model can be downloaded on the terminal device to generate descriptive text and labels for images.

[0040] The server 205 may receive model training requests, image retrieval requests, model download requests, etc., and process the requests. For example, the server 205 may be a background management server, a server cluster, etc. The background management server may analyze and process the received service requests, information requests, etc., and feed back the processing results (such as retrieved images, model parameters obtained by training models, etc.) to the terminal device.

[0041] It should be noted that the image retrieval method provided in the embodiment of the present disclosure can be executed by the terminal device 201, 202, 203 or the server 205. Accordingly, the image retrieval apparatus provided in the embodiment of the present disclosure can be set in the terminal device 201, 202, 203 or the server 205. It should be understood that the number of terminal devices, networks and servers is only illustrative. According to the implementation requirements, there can be any number of terminal devices, networks and servers.

[0042] Figure 3 The flowchart of the image retrieval method according to the embodiment of the present disclosure is schematically shown. Figure 3 As shown, the image retrieval method may include operations S302 to S306.

[0043] In operation S302, an image search instruction is obtained, where the image search instruction includes a keyword.

[0044] In this embodiment, the image retrieval instruction can be a text type instruction or a voice type instruction. For example, the user can input the text information "image taken in the first half of XX, I am playing a somatosensory game in the image, the location is the Science and Technology Museum". For another example, the user can issue a voice instruction to the digital human "help me find the photo of me playing a somatosensory game taken in the Science and Technology Museum in the first half of XX, I was wearing a T-shirt at the time". The image retrieval instruction shown above can include key information such as: the first half of XX, game, somatosensory game, science and technology museum, etc.

[0045] For text-type instructions, keywords can be extracted from text information through a variety of natural language processing methods. In addition, semantic information of keywords or text information can be extracted to improve the probability of successful retrieval.

[0046] In one embodiment, obtaining an image retrieval instruction may include the following operations.

[0047] First, in response to the acquired user voice information, voice recognition is performed on the user voice information to obtain text information. Then, natural language processing is performed on the text information to obtain keywords and semantic information of the keywords. Next, an image retrieval instruction is generated based on the keywords and the semantic information of the keywords.

[0048] For example, performing voice recognition on user voice information includes performing voice recognition on user voice information input by a digital person.

[0049] In recent years, with the rise of technologies such as smart speakers, the application areas of voice-driven question-answering, search and other functions have gradually expanded. Among them, voice search provides users with a more convenient way to search. However, existing image searches are still mainly based on text descriptions, because voice search images are essentially to capture the voice content as text and then search. Therefore, in traditional application scenarios, voice search images rarely have specific applications. However, digital human technology has brought new changes to this field.

[0050] Digital human is a digital mimicry that can interact with users visually on a variety of media based on artificial intelligence technology. Digital human technology uses modeling technology to construct virtual images, and through the integration of multiple disciplines such as deep learning neural networks and computer graphics, drives virtual images through voice, allowing mimicry to understand voice content and generate dialogues, and finely drives virtual images to make expressions and movements that match the context, ultimately giving mimicry intelligent and anthropomorphic interaction capabilities. Therefore, as a new generation of input interface, digital human must also have a place in voice search for images.

[0051] For voice-type instructions, specifically, the user's voice information can be first obtained through a human-machine interface such as a digital human, and then the digital human sends the voice information to a processor or uploads it to a voice recognition platform for voice recognition to obtain text information corresponding to the voice information. In this way, natural language processing can be performed on the text information to determine keywords and semantic information, etc.

[0052] In operation S304, in response to the image search instruction, the user tag set is traversed based on the keyword to obtain an identification set of image identifications matching the keyword, wherein the user tag set includes tags and mapping relationships between tags and image identifications.

[0053] Figure 4 A schematic diagram of a user tag set according to an embodiment of the present disclosure is schematically shown.

[0054] like Figure 4 As shown, Figure 4 The dashed box on the left represents the label set. Figure 4 The dashed box on the right of represents an image set (which may have a corresponding tag set). Each tag in the tag set may have a mapping relationship with each image identifier in the image identifier set. Each tag in the tag set may be shared, such as Figure 4 In the example, label 1 is used as the label of the image with image ID 1, the image with image ID 2, and the image with image ID 3. For example, the descriptions of the image with image ID 1 and the image with image ID 2 both include the keyword "T-shirt". In this case, there is no need to set a label for "T-shirt" for the image with image ID 1 and the image with image ID 2 respectively, but only needs to be mapped to label 1. This helps to reduce the storage space occupied, and there is no need to modify the image data of each image, such as no need to set annotation information for the image.

[0055] In operation S306, images matching the identification set are determined from the image set, and images that at least partially match are output.

[0056] After the matching image identifiers are determined, at least partially matching images may be output, such as sending at least partially matching images to the client, or displaying at least partially matching images on a terminal device for user selection.

[0057] Figure 5 The flowchart of an image retrieval method according to another embodiment of the present disclosure is schematically shown.

[0058] like Figure 5 As shown, after obtaining the image retrieval instruction, the method may further include operation S508.

[0059] In operation S508, semantic information of the keyword is acquired.

[0060] Among them, semantic information can appropriately expand the scope of keywords to reduce the possibility of missing correct results. For example, if the keyword is the first half of this year, through semantic understanding, it can be analyzed that the search range represented by it is: 0:0:00 on January 1, XX to 23:59:59 on June 30, XX. This makes it easier to match. In addition, semantic information can appropriately narrow the search scope. For example, if the user is a male, "I wear sports shoes", combined with the context, it can be known that the sports shoes are "men's sports shoes", which helps to reduce the search scope.

[0061] Accordingly, traversing the user tag set based on the keyword to obtain an identification set of image identifications matching the keyword includes: traversing the user tag set based on the keyword and the semantic information of the keyword to obtain an identification set of image identifications matching the keyword and the semantic information of the keyword.

[0062] In one embodiment, when there are multiple search results, in order to facilitate users to find the required images from the search results more quickly, after determining the images that match the identification set from the image set, the above method may also include the following operations: sorting the matching images in order from high to low matching degree to obtain a sorted set of images to be output, and the matching degree is determined based on the number and similarity of keywords and / or semantic information of the keywords that match the image labels.

[0063] For example, the more the number of image tags that match keywords, the greater the probability that it is the image required by the user. In addition, the higher the similarity between the tag and the keyword, the greater the probability that it is the image required by the user. Specifically, the similarity can be used as a weight, with the weight of the exact same keyword being 1, the weight of a synonym being 0.6, etc., and the ranking is determined based on the weighted sum result.

[0064] Correspondingly, outputting at least partially matched images includes: outputting at least part of the images in the ranked set of images to be output in descending order of ranking.

[0065] The disclosed embodiment can quickly retrieve the stored content through voice or text input, which can reduce the complexity of manual search and search. When searching, the user can describe the background situation through voice or text and pass it to the digital human. The digital human obtains the keywords in the description through voice recognition and natural language processing. The similarity between the keywords and the existing tags is used to match the images above a certain threshold and feedback them to the user for selection.

[0066] The following is an exemplary description of the process of determining the label of an image.

[0067] The labels of the images can be automatically generated or manually set according to the needs of the user. The process of automatically generating labels is exemplified below.

[0068] In one embodiment, the above method may further include the following operations.

[0069] First, an input image is obtained, and the input image has attribute information. The attribute information includes, but is not limited to: shooting time, shooting location, the size of the storage space occupied by the image, the resolution of the image, and other information that helps improve the accuracy of retrieval. The attribute information can be input by the user or automatically obtained by the device.

[0070] Then, a descriptive text for the input image is obtained, which may be text information converted from voice information input by the user, or text information input by the user, or may be a descriptive text of the image automatically generated.

[0071] For example, after a user uploads an image, the server uses the trained descriptive text acquisition model to process the image content, obtain descriptive text, generate UUID (as the unique identifier of the image) and upload time, for example, "On the lawn, a man in a blue coat is playing badminton" is generated based on the photo content. Specifically, the adaptive attention method can be used to train the model to input a picture and return the content description of the related image.

[0072] Next, keywords are extracted from the descriptive text. The method of extracting keywords can be the same as the prior art, such as determining keywords based on part of speech, searching a keyword dictionary, etc., which is not limited here.

[0073] Then, the attribute information and keywords are used as labels of the input image, such as keywords, time attributes, geographic location attributes, etc. are used as labels of the input image respectively.

[0074] Next, a user tag set is constructed based on the image identifier and tag of the input image, or the user tag set is updated. Figure 4 shown.

[0075] Then, core fields are extracted from the descriptive text, for example, based on named entity recognition methods (such as Bidirectional Encoder Representation from Transformers, BERT)-Conditional Random Fields (CRF). As photo tags, such as: "one", "lawn", "blue", "coat", "man", "badminton". After that, the photo tags, the corresponding image identifiers (UUIDs) and image attributes (such as upload time) are stored in the user's personal tag library, and the photos and the corresponding UUIDs are stored in the user's personal photo library.

[0076] In one embodiment, obtaining descriptive text for an input image may include: processing the input image using a trained descriptive text acquisition model to obtain descriptive text of the input image, wherein the descriptive text acquisition model includes a neural network.

[0077] In the process of generating descriptive text based on images, it is necessary to extract image features. Figure 6 The structure diagram of the convolutional neural network according to the embodiment of the present disclosure is schematically shown. For example, the convolutional neural network can be used as shown in FIG. Figure 6 The convolutional neural network (CNN) shown here performs feature extraction.

[0078] In one embodiment, the neural network includes a multi-layer perceptron (MLP), a self-attention mechanism (Atten) network, and a long short-term memory (LSTM) network.

[0079] Among them, multi-layer perceptrons, self-attention mechanism networks, and long short-term memory networks can have multiple connection methods.

[0080] Figure 7 The schematic diagram shows the structure of a neural network according to an embodiment of the present disclosure.

[0081] like Figure 7 As shown, the output of the LSTM network is connected to the input of the Atten network and the input of the MLP respectively, and the output of the Atten network is connected to the input of the MLP. Figure 7 Medium t It is the concatenation of global image features and word vectors. The global image features are extracted from the input image. t represents the context vector. V represents the local image feature and can be expressed as V = [v1, v2, v3, ..., v L ], L is the number of local image features. h t is the historical information of the generated text. t It is a descriptive text for the input image. The Atten network can determine the weights of multiple local image features based on the self-attention mechanism.

[0082] The mining of non-visual words such as "the" and "of" does not require visual information. Their gradients will affect the effectiveness of visual information. Therefore, it is necessary to control the proportion of visual information and historical information in the Atten network. t To represent the historical information of the generated text. This is equivalent to deciding when to read and how much to read from the dimension of time. t To decide the information to look at, this can reduce the uncertainty of the hidden layer for predicting the next word.

[0083] Figure 8A schematic diagram of the structure of a neural network according to another embodiment of the present disclosure is shown schematically.

[0084] The long short-term memory network is used to transmit the first history information and the second history information of the generated text to the self-attention mechanism network, and transmit the second history information of the generated text to the multi-layer perceptron.

[0085] The self-attention mechanism network is used to perform weighted fusion on the first historical information and the at least two local image features based on the first weight of the first historical information and the second weights of the at least two local image features inputted to obtain a context vector, and output it to the multilayer perceptron. The first weight and the second weight are determined by the self-attention mechanism network based on the self-attention mechanism using the second historical information of the generated text as supervision information. The at least two local image features represent the visual information.

[0086] A multi-layer perceptron is used to generate descriptive text of the input image based on the context vector and the second history information of the generated text.

[0087] For non-visual words, their generation should depend on historical information rather than visual information, so visual information should be controlled in this case. Figure 7 The difference is, Figure 8 There is an extra S in t information, and its generating structure and h t is consistent, so it contains the text information that has been generated.

[0088] Fig. 9 The structural diagram of the self-attention mechanism network according to an embodiment of the present disclosure is schematically shown.

[0089] like Fig. 9 As shown, the self-attention mechanism network includes a branch network for each of at least two local image features and a branch network for the first historical information of the generated text.

[0090] Specifically, S t After the information is introduced into the Atten network, the Atten network generates a new context vector As shown in formula (1).

[0091]

[0092] in, For example, α can be determined by normalization operation ti and β i .

[0093] Specifically, a new parameter β is introduced t , the original context vector Ct With S t When β t When β is 1, the generation of the current text depends only on the previously generated text information. t If it is 0, it depends only on visual information, then the output result is and Figure 7 Output result C t From this, we can decide whether to look at visual information at time t, and how much to look at.

[0094] When the model believes that visual information is useless, β i The coefficient is very large, which can suppress the visual feature v i Participate in word prediction; but when predicting words with object meaning, the model needs visual features v i Participate, at this time β i The coefficient will be small, and the visual features within the object area will be noticed.

[0095] For example, the weights of the branch networks are obtained by activation functions. For example, the weights of the corresponding branches are obtained by using the Sigmoid function. The weight normalization for α can be achieved by, for example, α / (α+β).

[0096] The above method can automatically generate descriptive text for the input image, and then generate labels for the input image based on the keywords extracted from the above descriptive text and the attributes of the acquired image, and then construct or update the above user label set to perform image retrieval based on the user label set.

[0097] In a specific embodiment, when the user searches for relevant pictures by voice, the voice is converted into text content through voice recognition. The voice content is divided using a word segmentation method, for example: "Photos of playing badminton in October last year" voice content named entity recognition is "last October", "badminton", and "of", "photos" are automatically removed because they are automatically recognized as shielded words. Through natural language processing, text recognition such as "last October" is converted into specific time. It is assumed that "last October" is converted to "[2019-10-01, 2019-10-31]". Then, according to "[2019-10-01, 2019-10-31]" and "badminton", the upload time and tag columns of the user tag set (such as a personal user tag library) are traversed to filter out the UUID list that meets the time period and tag. Finally, use the filtered UUID list to the user's personal photo library to retrieve all photos that meet the user's needs.

[0098] The image retrieval method provided by the embodiment of the present disclosure can effectively improve the retrieval speed and significantly reduce the time required for users to retrieve pictures. In addition, combined with digital human technology, the application scenarios of digital humans can be expanded.

[0099] Fig.10 The block diagram of the image retrieval device according to the embodiment of the present disclosure is schematically shown.

[0100] like Fig.10 As shown, the image retrieval device 1000 may include: a retrieval instruction acquisition module 1010 , an identification set determination module 1020 and an image determination module 1030 .

[0101] The search instruction acquisition module 1010 is used to acquire image search instructions, and the image search instructions include keywords.

[0102] The identification set determination module 1020 is used to respond to the image search instruction, traverse the user tag set based on the keyword, and obtain the identification set of image identifications matching the keyword. The user tag set includes tags and mapping relationships between tags and image identifications.

[0103] The image determination module 1030 is used to determine images that match the identification set from the image set, and output images that at least partially match.

[0104] It should be noted that the implementation methods, technical problems solved, functions realized, and technical effects achieved of each module / unit in the device part embodiment are the same or similar to the implementation methods, technical problems solved, functions realized, and technical effects achieved of each corresponding step in the method part embodiment, and they will not be repeated here one by one.

[0105] According to the modules and units of the embodiments of the present disclosure, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules and units of the embodiments of the present disclosure, any one or more can be split into multiple modules for implementation. According to the modules and units of the embodiments of the present disclosure, any one or more can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, according to the modules and units of the embodiments of the present disclosure, one or more can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0106] For example, any multiple of the retrieval instruction acquisition module 1010, the identification set determination module 1020, and the image determination module 1030 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the retrieval instruction acquisition module 1010, the identification set determination module 1020, and the image determination module 1030 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, at least one of the search instruction acquisition module 1010, the identification set determination module 1020 and the image determination module 1030 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be executed.

[0107] Fig.11 A block diagram of an electronic device according to an embodiment of the present disclosure is schematically shown. Fig.11 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0108] like Fig.11 As shown, the electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1102 or a program loaded from a storage part 1108 into a random access memory (RAM) 1103. The processor 1101 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), and the like. The processor 1101 may also include an onboard memory for caching purposes. The processor 1101 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0109] In RAM 1103, various programs and data required for the operation of electronic device 1100 are stored. Processor 1101, ROM 1102 and RAM 1103 are connected to each other through bus 1104. Processor 1101 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 1102 and / or RAM 1103. It should be noted that the program can also be stored in one or more memories other than ROM 1102 and RAM 1103. Processor 1101 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0110] According to an embodiment of the present disclosure, the electronic device 1100 may further include an input / output (I / O) interface 1105, which is also connected to the bus 1104. The electronic device 1100 may further include one or more of the following components connected to the I / O interface 1105: an input portion 1106 including a keyboard, a mouse, etc.; an output portion 1107 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 1108 including a hard disk, etc.; and a communication portion 1109 including a network interface card such as a LAN card, a modem, etc. The communication portion 1109 performs communication processing via a network such as the Internet. A drive 1110 is also connected to the I / O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that a computer program read therefrom is installed into the storage portion 1108 as needed.

[0111] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1109, and / or installed from the removable medium 1111. When the computer program is executed by the processor 1101, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0112] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0113] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus, or a device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 1102 and / or RAM 1103 described above and / or one or more memories other than ROM 1102 and RAM 1103.

[0114] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the image model training method or image retrieval method provided by the embodiment of the present disclosure.

[0115] When the computer program is executed by the processor 1101, the above functions defined in the system / device of the embodiment of the present disclosure are executed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0116] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 1109, and / or installed from a removable medium 1111. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0117] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0118] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. These embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the various embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. The scope of the present disclosure is defined by the attached claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, all of which should fall within the scope of the present disclosure.

Claims

1. An image retrieval method, comprising: Obtaining an image retrieval instruction, wherein the image retrieval instruction includes a keyword; In response to the image retrieval instruction, traverse a user tag set based on the keyword to obtain an identification set of image identifications matching the keyword, the user tag set including tags and mapping relationships between tags and image identifications, wherein the user tag set is pre-constructed based on attribute information, descriptive text and image identification of an input image; and Images matching the identification set are determined from the image set, and at least part of the matching images are output.

2. The method according to claim 1, further comprising: Acquire an input image, wherein the input image has attribute information; Obtaining descriptive text for the input image; extracting keywords from the descriptive text; Using the attribute information and the keyword as a label of the input image; as well as The user tag set is constructed based on the image identifier and the tag of the input image, or the user tag set is updated.

3. The method according to claim 2, wherein: The obtaining of descriptive text for the input image comprises: The input image is processed using a trained descriptive text acquisition model to obtain descriptive text of the input image, wherein the descriptive text acquisition model includes a neural network.

4. The method according to claim 3, wherein: The neural network includes a multi-layer perceptron, a self-attention mechanism network and a long short-term memory network, wherein: The long short-term memory network is used to transmit the first history information and the second history information of the generated text to the self-attention mechanism network, and transmit the second history information of the generated text to the multi-layer perceptron; The self-attention mechanism network is used to perform weighted fusion on the first historical information and the at least two local image features based on a first weight of the first historical information and a second weight of each of the at least two local image features input to obtain a context vector, and output the context vector to the multilayer perceptron, wherein the first weight and the second weight are determined by the self-attention mechanism network based on the self-attention mechanism using the second historical information of the generated text as supervisory information, and the at least two local image features represent visual information; and The multilayer perceptron is used to generate descriptive text of the input image based on the context vector and second historical information of the generated text.

5. The method according to claim 4, wherein: The self-attention mechanism network includes a branch network for each of at least two local image features and a branch network for the first historical information of the generated text.

6. The method according to claim 1, further comprising: After the image retrieval instruction is obtained, Acquiring semantic information of the keyword; The step of traversing the user tag set based on the keyword to obtain a set of image tags matching the keyword includes: The user tag set is traversed based on the keyword and the semantic information of the keyword to obtain an identification set of image identifications that match the keyword and the semantic information of the keyword.

7. The method according to claim 6, further comprising: After determining the image matching the identification set from the image set, Sorting the matched images in descending order of matching degree to obtain a set of images to be outputted with sorting, wherein the matching degree is determined based on the number and similarity of keywords and / or semantic information of keywords that match the labels of the images; as well as The outputting at least part of the matched images comprises: outputting at least part of the images in the ranked set of images to be output in descending order.

8. The method according to any one of claims 1 to 7, wherein: The image retrieval instruction includes: In response to the acquired user voice information, performing voice recognition on the user voice information to obtain text information; Performing natural language processing on the text information to obtain the keywords and semantic information of the keywords; and An image retrieval instruction is generated based on the keyword and semantic information of the keyword.

9. The method according to claim 8, wherein: The performing voice recognition on the user voice information includes performing voice recognition on the user voice information input by a digital person.

10. An image retrieval device, comprising: A search instruction acquisition module, used to acquire an image search instruction, wherein the image search instruction includes a keyword; an identification set determination module, configured to, in response to the image search instruction, traverse the user tag set based on the keyword to obtain an identification set of image identifications matching the keyword, wherein the user tag set includes tags and mapping relationships between tags and image identifications, wherein the user tag set is pre-constructed based on attribute information and descriptive text of the input image; as well as The image determination module is used to determine images matching the identification set from the image set and output at least part of the matching images.

11. An electronic device, comprising: one or more processors; A storage device for storing executable instructions, wherein when the executable instructions are executed by the processor, the image retrieval method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, implements the image retrieval method according to any one of claims 1 to 9.

Images