Image-text retrieval method and device, electronic equipment and storage medium

Abstract

The present disclosure relates to a picture-text retrieval method and device, electronic equipment and storage medium, and relates to the technical field of information processing. The present disclosure at least solves the problem of low picture-text matching accuracy in cross-modal retrieval in the related art. The method comprises: obtaining a text to be retrieved; identifying a target entity in the text to be retrieved based on a pre-established entity reference word library. The entity reference word library comprises reference words corresponding to entities of a preset type, the target entity is an entity determined based on a target reference word, and the target reference word is a word in the text to be retrieved that matches the reference word; determining the similarity between the text to be retrieved and each picture in a picture library, and taking pictures with a similarity greater than a first threshold value as candidate pictures in a candidate picture library; and determining a target picture corresponding to the target entity from the candidate picture library.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of information processing, and particularly relates to a picture-text retrieval method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the development of Internet technology, various applications and web pages generate a large amount of pictures and texts every day, and the pictures and texts can have certain connections.

[0003] In actual applications, a picture corresponding to a text can be retrieved based on a cross-modal retrieval algorithm. In related technologies, the cross-modal retrieval algorithm mainly extracts image features of all pictures in a picture library and text features of texts, and determines the similarity of each picture and text according to the image features and the text features, and then determines the picture with the highest similarity to the text from the picture library.

[0004] However, in the above method, the text features are usually determined according to entities in the text. In the case where the text includes ambiguous entity information (such as "he"), it is difficult to accurately identify the text features, resulting in low accuracy of picture-text matching. SUMMARY

[0005] The present disclosure provides a picture-text retrieval method, device, electronic equipment and storage medium to at least solve the problem of low accuracy of picture-text matching in cross-modal retrieval in related technologies. The technical solutions of the present disclosure are as follows:

[0006] According to a first aspect of the embodiments of the present disclosure, a picture-text retrieval method is provided, including: obtaining a text to be retrieved; identifying a target entity in the text to be retrieved based on a pre-established entity pronoun library; the entity pronoun library includes pronouns corresponding to entities of a preset type, the target entity is an entity determined based on a target pronoun, and the target pronoun is a word in the text to be retrieved that matches the pronoun; determining the similarity between the text to be retrieved and each picture in a picture library, and regarding the pictures with a similarity greater than a first threshold value as candidate pictures in a candidate picture library; and determining a target picture corresponding to the target entity from the candidate picture library.

[0007] In some embodiments, the identifying the target entity in the text to be searched based on the pre-established entity reference library comprises: reading the entity reference library, and matching the text to be searched to determine a target reference; when the target reference does not exist, determining the target entity as a default entity; when the target reference exists and the target reference corresponds to at least two types of preset entities, determining the target entity as a target preset type entity; the target preset type entity is an entity type corresponding to a target reference with a highest ranking in the text to be searched, or an entity type corresponding to a target reference with a highest occurrence frequency in the text to be searched; when the target reference exists and the target reference corresponds to one type of preset entity, determining the target entity as an entity type corresponding to the target reference.

[0008] In some embodiments, the determining the similarity between the text to be searched and each picture in the picture library comprises: determining a text feature vector of the text to be searched and an image feature vector of each picture in the picture library according to the trained image-text detection model; and determining the similarity between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture.

[0009] In some embodiments, the determining the similarity between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture comprises: determining a cosine distance between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture; and determining the cosine distance as the similarity between the text to be searched and each picture.

[0010] In some embodiments, the text to be searched further comprises a target object; and the determining the target picture corresponding to the target entity from the candidate picture library comprises: determining a target candidate picture comprising the target object from the candidate picture library according to the trained object recognition model; when the target candidate picture does not exist, determining a candidate picture with a highest similarity to the text to be searched in the candidate picture library as the target picture; and when the target candidate picture exists, determining the target picture corresponding to the target entity from the target candidate picture according to the trained entity recognition model.

[0011] In some embodiments, the text to be searched further comprises a target object, the target object is a person, and the preset type is used to indicate a gender or an age of the person.

[0012] According to a second aspect of the embodiments of the present disclosure, a picture-text retrieval device is provided, which comprises: an acquisition module configured to acquire a text to be retrieved; an entity recognition module configured to recognize a target entity in the text to be retrieved acquired by the acquisition module based on a pre-established entity reference library; the entity reference library comprises references corresponding to entities of preset types, and the target entity is an entity determined based on a target reference; the target reference is a word in the text to be retrieved matching the reference; a similarity determination module configured to determine a similarity between the text to be retrieved acquired by the acquisition module and each picture in a picture library; a picture candidate module configured to take pictures with a similarity greater than a first threshold value determined by the similarity determination module as candidate pictures in a candidate picture library; and a picture recognition module configured to determine a target picture corresponding to the target entity determined by the entity recognition module from the candidate picture library determined by the picture candidate module.

[0013] In some embodiments, the entity recognition module is specifically configured to: read the entity reference library, and determine the target reference by matching the text to be retrieved; when the target reference does not exist, determine the target entity as a default entity; when the target reference exists and corresponds to at least two entities of preset types, determine the target entity as a target entity of a preset type; the target entity of the preset type is an entity type corresponding to a target reference with a highest ranking in the text to be retrieved, or an entity type corresponding to a target reference with a highest occurrence frequency in the text to be retrieved; and when the target reference exists and corresponds to an entity of a preset type, determine the target entity as the entity type corresponding to the target reference.

[0014] In some embodiments, the similarity determination module is specifically configured to: determine a text feature vector of the text to be retrieved and an image feature vector of each picture in the picture library according to a trained picture-text detection model; and determine the similarity between the text to be retrieved and each picture according to the text feature vector and the image feature vector of each picture.

[0015] In some embodiments, the similarity determination module is specifically configured to: determine a cosine distance between the text to be retrieved and each picture according to the text feature vector and the image feature vector of each picture; and determine the cosine distance as the similarity between the text to be retrieved and each picture.

[0016] In some embodiments, the text to be retrieved further comprises a target object; and the picture recognition module is specifically configured to: determine a target candidate picture comprising the target object from the candidate picture library according to a trained object recognition model; when the target candidate picture does not exist, determine a candidate picture with a highest similarity to the text to be retrieved in the candidate picture library as the target picture; and when the target candidate picture exists, determine the target picture corresponding to the target entity from the target candidate picture according to the trained entity recognition model.

[0017] In some embodiments, the to-be-retrieved text further includes a target object, the target object is a person, and the preset type is used to indicate a gender or an age of the person.

[0018] According to a third aspect of the embodiments of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the image-text retrieval method provided in the first aspect.

[0019] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, including instructions; when the instructions are executed by a processor of an electronic device, the electronic device performs the image-text retrieval method provided in the first aspect.

[0020] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including instructions; when the instructions are executed on a processor of an electronic device, the electronic device performs the image-text retrieval method provided in the first aspect.

[0021] The embodiments of the present disclosure provide at least the following beneficial effects: in the present disclosure, the electronic device can determine a candidate picture library according to the similarity between the to-be-retrieved text and each picture in the picture library, and recognize the target entity in the to-be-retrieved text according to the pre-established entity reference word library, so as to recognize the target picture corresponding to the target entity from the candidate picture library. Since in the embodiments of the present disclosure, the electronic device can not only establish a candidate picture library according to the similarity between the to-be-retrieved text and each picture in the picture library, but also further screen the candidate pictures in the candidate picture library according to the target entity in the to-be-retrieved text, the accuracy of image-text matching is improved.

[0022] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0023] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure, and do not constitute an improper limitation on the present disclosure.

[0024] Figure 1 is an architecture diagram of an implementation environment according to an exemplary embodiment.

[0025] Figure 2 is one of flowcharts of an image-text retrieval method according to an exemplary embodiment.

[0026] Figure 3 is another one of flowcharts of an image-text retrieval method according to an exemplary embodiment.

[0027] Figure 4 FIG. 3 is a flowchart of a method for image-text retrieval according to an example embodiment.

[0028] Figure 5 FIG. 4 is a flowchart of a method for image-text retrieval according to an example embodiment.

[0029] Figure 6 FIG. 5 is a flowchart of a method for image-text retrieval according to an example embodiment.

[0030] Figure 7 FIG. 6 is a structural schematic diagram of an image-text retrieval device according to an example embodiment.

[0031] Figure 8 FIG. 7 is a structural schematic diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0032] In order to make the ordinary person in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be described clearly and completely below in conjunction with the drawings.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than that illustrated or described herein. The implementation described in the following example embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0034] In the related art, cross-modal retrieval of image-text is mainly achieved through the similarity between text features and image features. For example, for a given retrieval text, text features can be extracted through feature encoding, and image features of all pictures in an image material library, and the distance between the text features and the image features of all pictures is used as the similarity between the retrieval text and the corresponding pictures, so that the picture with the highest similarity is used as the target picture of retrieval.

[0035] However, due to the richness of the text content, the basic constituent unit thereof can include multiple different multi-element phrases, such as entity phrases, adjective phrases, etc. When extracting the text features of the text to be retrieved, the corresponding text feature extraction method often pays more attention to the entities in the text to be retrieved, and ignores other information in the text to be retrieved, so that the target picture of the retrieval may not be consistent with the text to be retrieved. For example, if the given text to be retrieved in the picture-text retrieval is "She walked through the street in the rain with an umbrella, suddenly stopped and looked up at the sky", the corresponding entities include "umbrella", "rain", "street", and "sky". Therefore, when determining the target picture according to the similarity between the text features and the image features, the target picture may include the umbrella, the rain, the street, and the sky, but not the person, which can be determined by "she" that the female person should also be included in the target picture. Therefore, when retrieving the picture corresponding to the text according to the above method, the picture retrieved may not be consistent with the text.

[0036] Based on the above problems, the embodiment of the present disclosure provides a picture-text retrieval method. An electronic device can determine a candidate picture library in combination with the similarity between the text to be retrieved and each picture in the picture library, and identify the target entity in the text to be retrieved based on the pre-established entity reference word, to identify the target picture corresponding to the target entity from the candidate picture library. In the embodiment of the present disclosure, the electronic device can not only establish a candidate picture library according to the similarity between the text to be retrieved and each picture in the picture library, but also further screen the candidate pictures in the candidate picture library according to the target entity in the text to be retrieved, thereby improving the accuracy of picture-text matching.

[0037] Figure 1 is an architecture diagram of an implementation environment according to an exemplary embodiment. As shown in Figure 1 The following picture-text retrieval method can be applied in the implementation environment. The implementation environment includes terminal 01 and electronic device 02. The terminal 01 and the electronic device 02 can be connected through a wireless network.

[0038] In an implementable manner, the terminal 01 can run an application program associated with the server 02, and based on the application program, the server 02 can be logged in to interact with the server 02. The application program can be a short video application program, a social network application program, or a browser. In the embodiment of the present disclosure, the application program is taken as a short video application program as an example, and the user can input the text to be retrieved in the short video application program to query the corresponding picture or video.

[0039] In an implementable manner, the electronic device 02 can be a server, which can be a server or a server cluster composed of multiple servers, or a cloud computing service center. The electronic device 02 can include a processor, a memory, and a network interface, etc.

[0040] In a possible implementation, the electronic device 02 can extract text features of the search text and image features of each picture in the picture library, and determine the similarity between the text features and the image features of each picture.

[0041] In an implementable manner, the terminal 01 is configured to provide voice and / or data connectivity services for a user. The terminal 01 can have different names, such as UE, end, terminal unit, terminal station, mobile station, mobile station, remote station, remote terminal, mobile device, wireless communication device, vehicle user equipment, terminal agent or terminal device, etc.

[0042] Optionally, the terminal 01 can be various handheld devices, vehicle-mounted devices, wearable devices, computers with communication functions, and the embodiments of the present disclosure do not make any limitation on this. For example, the handheld device can be a smart phone. The vehicle-mounted device can be a car navigation system. The wearable device can be a smart bracelet. The computer can be a personal digital assistant (PDA) computer, a tablet computer and a laptop computer.

[0043] Those skilled in the art should understand that the terminal described above is only an example, and other existing or future terminals, such as those applicable to the present disclosure, should also be included in the protection scope of the present disclosure and are hereby incorporated by reference.

[0044] Based on the above implementation environment, the text and picture retrieval method provided by the embodiments of the present disclosure has the implementation process as follows:

[0045] The text and picture retrieval method provided by the embodiments of the present disclosure will be described exemplarily in combination with various drawings.

[0046] Figure 2 is a flow chart of a text and picture retrieval method according to an exemplary embodiment. As shown in Figure 2 The text and picture retrieval method can be applied to the electronic device or terminal described above, and the method includes the following S11 to S15.

[0047] S11, the electronic device obtains a to-be-searched text.

[0048] The to-be-searched text can include a target entity and a target object, and the target object can be a person, an animal or an article, etc. The target entity can be used to indicate the attribute information of the target object. For example, when the target object is a person, the target entity can indicate the attributes of the target object corresponding to gender, age, fatness, etc. When the target object is an article, the target entity can indicate the attributes of the target object corresponding to shape, color, etc.

[0049] In a possible implementation manner, the image-text retrieval method can be executed by the electronic device, and the text to be retrieved can be acquired by the electronic device from the terminal. In another possible implementation manner, the image-text retrieval method can also be executed by the terminal, and the text to be retrieved can be determined by the terminal according to the input of the user.

[0050] In S12, the electronic device identifies the target entity in the text to be retrieved based on the pre-established entity pronoun library.

[0051] The entity pronoun library includes pronouns corresponding to entities of a preset type, and the target entity is an entity determined based on a target pronoun, and the target pronoun is a word in the text to be retrieved that matches the pronoun.

[0052] As a possible implementation manner, the entity pronoun library is a pre-established entity library, which is used to indicate the attribute information of various objects. For example, the entity can indicate the attributes of a person, or can indicate the attributes of an object.

[0053] For example, for a person, the corresponding entity can include attributes such as gender, age, and fatness. The entity corresponding to the gender can be "he", "she", "princess", "prince", "uncle", etc., the entity corresponding to the age can be "teenager", "middle-aged", "grandpa", "baby", etc., and the entity corresponding to the fatness can be "full", "plump", "slim", etc. These entities can only indicate one type of attribute of an object, or can indicate multiple types of attributes of an object, for example, "grandpa" can not only indicate that the gender of the person is male, but also indicate that the age of the person is greater than 40 years old, etc.

[0054] For the entity corresponding to the object, the attributes can include shape, color, etc. The entity corresponding to the shape can be "rectangle", "square", "circle", etc., and the entity corresponding to the color can be "black", "bright", "white", etc. Of course, different attributes can be included for different objects, and the attribute feature words in the above preset library are only exemplary, and in actuality, the attribute feature words for a person or an object can include more.

[0055] The above-mentioned preset type can indicate various attributes of an object, for example, when the object is a person, the preset type can be the above-mentioned attributes such as gender, age, and fatness. The pronoun corresponding to the entity can specifically indicate the attribute of the corresponding entity, for example, the entity is "he", and the pronoun corresponding to the entity is "male".

[0056] In a possible implementation manner, the entity pronoun library not only includes entities, but also includes pronouns corresponding to the entities. For example, the entity pronoun library can be as shown in Table 1:

[0057] Table 1

[0058] entity pronoun he gender "male" she gender "female" princess gender "female" grandpa gender "male" grandpa age "more than 40 years" … …

[0059] Of course, the above Table 1 is only exemplary, and the entity pronoun library can include more entities and their corresponding pronouns.

[0060] In a possible implementation, the entity pronoun library can also be divided into a plurality of sub-libraries according to different objects, for example, a character entity pronoun library, an animal entity pronoun library, and an article entity pronoun library can be divided in the entity pronoun library; further, these character entity pronoun library, animal entity pronoun library, and article entity pronoun library can be further divided, which is not limited by the embodiments of the present disclosure.

[0061] For example, if the text to be detected is "She walked through the rain street with an umbrella, and suddenly stopped and looked up at the sky", the target entity in the text to be detected includes "she", at this time it can be determined that the target picture corresponding to the text to be searched includes an object character, and the entity corresponding to the object character includes the gender "female".

[0062] For another example, if the text to be detected is "That red sports car not only has a streamlined shape, but also has an arc-shaped tail wing at the tail", at this time it can be determined that the target picture corresponding to the text to be searched includes an object sports car, and the entity corresponding to the object sports car includes the shape "streamlined" and "arc-shaped", and the color "red".

[0063] In some embodiments, when the target object is a character, the preset type here is used to indicate the gender or age of the character.

[0064] Since when the target object in the text to be detected is a character, the preset type of the corresponding entity can indicate the gender or age of the character, therefore when searching for the target picture later, the searched target picture can be more consistent with the description of the text to be searched, that is, the accuracy of searching for a character picture is improved.

[0065] S13, the electronic device determines the similarity between the text to be searched and each picture in the picture library.

[0066] As a possible implementation, the similarity between the text to be detected and each picture can be determined according to the text feature vector of the text to be searched and the image feature vector corresponding to each picture.

[0067] For example, if the picture library includes a first picture, a second picture, …, and an Nth picture, the image feature vector includes a first image feature vector, a second image feature vector, …, and an Nth image feature vector, the first picture corresponds to the first image feature vector, the second picture corresponds to the second image feature vector, …, and the Nth picture corresponds to the Nth image feature vector, the similarity between the to-be-detected text and the first picture can be determined according to the text feature vector of the to-be-detected text and the first image feature vector, the similarity between the to-be-detected text and the second picture can be determined according to the text feature vector of the to-be-detected text and the second image feature vector, …, and the similarity between the to-be-detected text and the Nth picture can be determined according to the text feature vector of the to-be-detected text and the Nth image feature vector.

[0068] In S14, the electronic device determines, as a candidate picture in the candidate picture library, a picture with a similarity greater than a first threshold.

[0069] As a possible implementation, the similarity in the embodiments of the present disclosure can be understood as the distance between the text feature vector and the image feature vector. The smaller the distance between the text feature vector and the image feature vector, the greater the similarity between them. Therefore, the picture with a similarity greater than a first threshold can be determined as a candidate picture, and the candidate picture library can be formed by these candidate pictures.

[0070] In S15, the electronic device determines, from the candidate picture library, a target picture corresponding to the target entity.

[0071] As a possible implementation, since the candidate pictures in the candidate picture library are determined according to the similarity between the to-be-detected text and each picture in the picture library, the determination of the similarity focuses on the object in the to-be-detected text, and ignores the entity in the to-be-detected text. Therefore, the candidate pictures in the candidate picture library are further screened according to the target entity in this step, and the target picture that matches the to-be-detected text can be determined.

[0072] In a possible implementation, when the candidate pictures corresponding to the target entity in the candidate picture library include only one, the one candidate picture can be determined as the target picture; when the candidate pictures corresponding to the target entity in the candidate picture library include multiple, the candidate picture with the highest similarity can be determined as the target picture.

[0073] The technical solutions provided by the above embodiments have at least the following beneficial effects: in the embodiments of the present disclosure, the electronic device can not only establish a candidate picture library according to the similarity between the to-be-detected text and each picture in the picture library, but also further screen the candidate pictures in the candidate picture library according to the target entity in the to-be-detected text, thereby improving the accuracy of the image-text matching.

[0074] In an implementable manner, in combination Figure 2 As shown in Figure 3 The S12 can be implemented by the following S121-S124.

[0075] In the S121, the electronic device reads the entity anaphora library, and matches the target anaphora in the text to be searched.

[0076] As a possible implementation manner, the entity anaphora library can be pre-stored in the storage space of the electronic device. After the electronic device obtains the text to be searched, the electronic device can read the entity anaphora library, and match the entity anaphora library with the text to be searched, to determine the target anaphora included in the text to be searched.

[0077] For example, if the text to be searched is "She walked through the street in the heavy rain, and suddenly stopped to look up at the sky", the entity anaphora library can determine that the anaphora included in the text to be searched is "She", and the target anaphora in the entity anaphora library is "female". The target anaphora is the target entity in the text to be searched.

[0078] In the S122, when the target anaphora does not exist, the electronic device determines the target entity as a default entity.

[0079] As a possible implementation manner, when the target anaphora does not exist in the text to be searched, the target entity of the text to be searched can be determined as a corresponding default entity in combination with the object included in the text to be searched. For example, when the object included in the text to be searched is a person, the corresponding default entity can include gender, age, fatness, etc. The default entity corresponding to the gender entity can be "male", and the default entity corresponding to the age entity can be "20-25 years old", etc.

[0080] Similarly, when other objects are included in the text to be searched, the default entity of the corresponding entity attribute can also be set. The default entity of the object can be set by the person skilled in the art according to experience.

[0081] For example, if the text to be searched is "She walked through the street in the heavy rain, and suddenly stopped to look up at the sky", the text to be searched does not include the target entity corresponding to the age entity, and therefore the age entity in the text to be searched can be determined as "20-25 years old".

[0082] In the S123, when the target anaphora exists, and the target anaphora corresponds to at least two preset types of entities, the electronic device determines the target entity as a target preset type of entity.

[0083] The target preset type entity is an entity type corresponding to a target pronoun with a highest frequency of occurrence in the text to be detected or a target pronoun with a highest order in the text to be detected.

[0084] In a possible implementation, when the text to be detected includes multiple target pronouns, the target pronouns can be contradictory to each other. For example, when the object in the text to be detected is a person, if the text to be detected includes multiple gender entities, such as "he", "she", and "woman", and the target pronouns correspond to two preset types of entities, such as "male" and "female", the target entity can be determined as the preset type of entity corresponding to the target pronoun with the highest order, for example, the gender entity "he" with the highest order is determined as the target preset type entity, indicating that the gender of the person in the target picture to be retrieved is "male".

[0085] In another possible implementation, for the above example, since the gender entities "she" and "woman" both indicate "female", the gender entity of the text to be detected can also be determined as "female", that is, the gender of the person in the target picture to be retrieved is indicated as "female".

[0086] In another possible implementation, when the text to be detected includes at least one target pronoun, if the target pronouns correspond to a preset type of entity, the target entity can be determined as the preset type of entity corresponding to the target pronouns. For example, the gender entities of the text to be detected include "she" and "woman", and the preset type of entity corresponding to the target pronouns is both "female", so the gender entity of the text to be detected can be determined as "female", that is, the gender of the person in the target picture to be retrieved is indicated as "female".

[0087] It should be noted that the preset type can indicate different types of the same entity, such as "male" and "female". The embodiments of the present disclosure are not limited to determining the target preset type entity by the above method, and a person skilled in the art can also determine the target preset type entity from the text to be detected by other manners, which are not limited by the present disclosure.

[0088] The technical solutions provided by the above embodiments have at least the following beneficial effects: The embodiments of the present disclosure can determine the target entity from the text to be detected to indicate the attribute information of the object in the text to be detected, provide retrieval indication for the target picture to be retrieved, and improve the accuracy of retrieving the picture.

[0089] In a possible implementation, the target entity is determined by the following method. For example, the target entity is determined by the following method.

[0090] In a possible implementation, the target entity is determined by the following method. Figure 3 For example, the target entity is determined by the following method. Figure 4As shown, S13 can be implemented by S131-S132.

[0091] S131, the electronic device determines the text feature vector of the text to be searched and the image feature vector of each picture in the picture library according to the trained text-image detection model.

[0092] As a possible implementation manner, the text-image detection model can include a text detection model and an image detection model. The text detection model can extract the text feature vector of the text to be searched, and the image detection model can extract the image feature vector of the picture.

[0093] The text detection model can extract the text feature vector of the text to be searched by one-hot encoding, by a linear discriminant analysis (LDA) topic model, by a pre-trained neural network model, by a Word2vec neural network, or by other manners, which are not limited in the embodiments of the present disclosure.

[0094] The image detection model can extract the image feature vector of the picture by a histogram of oriented gradient (HOG), by a local binary pattern (LBP), by a pre-trained neural network model, by a deep learning network of Image Net, or by a visual geometry group network (VGG), or by other manners, which are not limited in the embodiments of the present disclosure.

[0095] In another possible implementation manner, the text detection model and the image detection model in the embodiments of the present disclosure can both be Click Models. The extraction of the text feature vector and the image feature vector can be realized by training the Click Models. The text detection model and the image detection model in the embodiments of the present disclosure can be determined by referring to the technical means commonly used in the art, which are not limited in the present disclosure.

[0096] It should be noted that the text-image detection model in the embodiments of the present disclosure can be trained by referring to the technical means commonly used in the art, which will not be repeated here.

[0097] S132, the electronic device determines the similarity between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture.

[0098] As a possible implementation, the similarity between the text feature vector and the image feature vector can be determined by the distance between the two, and the distance between the two is determined as the corresponding similarity, that is, the similarity between the text to be searched and the corresponding picture.

[0099] The technical solutions provided by the above embodiments have at least the following beneficial effects: The disclosure embodiments determine the similarity between the text to be searched and the corresponding picture through the similarity between the text feature vector and the image feature vector, can filter out pictures similar to the text to be searched, narrow the range of searching target pictures, and improve the search efficiency.

[0100] In an implementable manner, in combination with Figure 4 As shown in Figure 5 The above S132 can be implemented by the following S1321-S1322.

[0101] S1321, the electronic device determines the cosine distance between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture.

[0102] S1322, the electronic device determines the cosine distance as the similarity between the text to be searched and each picture.

[0103] As a possible implementation, if the text feature vector corresponding to the text to be searched is (x1, x2, x3, …, x m ), the first image feature vector corresponding to the first picture is (y1, y2, y3, …, y m ), the cosine distance between the first text feature vector and the first image feature vector is:

[0104]

[0105] The cosine value cosθ above is the cosine distance between the text feature vector and the first image feature vector, that is, the cosine distance between the text to be searched and the first picture.

[0106] In another possible implementation, the text feature vectors may have different lengths than the image feature vectors. In this case, principal component analysis (PCA) can be used to reduce the dimensionality of the text or image feature vectors so that their lengths are the same. Then, the cosine distance between the text and image feature vectors can be determined using the above method. Of course, the dimensionality reduction processing of the text or image feature vectors here is merely exemplary. Those skilled in the art can also use other methods to process the text or image feature vectors to make their lengths the same; this disclosure does not limit this approach.

[0107] In this embodiment, the cosine distance between the text feature vector and the image feature vector is determined as the similarity between the text to be detected and the image. The larger the cosine distance between the two, the lower the similarity between them.

[0108] In one possible implementation, embodiments of this disclosure may also determine the similarity between text feature vectors and image feature vectors using the Euclidean distance. Since cosine distance better characterizes the directional differences between two vectors compared to Euclidean distance, embodiments of this disclosure use cosine distance to determine the similarity between two vectors. Of course, the similarity between text feature vectors and image feature vectors can also be determined in other ways, and is not limited to the cosine distance or Euclidean distance described above; embodiments of this disclosure do not impose such limitations.

[0109] The technical solutions provided by the above embodiments have at least the following beneficial effects: In the embodiments of this disclosure, the similarity between the text to be detected and the image is characterized by cosine distance, which can more accurately represent the similarity between the two.

[0110] In one feasible approach, the text to be retrieved also includes the target object; combined with Figure 5 ,like Figure 6 As shown, the above S15 can be specifically implemented through the following S151-S153.

[0111] S151. The electronic device determines target candidate images containing the target object from the candidate image library based on the trained object recognition model.

[0112] As one possible implementation, the image recognition model in this embodiment may include an object recognition model and an entity recognition model. Since the aforementioned candidate image library is determined solely based on the text feature vector corresponding to the text to be detected and the image feature vector of the image, the content of the candidate images is not filtered. Therefore, the candidate images can be further filtered using an object recognition model.

[0113] For example, if the object in the text to be detected is a person, the object recognition model can be a person detection model for screening the candidate pictures including the person from the candidate pictures; if the object in the text to be detected is a sports car, the object recognition model can be a vehicle detection model for screening the candidate pictures including the sports car from the candidate pictures.

[0114] Here, the candidate picture screened from the candidate picture library can be determined as the target candidate picture.

[0115] S152, when the target candidate picture does not exist, the electronic device determines the candidate picture in the candidate picture library with the highest similarity to the text to be searched as the target picture.

[0116] As a possible implementation, when all the pictures in the candidate picture library do not include the object in the text to be detected, the candidate picture in the candidate picture library with the highest similarity to the text to be searched can be determined as the target picture corresponding to the text to be detected.

[0117] S153, when the target candidate picture exists, the electronic device determines the target picture corresponding to the target entity from the target candidate picture according to the trained entity recognition model.

[0118] As a possible implementation, when the target candidate picture determined by the object recognition model includes only one, the target candidate picture can be determined as the target picture.

[0119] When the target candidate picture determined by the object recognition model includes multiple, the target picture more matched to the text to be detected can be further screened from the target candidate pictures according to the entity recognition model.

[0120] For example, when the object in the text to be detected is a person, the entity recognition model can include a gender recognition model and an age recognition model, and the target picture corresponding to the gender entity and the age entity in the text to be detected can be determined from the target candidate picture by the gender recognition model and the age recognition model. Of course, if the target candidate picture corresponding to the gender entity and the age entity screened at this time includes multiple, the target candidate picture with the highest similarity can be determined as the target picture.

[0121] In a possible implementation, to make the screened candidate picture more accurate, the object recognition model corresponding to the above-mentioned entity recognition model can further include other models. For example, when the object in the text to be detected is a person, the object recognition model can further include a face recognition model to assist the gender recognition model and the age recognition model to determine the gender and age of the person in the target candidate picture.

[0122] Of course, the above model is only exemplary, and the object recognition model and the entity recognition model can also include other models, and the embodiments of the present disclosure are not limited. And the object recognition model and the entity recognition model can be a pre-trained model, such as a pre-trained neural network model, etc., and the training process can refer to the technical means commonly used in the art, which will not be repeated here.

[0123] In some embodiments, each of the above models has a certain recognition order. For example, when the object included in the text to be detected is a person, the candidate picture can be recognized in the order of the person detection model, the face detection model, the gender detection model, and the age detection model. In the process of sequential recognition, if the previous model does not recognize the corresponding detection object, the target candidate picture with the highest similarity to the text to be detected in the target candidate picture at this time can be determined as the target picture.

[0124] The technical solutions provided by the above embodiments have at least the following beneficial effects: in the present disclosure, the combination of the object recognition model and the entity recognition model can further screen the candidate pictures, so that the target picture is more matched with the object and entity of the text to be detected, and the accuracy of the image-text retrieval is improved.

[0125] Figure 7 is a structural schematic diagram of an image-text retrieval device according to an exemplary embodiment, as shown in Figure 7 The image-text retrieval device 20 includes an acquisition module 201, an entity recognition module 202, a similarity determination module 203, a picture candidate module 204, and a picture recognition module 205.

[0126] The acquisition module 201 is configured to acquire the text to be searched. The acquisition module 201 can perform the above step S11.

[0127] The entity recognition module 202 is configured to identify the target entity in the text to be searched acquired by the acquisition module 201 based on a pre-established entity reference word library. The entity reference word library includes reference words corresponding to preset types of entities, and the target entity is an entity determined based on a target reference word. The target reference word is a word in the text to be searched that matches the reference word. The entity recognition module 202 can perform the above step S12.

[0128] The similarity determination module 203 is configured to determine the similarity between the text to be searched acquired by the acquisition module 201 and each picture in the picture library. The similarity determination module 203 can perform the above step S13.

[0129] The picture candidate module 204 is configured to determine the pictures with a similarity greater than a first threshold value as candidate pictures in the candidate picture library. The picture candidate module 204 can perform the above step S14.

[0130] The picture identification module 205 is configured to determine the target picture corresponding to the target entity from the candidate picture library determined by the picture candidate module 204. The picture identification module 205 can perform the step S15.

[0131] Optionally, the entity identification module 202 is specifically configured to: read an entity reference word library, and match the target reference word with the text to be searched; when the target reference word does not exist, determine the target entity as a default entity; when the target reference word exists and the target reference word corresponds to at least two types of preset entities, determine the target entity as a target preset type entity; the target preset type entity is an entity type corresponding to a target reference word with a highest ranking in the text to be detected, or an entity type corresponding to a target reference word with a highest occurrence frequency in the text to be detected; when the target reference word exists and the target reference word corresponds to one type of preset entity, determine the target entity as an entity type corresponding to the target reference word. The entity identification module 202 can perform the steps S121-S124.

[0132] Optionally, the similarity determination module 203 is specifically configured to: determine the text feature vector of the text to be searched and the image feature vector of each picture in the picture library according to the trained image-text detection model; and determine the similarity between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture. The similarity determination module 203 can perform the steps S131-S132.

[0133] Optionally, the similarity determination module 203 is specifically configured to: determine the cosine distance between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture; and determine the cosine distance as the similarity between the text to be searched and each picture. The similarity determination module 203 can perform the steps S1321-S1322.

[0134] Optionally, the text to be searched further includes a target object; and the picture identification module 205 is specifically configured to: determine a target candidate picture including the target object from the candidate picture library according to the trained object recognition model; when the target candidate picture does not exist, determine the candidate picture with the highest similarity to the text to be searched in the candidate picture library as the target picture; and when the target candidate picture exists, determine the target picture corresponding to the target entity from the target candidate picture according to the trained entity identification model. The picture identification module 205 can perform the steps S151-S153.

[0135] Optionally, the text to be searched further includes a target object, and the target object is a person, and the preset type is used to indicate the gender or age of the person.

[0136] As to the apparatus in the above-mentioned embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and thus will not be described in detail here.

[0137] Figure 8 is a structural schematic diagram of an electronic device 30 provided by the present disclosure. As Figure 8 , the electronic device 30 can include at least one processor 301 and a memory 303 for storing instructions executable by the processor 301. The processor 301 is configured to execute the instructions in the memory 303 to implement the image-text retrieval method in the above-mentioned embodiments. As an example, in combination with Figure 7 , the functions implemented by the acquisition module 201, the entity recognition module 202, the similarity determination module 203, the picture candidate module 204, and the picture recognition module 205 in the image-text retrieval apparatus are the same as the functions of the processor 301 in Figure 8 .

[0138] In addition, the electronic device 30 can further include a communication bus 302 and at least one communication interface 304.

[0139] The processor 301 can be a GPU, a micro processing unit, an ASIC, or one or more integrated circuits for controlling the execution of programs of the present disclosure.

[0140] The communication bus 302 can include a path for transmitting information between the above-mentioned components.

[0141] The communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0142] Memory 303 may be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit as a volatile storage medium within the GPU.

[0143] The memory 303 stores instructions for executing the present invention, and the processor 301 controls the execution of these instructions. The processor 301 executes the instructions stored in the memory 303 to implement the functions of the method disclosed herein.

[0144] In a specific implementation, as one example, processor 301 may include one or more GPUs, for example... Figure 8 GPU0 and GPU1 in the dataset.

[0145] In a specific implementation, as one example, the electronic device 30 may include multiple processors, such as... Figure 8 Processors 301 and 307 are described herein. Each of these processors may be a single-core (CPU) processor or a multi-core (GPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0146] In a particular implementation, as an example, the electronic device 30 can further include an output device 305 and an input device 306. The output device 305 is in communication with the processor 301 and can display information in a variety of ways. For example, the output device 305 can be a liquid crystal display (LCD), a light emitting diode (LED) display device, a cathode ray tube (CRT) display device, a projector, or the like. The input device 306 is in communication with the processor 301 and can accept user input in a variety of ways. For example, the input device 306 can be a mouse, a keyboard, a touch screen device, a sensor device, or the like.

[0147] Those skilled in the art can understand that the structure shown in the above embodiments does not constitute a limitation on the electronic device 30, and the electronic device 30 can include more or fewer components than shown, or combine certain components, or use different component arrangements. Figure 8

[0148] In addition, the present disclosure also provides a computer readable storage medium, the computer readable storage medium includes instructions, when the instructions are executed by the processor of the electronic device, the electronic device executes the image-text retrieval method provided by the above embodiments.

[0149] In addition, the present disclosure also provides a computer program product, the computer program product includes instructions, when the instructions are executed on the processor of the electronic device, the electronic device executes the image-text retrieval method provided by the above embodiments.

[0150] Other embodiments of the present disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the aspects of the present disclosure disclosed herein. It is intended that the present disclosure cover any and all variations of the present disclosure including those variations contained within the spirit and scope of the areas of art to which the present disclosure pertains. It is intended that the specification and examples be considered exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0151] It should be understood that the present disclosure is not limited to the precise structures as set forth above and shown in the drawings and that various modifications and changes can be made without departing from the scope thereof, which is set forth in the following claims.​

Claims

1. A method of retrieving text from an image, characterized by, The method comprises: acquiring a text to be searched, the text to be searched comprising a target entity and a target object, the target entity being used to indicate attribute information of the target object; reading an entity pronoun library, and determining a target pronoun by matching the text to be searched, the entity pronoun library comprising pronouns corresponding to entities of preset types, the target pronoun being a word in the text to be searched that matches a pronoun, and the entity pronoun library being used to indicate attribute information of various objects; when the target pronoun exists and corresponds to an entity of a preset type, determining the target entity as an entity type corresponding to the target pronoun, the target entity being an entity determined based on the target pronoun; determining a text feature vector of the text to be searched and an image feature vector of each image in an image library according to a trained image-text detection model; performing dimension reduction processing on the text feature vector or the image feature vector of each image by principal component analysis (PCA) so that the text feature vector and the image feature vector of each image have the same length; determining a similarity between the text to be searched and each image according to the text feature vector and the image feature vector of each image; taking an image with a similarity greater than a first threshold value as a candidate image in a candidate image library; determining a target image corresponding to the target entity from the candidate image library.

2. The document retrieval method according to claim 1, wherein, The method further comprises: when the target pronoun does not exist, determining the target entity as a default entity; when the target pronoun exists and corresponds to at least two entities of preset types, determining the target entity as a target preset type entity, the target preset type entity being an entity type corresponding to the target pronoun that ranks first in the text to be searched or an entity type corresponding to the target pronoun that appears most frequently in the text to be searched.

3. The document retrieval method according to claim 1, wherein, The determining of the similarity between the text to be searched and each image according to the text feature vector and the image feature vector of each image comprises: determining a cosine distance between the text to be searched and each image according to the text feature vector and the image feature vector of each image; determining the cosine distance as the similarity between the text to be searched and each image.

4. The document retrieval method according to claim 3, wherein, The text to be searched further comprises a target object; and the determining of the target image corresponding to the target entity from the candidate image library comprises: determining a target candidate image comprising the target object from the candidate image library according to a trained object recognition model; when the target candidate image does not exist, determining a candidate image with the highest similarity to the text to be searched in the candidate image library as the target image; when the target candidate image exists, determining the target image corresponding to the target entity from the target candidate image according to a trained entity recognition model.

5. The method of claim 1, wherein, The text to be searched further comprises a target object, the target object being a person, and the preset types being used to indicate a gender or an age of the person.

6. A document retrieval apparatus characterized by comprising: The device comprises: The acquisition module is configured to acquire a text to be searched, the text to be searched comprising a target entity and a target object, the target entity being used to indicate attribute information of the target object; The entity recognition module is configured to read an entity pronoun library, and match a target pronoun in the text to be searched, the entity pronoun library comprising pronouns corresponding to entities of preset types, the target pronoun being a word in the text to be searched that matches a pronoun, and the entity pronoun library being used to indicate attribute information of objects of various types; The entity recognition module is further configured to, when the target pronoun exists and corresponds to an entity of a preset type, determine the target entity as an entity type corresponding to the target pronoun, and the target entity being an entity determined based on the target pronoun; The similarity determination module is configured to determine a text feature vector of the text to be searched and an image feature vector of each picture in a picture library according to a trained image-text detection model; The similarity determination module is further configured to perform dimension reduction processing on the text feature vector or the image feature vector of each picture by principal component analysis (PCA), so that the text feature vector and the image feature vector of each picture have the same length; The similarity determination module is further configured to determine a similarity between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture; The picture candidate module is configured to determine, as a candidate picture in a candidate picture library, a picture having a similarity greater than a first threshold value and determined by the similarity determination module; The picture recognition module is configured to determine, from the candidate picture library determined by the picture candidate module, a target picture corresponding to the target entity determined by the entity recognition module.

7. The document retrieval apparatus according to claim 6, wherein The entity recognition module is further configured to: determine the target entity as a default entity when the target pronoun does not exist; and determine the target entity as a target preset type entity when the target pronoun exists and corresponds to at least two entities of preset types, the target preset type entity being an entity type corresponding to the target pronoun that has a highest ranking in the text to be searched or an entity type corresponding to the target pronoun that appears most frequently in the text to be searched.

8. The document retrieval apparatus according to claim 6, wherein The similarity determination module is further configured to: determine a cosine distance between the text to be searched and each picture according to the text feature vector and the image feature vector of each picture; and determine the cosine distance as the similarity between the text to be searched and each picture.

9. The apparatus according to claim 8, wherein The text to be searched further comprises a target object, and the picture recognition module is specifically configured to: determine, according to a trained object recognition model, a target candidate picture comprising the target object from the candidate picture library; determine, when the target candidate picture does not exist, a candidate picture having a highest similarity to the text to be searched in the candidate picture library as the target picture; determine, when the target candidate picture exists, the target picture corresponding to the target entity from the target candidate picture according to a trained entity recognition model.

10. The document retrieval apparatus according to claim 6, wherein The text to be retrieved further includes a target object, the target object being a person, and the preset type is used to indicate a gender or an age of the person.

11. An electronic device, comprising: The method comprises: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the image-text retrieval method of any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises instructions; when the instructions are executed by a processor of an electronic device, the electronic device performs the image-text retrieval method of any one of claims 1 to 5.

13. A computer program product, characterised in that, The computer program product comprises instructions; when the instructions are executed on a processor of an electronic device, the electronic device performs the image-text retrieval method of any one of claims 1 to 5.

Images