Image-text retrieval method and device, equipment and medium

Abstract

The invention relates to the technical field of image-text retrieval, and discloses an image-text retrieval method, device and equipment and a medium, which can be used for performing named entity recognition on a query statement input by a user to obtain a corresponding text tag and extracting a text feature of the text tag. For each image stored in the image library, carrying out multi-type target detection on the image, detecting an object corresponding to at least one set object type in the image, determining that the text tag comprises a keyword corresponding to the target object type, extracting image features of the target object in the image, and extracting the target object from the image; and determining a first similarity between the image feature of the target object and the text feature of the text tag, and determining a matching degree between the text tag and the image. And determining a target image matched with the query statement according to the matching degree between the text tag and each image in the image library. The image features of the target object extracted from the images of the image library are key feature information. According to the method, the implementation modes of image-text retrieval can be effectively enriched.

Description

Technical Field

[0001] The present invention relates to the field of image and text retrieval technology, and in particular to an image and text retrieval method, device, equipment and medium. Background Art

[0002] With the development of artificial intelligence, information retrieval technology continues to improve.

[0003] Image-text retrieval aims to find relevant image-text pairs or image-text matching content based on the input image or text query from a database containing image and text information.

[0004] Related technologies generally first obtain user input text, then extract text features from the input text, search the database for the text or image with the highest similarity to the input text features, and determine the found text or image as the content that matches the input text. The implementation method of image-text retrieval is relatively simple. Summary of the Invention

[0005] The present invention provides a method, device, equipment and medium for image and text retrieval, which are used to solve the defect that the implementation method of image and text retrieval in the related art is relatively single, and effectively enrich the implementation method of image and text retrieval.

[0006] In a first aspect, the present invention provides a method for image and text retrieval, comprising: Perform named entity recognition on the query statement entered by the user to obtain the corresponding text label; Extracting text features of the text label; For any one of a plurality of images stored in a gallery, multi-type target detection is performed on the image to detect at least one object corresponding to a set object type in the image, determine that the text label includes a keyword corresponding to the target object type, extract image features of the target object from the image, determine a first similarity between the image features of the target object and the text features of the text label, and determine a matching degree between the text label and the image based on the first similarity; wherein the target object type is one of the at least one set object type, and the target object is an object corresponding to the target object type; A target image matching the query statement is determined based on the matching degree between the text label and each of the images in the image library.

[0007] Optionally, determining the matching degree between the text label and the image according to the first similarity includes: Extracting overall image features of the image, and determining a second similarity between the text features of the text label and the overall image features; performing text recognition on the image to obtain target text in the image, extracting text features of the target text, and determining a third similarity between the text features of the text label and the text features of the target text; A maximum similarity among the first similarity, the second similarity, and the third similarity is determined and used as a matching degree between the text label and the image.

[0008] Optionally, after detecting at least one object corresponding to a set object type in the image, the method further includes: If it is determined that the text label does not include any keyword corresponding to the set object type, the maximum similarity between the second similarity and the third similarity is determined as the matching degree between the text label and the image.

[0009] Optionally, determining a target image matching the query statement according to a matching degree between the text label and each image in the image library includes: Determining at least one target matching degree that is not less than a preset matching degree threshold value among the matching degrees between the text label and each of the images; The image corresponding to each target matching degree is determined as a target image matching the query statement, and each target image is output.

[0010] Optionally, the multiple images stored in the gallery are railway scene-related images, and the at least one set object type includes at least one of a person, a car, and a backpack.

[0011] Optionally, when the target object includes multiple objects corresponding to the target object type, extracting image features of the target object from the image includes: For any of the target objects, extracting image features of each of the objects in the image; Determining a first similarity between the image feature of the target object and the text feature of the text label includes: For any one of the target objects, determining a target similarity between an image feature of the object and a text feature of the text label; Taking the target similarity between the image feature of each of the target objects and the text feature of the text label as the first similarity; The determining of the maximum similarity among the first similarity, the second similarity and the third similarity as the matching degree between the text label and the image includes: A maximum similarity is determined among all the target similarities of the first similarities, the second similarities, and the third similarities, and is used as the matching degree between the text label and the image.

[0012] In a second aspect, the present invention provides a graphic and text retrieval device, comprising: The entity recognition unit is used to perform named entity recognition on the query statement input by the user and obtain the corresponding text label; A first extraction unit, configured to extract text features of the text label; an object detection unit, configured to perform multi-type object detection on any one of a plurality of images stored in the image library, so as to detect at least one object corresponding to a set object type in the image; a second extraction unit, configured to determine that the text label includes a keyword corresponding to a target object type, and then extract image features of the target object from the image; wherein the target object type is one of the at least one set object type, and the target object is an object corresponding to the target object type; a first determining unit, configured to determine a first similarity between the image feature of the target object and the text feature of the text label; a second determining unit, configured to determine a degree of matching between the text label and the image based on the first similarity; The third determining unit is configured to determine a target image matching the query statement according to a matching degree between the text label and each of the images in the image library.

[0013] Optionally, the second determining unit is further configured to: Extracting overall image features of the image, and determining a second similarity between the text features of the text label and the overall image features; performing text recognition on the image to obtain target text in the image, extracting text features of the target text, and determining a third similarity between the text features of the text label and the text features of the target text; A maximum similarity among the first similarity, the second similarity, and the third similarity is determined and used as a matching degree between the text label and the image.

[0014] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image and text retrieval method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image-text retrieval method of the first aspect or any corresponding embodiment thereof.

[0016] The image-text retrieval method, apparatus, device, and medium provided by the present invention can perform named entity recognition on a query statement input by a user, obtain a corresponding text label, and extract text features of the text label. For any image among the multiple images stored in a gallery, multi-type target detection is performed on the image to detect at least one object corresponding to a set object type in the image, determine that the text label includes keywords corresponding to the target object type, extract image features of the target object in the image, determine a first similarity between the image features of the target object and the text features of the text label, and determine the matching degree between the text label and the image based on the first similarity; wherein, the target object type is one of the at least one set object type, and the target object is an object corresponding to the target object type. Based on the matching degree between the text label and each image in the gallery, a target image that matches the query statement is determined. The image features of the target object extracted from the images in the gallery by the present invention are key feature information. The present invention can effectively enrich the implementation methods of image-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A flowchart of a method for image and text retrieval provided by an embodiment of the present invention; Figure 2 A flowchart of another image-text retrieval method provided by an embodiment of the present invention; Figure 3 A flowchart of named entity recognition provided by an embodiment of the present invention; Figure 4 A schematic structural diagram of a graphic and text retrieval device provided by an embodiment of the present invention; Figure 5 A schematic structural diagram of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0020] The following combination Figure 1-Figure 3 The image-text retrieval method of the present invention is described.

[0021] like Figure 1 As shown, this embodiment proposes a first image-text retrieval method, which may include the following steps: S101: Perform named entity recognition on the query statement input by the user to obtain the corresponding text label.

[0022] It should be noted that this embodiment can be applied to electronic devices such as mobile phones, tablet computers, laptop computers, and desktop computers.

[0023] The user may input a query statement via text or via voice.

[0024] Specifically, when a user enters a query statement via text, this embodiment can directly use the user-entered text as the query statement. When a user enters a query statement via voice, this embodiment can first perform voice recognition, convert the user's voice input into corresponding text, and use the converted text as the input query statement.

[0025] Specifically, this embodiment can use a large language model to perform named entity recognition on the query statement input by the user, and extract key information in the query statement as a text label.

[0026] Optionally, this embodiment can determine the object of interest based on the application scenario. For example, in a railway-related scenario, the objects of interest include people, backpacks, and cars. This embodiment can use a large language model to perform named entity recognition on the query statement input by the user, detect the text corresponding to the object of interest in the query statement and use it as key information, and then construct a corresponding text label based on the detected key information and semantics. For example, the query statement input by the user is "My car is black". This embodiment uses a large language model to perform named entity recognition on the query statement. The key information detected includes car and black. Based on the semantics, "black car" can be constructed as a text label.

[0027] S102: Extract text features of the text label.

[0028] Specifically, this embodiment can use the Qllama module in the multimodal large model internvl as a text feature extractor, and use the text feature extractor to extract text features from text tags. Of course, this embodiment can also use other types of text feature extractors.

[0029] S103 : For any image among the multiple images stored in the image library, perform multi-type object detection on the image to detect at least one object corresponding to a set object type in the image.

[0030] Specifically, a plurality of images may be stored in the gallery.

[0031] The set object type may be the object type focused on and set in this embodiment, such as a person, a backpack, or a car.

[0032] Specifically, this embodiment can adopt yolov11m as a target detection module, and use the target detection module to perform multi-type target detection on the image to detect objects corresponding to each set object type in the image. For example, when the set object types include people, backpacks and cars, this embodiment can detect people, backpacks and cars in the image.

[0033] Optionally, the multiple images stored in the gallery are railway scene-related images, and at least one set object type includes at least one of a person, a car, and a backpack.

[0034] S104: If it is determined that the text tag includes a keyword corresponding to the target object type, image features of the target object are extracted from the image, wherein the target object type is one of at least one set object type, and the target object is an object corresponding to the target object type.

[0035] Specifically, the target object type is a set object type, and the target object is an object corresponding to the target object type.

[0036] The keyword corresponding to the target object type may be a key text used to define the target object type. For example, when the target object type is car, the corresponding keyword may be car.

[0037] Specifically, this embodiment can extract the image features of the target object contained in the image when it is determined that the text tag includes a keyword corresponding to the target object type. For example, this embodiment can extract the image features of the car contained in the image when it is determined that the text tag includes the word "car".

[0038] It should be noted that this embodiment can use the ViT module of the internvl large model as an image feature extractor to extract image features. Specifically, this embodiment can first obtain a region image corresponding to the target object in the image, input the region image into the image feature extractor to extract image features, and obtain the image features of the target object.

[0039] S105: Determine a first similarity between the image feature of the target object and the text feature of the text tag.

[0040] The first similarity is the feature similarity between the image feature of the target object and the text feature of the text tag. It is understandable that the feature similarity can be the cosine similarity between the two features.

[0041] S106: Determine a matching degree between the text label and the image according to the first similarity.

[0042] Specifically, in this embodiment, the first similarity may be directly determined as the matching degree between the text label and the image.

[0043] It is understandable that the target object is a key object that has an important relationship with the text label. The image features of the target object extracted from the image in this embodiment are key feature information. Compared with the related art that extracts the overall image features from the image and only uses the overall image features to determine matching images, this embodiment can directly extract and display the key feature information, explicitly showing the information of the key object, avoiding the compression or hiding of the key feature information. Using this key feature information to search for matching images has a better recall effect for images containing multiple targets.

[0044] Optionally, step S106 may include: Extracting overall image features of the image and determining a second similarity between the text features of the text label and the overall image features; performing text recognition on the image to obtain target text in the image, extracting text features of the target text, and determining a third similarity between the text features of the text label and the text features of the target text; The maximum similarity among the first similarity, the second similarity, and the third similarity is determined and used as the matching degree between the text label and the image.

[0045] Specifically, in this embodiment, the image may be input into an image feature extractor to extract image features and obtain the overall image features of the image.

[0046] The second similarity is the feature similarity between the text feature of the text label and the overall image feature.

[0047] Specifically, this embodiment performs optical character recognition (OCR) on the image to extract text content in the image, namely, the target text.

[0048] Optionally, in this embodiment, the PP-OCRv3 model may be selected as the OCR recognition module, and the OCR recognition module may be used to extract text content from the image.

[0049] Specifically, the third similarity is the feature similarity between the text features of the text tag and the text features of the target text.

[0050] Optionally, in other image-text retrieval methods proposed in this embodiment, after detecting at least one object corresponding to a set object type in the image, the following steps may be further included: If it is determined that the text label does not include any keyword corresponding to the set object type, the maximum similarity between the second similarity and the third similarity is determined as the matching degree between the text label and the image.

[0051] Specifically, after detecting at least one object corresponding to a set object type in the image, this embodiment may determine whether a keyword corresponding to the set object type exists in the image. If it is determined that no keyword corresponding to the set object type exists, the maximum similarity among the second and third similarities may be determined, and this maximum similarity may be determined as the degree of match between the text label and the image. If it is determined that a keyword corresponding to the target object type exists, the maximum similarity among the first, second, and third similarities may be determined as the degree of match between the text label and the image.

[0052] S107: Determine a target image that matches the query statement based on the matching degree between the text label and each image in the image library.

[0053] Specifically, this embodiment may determine the degree of matching between the text label and each image in the image gallery, and then determine the target image that matches the query statement based on all the determined matching degrees.

[0054] Optionally, in other image-text retrieval methods proposed in this embodiment, step S107 may include: Among the matching degrees between the text label and each image, determining at least one target matching degree that is not less than a preset matching degree threshold; The image corresponding to each target matching degree is determined as the target image matching the query statement, and each target image is output.

[0055] The preset matching degree threshold can be set by technicians according to actual conditions and is not limited in this embodiment.

[0056] Specifically, this embodiment may also determine the maximum matching degree among the matching degrees between the text label and each image in the gallery, and determine the image corresponding to the maximum matching degree as the image matching the query statement.

[0057] It should be noted that the image-text retrieval method in the related art relies on feature extraction of the entire image to obtain the overall image features of the image, and then calculates the similarity with the text features of the user input text. However, this method is heavily dependent on the capabilities of the feature extractor. If the feature extractor fails to accurately encode the semantic information in the image, especially the multi-target semantics in the image, the retrieval effect will be greatly affected. Moreover, when the related art uses a feature extractor for feature extraction, the information of the target object in the image is also encoded into the extracted feature vector. However, this implicit representation method easily hides the information of key objects, increasing the difficulty of image-text retrieval to find images containing corresponding objects.

[0058] This embodiment can extract the text contained in the image and further extract the text features of the text while extracting the overall image features of the image. It can also perform feature extraction on each target in the image to avoid compression or hiding of the semantic information of important targets, and can explicitly express the information of key objects. Thereafter, the image matching the text label can be determined based on the overall image features of the image, the text features of the text in the image, and the image features of each specific target in the image. It has a good recall effect for images containing multiple targets, effectively enriching the image-text retrieval method and improving the accuracy of image-text retrieval.

[0059] The image-text retrieval method proposed in this embodiment can perform named entity recognition on the query statement input by the user, obtain the corresponding text label, and extract the text features of the text label. For any image among the multiple images stored in the gallery, multi-type target detection is performed on the image to detect at least one object corresponding to a set object type in the image, determine that the text label includes keywords corresponding to the target object type, then extract the image features of the target object from the image, determine the first similarity between the image features of the target object and the text features of the text label, and determine the matching degree between the text label and the image based on the first similarity; wherein the target object type is one of the at least one set object type, and the target object is the object corresponding to the target object type. Based on the matching degree between the text label and each image in the gallery, the target image that matches the query statement is determined. The image features of the target object extracted from the images in the gallery in this embodiment are key feature information. This embodiment can effectively enrich the implementation methods of image-text retrieval.

[0060] based on Figure 1This embodiment proposes a second image-text retrieval method. In this method, when the target object includes multiple objects corresponding to the target object type, step S104 may include: Determine that the text label includes keywords corresponding to the target object type, and for any of the target objects, extract image features of each object in the image; At this time, step S105 may include: For any one of the target objects, determining a target similarity between an image feature of the object and a text feature of a text label; The target similarity between the image features of each object in the target object and the text features of the text label is taken as the first similarity as a whole.

[0061] At this time, the maximum similarity among the first similarity, the second similarity and the third similarity is determined as the matching degree between the text label and the image, including The maximum similarity is determined among all target similarities of the first similarity, the second similarity, and the third similarity, and is used as the matching degree between the text label and the image.

[0062] It should be noted that when the target object includes multiple objects corresponding to the target object type, that is, when there are multiple objects corresponding to the target object type in the image, this embodiment can extract the image features of the multiple objects in the image respectively, and then determine the target similarity between the image features of the multiple objects and the text features of the text labels respectively.

[0063] Specifically, this embodiment may determine the maximum similarity among all target similarities, second similarities, and third similarities, and use the maximum similarity as the matching degree between the text label and the image.

[0064] In order to better illustrate the image-text retrieval method, this embodiment introduces Example 1.

[0065] Example 1 is a graphic retrieval process that accurately indexes multiple targets in an image. Based on the original image features, that is, the overall image features are extracted, features are also extracted for each target in the image to avoid the semantic information of important targets being compressed or hidden. It can be mainly used in railway-related scenarios, and text labels can include categories such as people, backpacks, and cars. Figure 2 The specific implementation steps are as follows: Step 1: Select optimal image and text feature extraction modules. Specifically, you can use the ViT module from the internvl large model as the image feature extraction module, and the Qllama module from the internvl large model as the text feature extraction module. Then, select the object detection module and the OCR recognition module. Specifically, you can use the yolov11m model as the object detection module and the PP-OCRv3 model as the OCR recognition module.

[0066] Step 2: Text feature extraction module, such as Figure 3 As shown, the query instruction input by the user is obtained, which includes the query sentence "My car is black". Then, the language large model is used to perform named entity recognition and the result "black car" is used as the text label. Then, a text feature extractor (that is, a text feature extraction module, such as the Qllama structure of internvl) is used to extract the text feature vector_text from the text label, which is specifically a 1*d-dimensional text feature vector.

[0067] Step three, image feature extraction module, use the image feature extraction module to process the picture i in the gallery, and obtain the overall image feature vector, that is, the 1*d-dimensional feature vector vector_ori, where d is the feature dimension. Then use the target detection module to detect the categories of people, cars and backpacks in picture i. Assume that a person, b cars and c backpacks are detected, and the image feature vectors corresponding to each object are obtained. The object targets are cropped out separately and feature extraction is performed using the image feature extraction module. Finally, the OCR recognition module is used to perform OCR recognition and text extraction on picture i. Assume that p texts are extracted and input into the text extractor to obtain p text feature vectors. It should be noted that in this embodiment, the feature dimensions of the image feature vector and the text feature vector can both be d.

[0068] Step 4: Calculate the cosine similarity module, use the text feature extraction module to extract the text feature vector_text from the text label, and determine whether there are keywords such as people, cars, and backpacks in the text label.

[0069] If the text label contains a keyword, all image feature vectors of the corresponding type of object extracted from picture i are determined. For example, when it is determined that the text label contains the keyword "car", the image feature vectors of all cars extracted from picture i, i.e., b car image feature vectors, can be determined. Then, the similarities between the b car image feature vectors and the text feature vector_text are calculated, the similarities between the p text feature vectors and the text feature vector_text are calculated, and the similarities between the overall image feature vector of picture i and the text feature vector_text are calculated. A total of (b+p+1) similarities can be obtained, and the maximum similarity among these similarities is determined as the matching degree between the text label and picture i. In this embodiment, the image feature vectors of the b cars, the p text feature vectors, and the overall image feature vector of picture i can be concatenated to obtain a (b+p+1)*d-dimensional feature vector. Then, the text feature vector of the text label can be directly dot-multiplied by the (b+p+1)*d-dimensional feature vector to obtain the cosine similarity between the text feature vector of the text label and the image feature vectors of the b cars, the p text feature vectors, and the overall image feature vector of picture i.

[0070] If there are no keywords in the text tag, only the similarities between the p text feature vectors and the text feature vector_text are calculated, as well as the similarities between the overall image feature vector of picture i and the text feature vector_text. The maximum similarity among these similarities is determined as the matching degree between the text tag and picture i. In this embodiment, the p text feature vectors and the overall image feature vector of picture i are first concatenated to obtain a (p+1)*d-dimensional feature vector, and then the text feature vector of the text tag is directly multiplied by the (p+1)*d-dimensional feature vector to obtain the cosine similarity between the text feature vector of the text tag and the p text feature vectors and the overall image feature vector of picture i.

[0071] It should be noted that if the text tag contains specified keywords, the corresponding target feature vector is matched first according to the corresponding object category, and the similarity is calculated by combining the overall image features of picture i and p text feature vectors.

[0072] Step 5: Execute the process of steps 3 and 4 for each image in the gallery to obtain the matching degree between each image in the gallery and the text label.

[0073] Step 6: Determine the retrieval recall images based on the matching degree between each image in the gallery and the text label, and select the most relevant images.

[0074] The image-text retrieval method proposed in this embodiment can extract the text contained in the image and then extract the text features of the text while extracting the overall image features of the image. It can perform feature extraction on each target in the image to avoid compression or hiding of the semantic information of important targets, and can explicitly express the information of key objects. Based on the overall image features of the image, the text features of the text in the image, and the image features of each specific target in the image, it can find images in the image library that match the text labels. It has a good recall effect for images containing multiple targets and can effectively improve the accuracy of image-text retrieval.

[0075] like Figure 4 As shown, this embodiment provides a graphic and text retrieval device, which may include: The entity recognition unit 101 is used to perform named entity recognition on the query statement input by the user to obtain the corresponding text label; A first extraction unit 102 is used to extract text features of text tags; The object detection unit 103 is configured to perform multi-type object detection on any one of the multiple images stored in the image library to detect at least one object corresponding to a set object type in the image; The second extraction unit 104 is configured to extract image features of the target object from the image if the text tag includes a keyword corresponding to the target object type; wherein the target object type is one of at least one set object type, and the target object is an object corresponding to the target object type; A first determining unit 105 is configured to determine a first similarity between an image feature of a target object and a text feature of a text tag; A second determining unit 106 is configured to determine a matching degree between the text label and the image based on the first similarity; The third determining unit 107 is configured to determine a target image that matches the query statement based on a matching degree between the text label and each image in the image library.

[0076] It should be noted that the processing of the entity recognition unit 101, the first extraction unit 102, the target detection unit 103, the second extraction unit 104, the first determination unit 105, the second determination unit 106 and the third determination unit 107 and their beneficial effects can be referred to respectively. Figure 1 Steps S101 to S107 in the above are not described in detail.

[0077] Optionally, the second determining unit 106 is further configured to: Extracting overall image features of the image and determining a second similarity between the text features of the text label and the overall image features; performing text recognition on the image to obtain target text in the image, extracting text features of the target text, and determining a third similarity between the text features of the text label and the text features of the target text; The maximum similarity among the first similarity, the second similarity, and the third similarity is determined and used as the matching degree between the text label and the image.

[0078] Optionally, the above device further includes: The fourth determining unit is configured to, after detecting at least one object corresponding to a set object type in the image, determine that the text label does not include any keyword corresponding to the set object type, and then determine the maximum similarity between the second similarity and the third similarity as the matching degree between the text label and the image.

[0079] Optionally, the third determining unit 107 is further configured to: Among the matching degrees between the text label and each image, determining at least one target matching degree that is not less than a preset matching degree threshold; The image corresponding to each target matching degree is determined as the target image matching the query statement, and each target image is output.

[0080] Optionally, the multiple images stored in the gallery are railway scene-related images, and at least one set object type includes at least one of a person, a car, and a backpack.

[0081] Optionally, when the target objects include multiple objects corresponding to the target object type, the second extraction unit 104 is further configured to: for any object among the target objects, extract image features of each object in the image; The first determining unit 105 is further configured to: determine, for any one of the target objects, a target similarity between the image feature of the object and the text feature of the text label; and use the target similarity between the image feature of each object in the target object and the text feature of the text label as a first similarity; The second determining unit 106 is further configured to determine a maximum similarity among all target similarities of the first similarity, the second similarity, and the third similarity, and use the maximum similarity as the matching degree between the text label and the image.

[0082] The image-text retrieval device proposed in this embodiment can perform named entity recognition on the query statement input by the user, obtain the corresponding text label, and extract the text features of the text label. For any image among the multiple images stored in the gallery, multi-type target detection is performed on the image to detect at least one object corresponding to a set object type in the image, determine that the text label includes keywords corresponding to the target object type, then extract the image features of the target object in the image, determine the first similarity between the image features of the target object and the text features of the text label, and determine the matching degree between the text label and the image based on the first similarity; wherein the target object type is one of the at least one set object type, and the target object is the object corresponding to the target object type. Based on the matching degree between the text label and each image in the gallery, the target image that matches the query statement is determined. The image features of the target object extracted from the images in the gallery in this embodiment are key feature information. This embodiment can effectively enrich the implementation methods of image-text retrieval.

[0083] The image and text retrieval device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0084] The embodiment of the present invention also provides a computer device having the above Figure 4 The image and text retrieval device shown.

[0085] See also Figure 5 , a structural diagram of a computer device provided by an optional embodiment of the present invention, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0086] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0087] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0088] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0089] The memory 20 may include volatile memory, such as random access memory. The memory may also include non-volatile memory, such as flash memory, a hard disk, or a solid-state drive. The memory 20 may also include a combination of the above types of memory.

[0090] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0091] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for image and text retrieval, characterized in that: include: Perform named entity recognition on the query statement entered by the user to obtain the corresponding text label; Extracting text features of the text label; For any image among the multiple images stored in the gallery, multi-type target detection is performed on the image to detect at least one object corresponding to a set object type in the image, determine that the text label includes a keyword corresponding to the target object type, extract image features of the target object in the image, determine a first similarity between the image features of the target object and the text features of the text label, and determine a matching degree between the text label and the image according to the first similarity; wherein the target object type is one of the at least one set object type, and the target object is an object corresponding to the target object type; According to the matching degree between the text label and each of the images in the gallery, a target image matching the query statement is determined.

2. The method according to claim 1, characterized in that: The determining the matching degree between the text label and the image according to the first similarity includes: Extracting overall image features of the image, and determining a second similarity between the text features of the text label and the overall image features; Performing text recognition on the image to obtain a target text in the image, extracting text features of the target text, and determining a third similarity between the text features of the text label and the text features of the target text; The maximum similarity among the first similarity, the second similarity and the third similarity is determined and used as the matching degree between the text label and the image.

3. The method according to claim 2, characterized in that After detecting at least one object corresponding to a set object type in the image, the method further includes: If it is determined that the text label does not include any keyword corresponding to the set object type, the maximum similarity between the second similarity and the third similarity is determined as the matching degree between the text label and the image.

4. The method according to claim 1, characterized in that: The step of determining a target image matching the query statement according to a matching degree between the text label and each of the images in the gallery includes: Among the matching degrees between the text label and each of the images, determining at least one target matching degree that is not less than a preset matching degree threshold; The image corresponding to each of the target matching degrees is determined as a target image matching the query statement, and each of the target images is output.

5. The method according to claim 1, characterized in that The multiple images stored in the gallery are images related to railway scenes, and the at least one set object type includes at least one of a person, a car, and a backpack.

6. The method according to claim 2, characterized in that When the target object includes a plurality of objects corresponding to the target object type, extracting the image features of the target object from the image includes: For any of the target objects, extracting image features of each of the objects in the image; The determining a first similarity between the image feature of the target object and the text feature of the text label includes: For any of the target objects, determining a target similarity between an image feature of the object and a text feature of the text label; Taking the target similarity between the image feature of each of the target objects and the text feature of the text label as the first similarity; The determining of the maximum similarity among the first similarity, the second similarity and the third similarity and using the maximum similarity as the matching degree between the text label and the image includes: A maximum similarity is determined among all the target similarities of the first similarities, the second similarities, and the third similarities, and is used as a matching degree between the text label and the image.

7. A graphic and text retrieval device, characterized in that: include: An entity recognition unit is used to perform named entity recognition on the query statement input by the user to obtain the corresponding text label; A first extraction unit, used to extract text features of the text label; An object detection unit, configured to perform multi-type object detection on any one of the plurality of images stored in the gallery, so as to detect at least one object corresponding to a set object type in the image; A second extraction unit is used to determine that the text label includes a keyword corresponding to a target object type, and then extract image features of the target object in the image; wherein the target object type is one of the at least one set object type, and the target object is an object corresponding to the target object type; A first determining unit, configured to determine a first similarity between the image feature of the target object and the text feature of the text label; a second determining unit, configured to determine a degree of matching between the text label and the image according to the first similarity; The third determining unit is used to determine a target image matching the query statement according to a matching degree between the text label and each of the images in the gallery.

8. The device according to claim 7, characterized in that The second determining unit is further configured to: Extracting overall image features of the image, and determining a second similarity between the text features of the text label and the overall image features; Performing text recognition on the image to obtain a target text in the image, extracting text features of the target text, and determining a third similarity between the text features of the text label and the text features of the target text; The maximum similarity among the first similarity, the second similarity and the third similarity is determined and used as the matching degree between the text label and the image.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image-text retrieval method according to any one of claims 1 to 6 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image-text retrieval method according to any one of claims 1 to 6.

Images