Image retrieval method and device, electronic equipment and readable storage medium
By finding matching feature vectors from the scene feature vector database and the target-level feature vector database, and integrating global and local search results, the problem of difficult to accurately retrieve images under specified scene events in the prior art is solved, and a more efficient image retrieval effect is achieved.
Patent Information
- Application Number
- CN202411933904.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-24
- Publication Date
- 2025-04-08
AI Technical Summary
It is difficult to accurately retrieve images under specified scene events, especially when describing complex scene events, and it is impossible to effectively match related images.
By obtaining the feature vector of text information to be retrieved, the matching feature vectors are found from the scene feature vector database and the target-level feature vector database, and the search results of the scene global and target local areas are fused to select the target image matching the text information.
The accuracy and effectiveness of image retrieval under specified scene events are achieved, and the scene feature information can be better utilized, thereby improving the search effect of scene event images.
Smart Images
Figure CN119782572A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing, and in particular to an image retrieval method, device, electronic device and readable storage medium. Background Art
[0002] The so-called text-based image search refers to retrieving images that match the text description information from multiple images based on the text description information. At present, in the application scenario of text-based image search, the search is performed by extracting keywords related to the text description information and the target object. For example, if the text description information indicates that you want to search for "a person wearing a helmet", the image search can be achieved by searching for helmets or people, etc.
[0003] However, this method is difficult to deal with image search under scene events. For example, when the text description information indicates that you want to search for "people fighting", the above method can only search for images related to people, and cannot accurately search for images of people in the scene event of "fighting".
[0004] Therefore, being able to accurately retrieve images under specified scene events is a technical problem that needs to be solved urgently. Summary of the invention
[0005] In view of this, embodiments of the present application provide an image retrieval method, an apparatus, an electronic device, and a readable storage medium, so as to accurately retrieve images under specified scene events.
[0006] The present application provides an image retrieval method, the method comprising:
[0007] Obtaining a feature vector of the text information to be retrieved;
[0008] Find a first type of feature vector that matches the feature vector from a scene feature vector database; the first type of feature vector includes N reference feature vectors; N is greater than 1;
[0009] Determine a corresponding image for each of the N reference feature vectors;
[0010] Find a second type of feature vector that matches the feature vector from the target-level feature vector database; the second type of feature vector includes M candidate feature vectors; M is greater than 1;
[0011] Determine a corresponding image for each of the M candidate feature vectors; wherein the scene feature vector database and the target-level feature vector database are obtained by processing each image to be stored based on the multimodal large model, the global text features of the image and the local text features corresponding to each detected target object on the image;
[0012] From the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors, K target images matching the text information are selected and output; K is greater than 1.
[0013] The present application also provides an image retrieval device, which includes:
[0014] An acquisition module, used for acquiring a feature vector of text information to be retrieved;
[0015] A first retrieval module is used to find a first type of feature vector matching the feature vector from a scene feature vector database; the first type of feature vector includes N reference feature vectors; N is greater than 1;
[0016] Determine a corresponding image for each of the N reference feature vectors;
[0017] The second retrieval module is used to find a second type of feature vector matching the feature vector from the target-level feature vector database; the second type of feature vector includes M candidate feature vectors; M is greater than 1;
[0018] Determine a corresponding image for each of the M candidate feature vectors; wherein the scene feature vector database and the target-level feature vector database are obtained by processing each image to be stored based on the multimodal large model, the global text features of the image and the local text features corresponding to each detected target object on the image;
[0019] The determination module is used to select and output K target images matching the text information from the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors; K is greater than 1.
[0020] An embodiment of the present application also provides an electronic device, including: a processor and a memory for storing computer program instructions, and when the computer program instructions are executed by the processor, the processor executes the steps of the above method.
[0021] An embodiment of the present application also provides a machine-readable storage medium, which stores computer program instructions. When the computer program instructions are executed, the steps of the above method can be implemented.
[0022] It can be seen from the above technical scheme that in the embodiment of the present application, N reference feature vectors matching the feature vector of the text information to be retrieved are first obtained from the scene feature vector database, and then M candidate feature vectors matching the feature vector are obtained from the target-level feature vector database. Then, the final K target images are obtained based on the N reference feature vectors and the M candidate feature vectors. In this way, the retrieval results of the scene global and the target local are fused to obtain more accurate retrieval results.
[0023] Furthermore, especially the final K target images also draw on the N reference feature vectors found from the scene feature vector database, which can make full use of the scene information in the scene feature vector database, thereby improving the retrieval effect of scene event picture classes, that is, achieving accurate retrieval of images under specified scene events. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A flowchart of a method for establishing a database provided in an embodiment of the present application;
[0025] Figure 2 A schematic diagram of a flow chart of an image retrieval method provided in an embodiment of the present application;
[0026] Figure 3 A flowchart of a method is provided for an embodiment of the present application;
[0027] Figure 4 A schematic diagram of the structure of the device provided in the embodiment of the present application;
[0028] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0029] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.
[0030] Before introducing the method provided in the embodiment of the present application, the establishment of a scene feature vector database and an object-level feature vector database is first described.
[0031] See also Figure 1 , Figure 1 A schematic diagram of a method for establishing a database provided in an embodiment of the present application. The method is used to establish a scene feature vector database and a target-level feature vector database.
[0032] like Figure 1 As shown, the process may include the following steps:
[0033] S101, for each image to be stored, input the image into an open target detection model for target detection to obtain a target detection result; the target detection result of any image at least includes the position information of each detected target object in the image.
[0034] In this embodiment, an open-vocabulary object detection model is used to detect each target object in an input image and identify the category and position information of each target object.
[0035] S102, input the image and the target detection result of the image to the trained multimodal large model to obtain the global text features corresponding to the image and the local text features corresponding to each detected target object on the image.
[0036] In this embodiment, the multimodal large model after training has the functions of image feature extraction and modality conversion, such as converting image modality into text modality.
[0037] In this embodiment, there are many ways to implement the multimodal large model to process the image and the target detection result of the image. For example, as an embodiment, the multimodal large model can be implemented in a specific way to process the image and the target detection result of the image as follows: according to the position information of each target object, a local image corresponding to each target object is obtained. For example, the position information of each target object is represented by a detection frame, and the image of the area framed by the detection frame is used as a local image, or the detection frame is expanded outward, and the image of the expanded area is used as a local image, for example, the detection frame is expanded outward at a ratio of 1.2 times with the center of the detection frame as the center, and the image in the expanded detection frame is used as a local image.
[0038] The multimodal large model extracts features from the image and the local images of each target object, respectively, to obtain the global image features corresponding to the image and the local image features corresponding to each target object. The global image features and the local image features are modally transformed from the image feature modality to the text feature modality to obtain the global text features corresponding to the image and the local text features corresponding to each target object on the image.
[0039] It should be noted that, from the above description, the open target detection model has the ability to detect all kinds of target objects commonly seen in images. By using it for target detection, each target can be accurately obtained, laying a solid foundation for the subsequent multimodal large model to extract more accurate local text features. In this way, it helps to establish more accurate local text features and provides a more accurate basis for subsequent retrieval.
[0040] S103, extracting scene feature information from the global text features, and generating a feature scene feature vector of the image based on the scene feature information; the feature scene feature vectors of each image constitute a scene feature vector database.
[0041] In this embodiment, scene feature information is extracted from global text features, and then the scene feature information is converted into a scene feature vector (for example, illustratively, the scene feature information is converted into a scene feature vector using an Embedding model) to obtain the scene feature vector of the image. If a scene feature vector database has not been established, a scene feature vector database is established and the scene feature vector is stored in the scene feature vector database. If the scene feature vector database has not been established, the scene feature vector is directly stored in the scene feature vector database.
[0042] In this embodiment, the scene feature information includes at least: On-Screen Display (OSD) information of the image acquisition device that acquires the image in the global text feature, and context text within a specified range centered on the scene-type keyword in the global text feature.
[0043] For example, the specific implementation method of extracting scene feature information from global text features can be: if OSD information is identified in the global text features, then the OSD information is extracted to obtain scene feature information. If a keyword related to the scene, such as bicycle, is identified in the global text features, then the context text within a specified range centered on the keyword is determined as scene feature information.
[0044] S104, extracting target-level feature information associated with the target object from each local text feature, and generating each target-level feature vector based on each target-level feature information; each target-level feature vector of each image constitutes the target-level feature vector database.
[0045] In this embodiment, target-level feature information associated with the target object is extracted from each local text feature, where the target-level feature information at least includes category information of the target object. Each target-level feature information is then converted into a target-level feature vector (for example, the target-level feature information is converted into a target-level feature vector using an Embedding model), thereby obtaining each target-level feature vector corresponding to the image. If it is determined that the target-level feature vector database has not been established, a target-level feature vector database is established, and each target-level feature vector is added to the target-level feature vector database. If the target-level feature vector database has not been established, the scene feature vector is directly stored in each target-level feature vector to the target-level feature vector database.
[0046] Furthermore, in one embodiment, after executing step S103, the method further includes: storing the correspondence between the image identifier and the scene feature vector of the image in a scene feature vector database. In this way, the scene feature vector database and the target-level feature vector database do not need to store the image itself. In subsequent retrieval, after the scene feature vector is retrieved, the corresponding image can be determined directly based on the correspondence, which can improve the retrieval speed in the scene feature vector.
[0047] Similarly, after executing step S103, the method further includes: storing the correspondence between the plurality of target-level text features, the image identifier of the image, and the object identifier of each target object of the image in the target-level feature vector database. In this way, the target-level feature vector database does not need to store the image itself, and in subsequent retrieval, after the target-level feature vector is retrieved, the corresponding image can be determined directly based on the correspondence, which can improve the retrieval speed in the target-level scene feature vector.
[0048] So far, completed Figure 1 The process shown.
[0049] pass Figure 1 In the process shown, in the process of establishing the scene feature vector database and the target-level feature vector database, target detection is performed through an open target detection model, and then a multimodal large model is used to construct more accurate and target-oriented local text features based on the identified target objects. On this basis, more accurate scene feature vectors and target-level feature vectors that are more relevant to the target object are obtained, which provides a solid foundation for subsequent image searches under scene events.
[0050] After introducing the establishment of the scene feature vector database and the target-level feature vector database, the retrieval method provided in the embodiment of the present application is described in detail below:
[0051] See also Figure 2 , Figure 2 A flowchart of the image retrieval method provided in an embodiment of the present application.
[0052] like Figure 2 As shown, the process may include the following steps:
[0053] S201, obtaining a feature vector of text information to be retrieved.
[0054] In this embodiment, text information is converted into feature vectors for retrieval, and then feature vector matching is performed in a feature vector database and a target-level feature vector database, which can improve the semantic matching capability of the retrieval and improve the accuracy of the retrieval.
[0055] In this embodiment, the text information to be retrieved is converted into a feature vector through the Embedding model.
[0056] S202, searching a first type of feature vector that matches the feature vector from a scene feature vector database; the first type of feature vector includes N reference feature vectors; and N is greater than 1.
[0057] In this embodiment, based on the first similarity value between each scene feature vector and the feature vector in the scene feature vector database, N reference feature vectors matching the feature vector are found from the scene feature vector database. For example, the first similarity value between each scene feature vector and the feature vector is calculated, and the first N scene feature vectors are sorted in descending order according to the first similarity value, and the first N scene feature vectors are selected as the above N reference feature vectors.
[0058] S203 , determining a corresponding image for each of the N reference feature vectors.
[0059] In this embodiment, it is known that when the scene feature vector database is established, the correspondence between the image identification and the scene feature vector of each image has been stored in the scene feature vector database. Then the specific implementation method of determining the image corresponding to each reference feature vector can be: obtaining the image identification corresponding to each reference feature vector.
[0060] S204, searching a second type of feature vector matching the feature vector from a target-level feature vector database; the second type of feature vector includes M candidate feature vectors.
[0061] In this example, see Figure 3 Combined with Figure 1 From the description of the illustrated embodiment, it can be seen that the scene feature vector database and the target-level feature vector database are obtained by respectively processing each image to be stored in the database based on the global text features of the image and the local text features corresponding to each detected target object on the image using a multimodal large model.
[0062] In this embodiment, according to the second similarity value between each target-level feature vector and the feature vector in the target-level feature vector database, M candidate feature vectors matching the feature vector are found from the target-level feature vector database. For example, the second similarity value between each target-level feature vector and the feature vector is calculated, and the second similarity values are sorted in descending order, and the first M target-level feature vectors are selected as the above-mentioned M candidate feature vectors.
[0063] S205 , determining a corresponding image for each of the M candidate feature vectors.
[0064] In this embodiment, it is known that when establishing the scene feature vector database, the corresponding relationship between the image identification of each image, the multiple target-level text features corresponding to the image, and the object identification of each target object in the image has been stored in the target-level feature vector database. Then the specific implementation method of determining the target object corresponding to each candidate feature vector and the image to which each target object belongs can be: obtaining the target identification and image identification corresponding to M candidate feature vectors.
[0065] S206, selecting and outputting K target images matching the text information from the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors; K is greater than 1.
[0066] The specific implementation of step S206 will be described in detail later in the text in the form of a specific embodiment, and will not be repeated here.
[0067] So far, completed Figure 2 The process shown.
[0068] pass Figure 2 As shown in the process, in the embodiment of the present application, N reference feature vectors matching the feature vector of the text information to be retrieved are first obtained from the scene feature vector database, and then M candidate feature vectors matching the feature vector are obtained from the target-level feature vector database. Then, the final K target images are obtained based on the N reference feature vectors and the M candidate feature vectors. In this way, the retrieval results of the scene global and the target local are fused to obtain more accurate retrieval results.
[0069] Furthermore, especially the final K target images also draw on the N reference feature vectors found from the scene feature vector database, which can make full use of the scene information in the scene feature vector database, thereby improving the retrieval effect of scene event picture classes, that is, achieving accurate retrieval of images under specified scene events.
[0070] The following is a detailed description of the above-mentioned K target images:
[0071] As an embodiment, for each image in the union of images corresponding to N reference feature vectors and images corresponding to M candidate feature vectors.
[0072] If the image is one of the images corresponding to the N reference feature vectors but not one of the images corresponding to the M candidate feature vectors, the first similarity value between the reference feature vector corresponding to the image and the feature vector is determined as the fusion similarity value of the image.
[0073] If the image is not one of the images corresponding to the N reference feature vectors, but is one of the images corresponding to the M candidate feature vectors, then the second similarity value between each candidate feature vector and the feature vector in at least one candidate feature vector corresponding to the image is obtained, and the largest second similarity value is determined as the fusion similarity value of the image.
[0074] If the image is one of the images corresponding to the N reference feature vectors and one of the images corresponding to the M candidate feature vectors, a first similarity value between the reference feature vector corresponding to the image and the feature vector, a second similarity value between each candidate feature vector and the feature vector in at least one candidate feature vector corresponding to the image, and a maximum similarity value between the first similarity value and the second similarity value is determined as a fusion similarity value of the image;
[0075] The images in the union are sorted in descending order according to the fusion similarity values, and the images corresponding to the first K fusion similarity values are determined as the K target images.
[0076] In a further embodiment, the image is represented by an image ID, and the specific implementation method of obtaining K target images is:
[0077] Obtain the union of the image IDs corresponding to the N reference feature vectors and the image IDs corresponding to the M candidate feature vectors. For each image ID, if the ID only comes from the image IDs corresponding to the N reference feature vectors, then the first similarity value between the reference feature vector corresponding to the image ID and the feature image is determined as the fusion similarity value corresponding to the image ID.
[0078] If the ID only comes from the image ID corresponding to the M candidate feature vectors, the candidate feature vector carrying the image ID in the M candidate feature vectors having the largest second similarity value with the feature image is determined as the fusion similarity value corresponding to the image ID.
[0079] If the ID comes from both the image ID corresponding to the M candidate feature vectors and the image ID corresponding to the N reference feature vectors, a specified weighted operation is performed on the first similarity value between the reference feature vector corresponding to the image ID and the feature image, and the second similarity value with the largest value between the candidate feature vector carrying the image ID in the M candidate feature vectors and the feature image to obtain the fused similarity value corresponding to the image ID.
[0080] Arrange the image IDs in descending order according to the fusion similarity value, select the first K image IDs, and determine the images represented by these K image IDs as K target images.
[0081] At this point, the description of the method provided in this embodiment is completed. The following describes the device provided in this embodiment of the application:
[0082] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of the device provided in the embodiment of the present application. Figure 4 As shown, the device is applied to an Internet of Things gateway, and the device 400 includes: an acquisition module 401, a first retrieval module 402, a second retrieval module 403 and a determination module 404.
[0083] The acquisition module 401 is used to obtain the feature vector of the text information to be retrieved.
[0084] The first retrieval module 402 is used to find a first type of feature vector that matches the feature vector from the scene feature vector database; the first type of feature vector includes N reference feature vectors; N is greater than 1;
[0085] Determine a corresponding image for each of the N reference feature vectors;
[0086] The second retrieval module 403 is used to search for a second type of feature vector that matches the feature vector from the target-level feature vector database; the second type of feature vector includes M candidate feature vectors; M is greater than 1;
[0087] Determine a corresponding image for each of the M candidate feature vectors; wherein the scene feature vector database and the target-level feature vector database are obtained by processing each image to be stored based on the multimodal large model, the global text features of the image and the local text features corresponding to each detected target object on the image;
[0088] The determination module 404 is used to select and output K target images with matching text information from the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors; K is greater than 1.
[0089] As an example,
[0090] Obtaining the scene feature vector database and the target-level feature vector database is achieved through the following steps:
[0091] For each image to be stored, the image is input into the open object detection model for object detection to obtain the object detection result;
[0092] Input the image and the target detection result of the image into the trained multimodal large model to obtain the global text features corresponding to the image and the local text features corresponding to each detected target object in the image;
[0093] Extracting scene feature information from global text features, and generating a feature scene feature vector of the image based on the scene feature information; the feature scene feature vectors of each image constitute a scene feature vector database;
[0094] Target-level feature information associated with the target object is extracted from each local text feature, and each target-level feature vector is generated based on each target-level feature information; each target-level feature vector of each image constitutes a target-level feature vector database.
[0095] As an embodiment, the target detection result of any image includes at least the position information of each detected target object in the image;
[0096] The multimodal large model processes the image and the object detection results of the image through the following steps:
[0097] According to the position information of each target object, a local image corresponding to each target object is obtained;
[0098] Extract features from the image and the local images of each target object respectively to obtain global image features corresponding to the image and local image features corresponding to each target object;
[0099] The global image features and each local image feature are modally transformed to obtain the global text features corresponding to the image and the local text features corresponding to each target object on the image.
[0100] As an embodiment, the scene feature information includes at least: OSD information displayed on the screen of the image acquisition device that acquires the image in the global text feature, and context text within a specified range centered on the scene-type keyword in the global text feature.
[0101] As an embodiment, after obtaining the characteristic scene feature vector and the target level feature vector, obtaining the scene feature vector database and the target level feature vector database further includes the following steps:
[0102] storing the corresponding relationship between the image identifier and the scene feature vector of the image in a scene feature vector database;
[0103] The correspondence between the plurality of target-level text features, the image identifier of the image, and the object identifier of each target object in the image is stored in a target-level feature vector database.
[0104] As an embodiment, selecting K target images for text information matching from images corresponding to N reference feature vectors and images corresponding to M candidate reference feature vectors includes:
[0105] For each image in the union of the images corresponding to the N reference feature vectors and the images corresponding to the M candidate feature vectors;
[0106] If the image is one of the images corresponding to the N reference feature vectors and is not one of the images corresponding to the M candidate feature vectors, then a first similarity value between the reference feature vector corresponding to the image and the feature vector is determined as a fusion similarity value of the image;
[0107] If the image is not one of the images corresponding to the N reference feature vectors, but is one of the images corresponding to the M candidate feature vectors, then a second similarity value between each candidate feature vector and the feature vector in at least one candidate feature vector corresponding to the image is obtained, and the largest second similarity value is determined as the fusion similarity value of the image;
[0108] If the image is one of the images corresponding to the N reference feature vectors and one of the images corresponding to the M candidate feature vectors, a first similarity value between the reference feature vector corresponding to the image and the feature vector, a second similarity value between each candidate feature vector and the feature vector in at least one candidate feature vector corresponding to the image, and a maximum similarity value between the first similarity value and the second similarity value is determined as a fusion similarity value of the image;
[0109] The images in the union are sorted in descending order according to the fusion similarity values, and the images corresponding to the first K fusion similarity values are determined as the K target images.
[0110] So far, completed Figure 4 Structural description of the device shown.
[0111] See also Figure 5 , Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present application. Figure 5 As shown, the hardware structure may include: a processor and a machine-readable storage medium, the machine-readable storage medium storing machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the method disclosed in the above example of this application.
[0112] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the method disclosed in the above example of the present application can be implemented.
[0113] Exemplarily, the above-mentioned machine-readable storage medium can be any electronic, magnetic, optical or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium can be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.
[0114] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. An image retrieval method, characterized in that: The method comprises: Obtaining a feature vector of the text information to be retrieved; Finding a first type of feature vector matching the feature vector from a scene feature vector database; the first type of feature vector includes N reference feature vectors; N is greater than 1; Determine a corresponding image for each of the N reference feature vectors; Finding a second type of feature vector matching the feature vector from a target-level feature vector database; the second type of feature vector includes M candidate feature vectors; M is greater than 1; Determine a corresponding image for each of the M candidate feature vectors; wherein the scene feature vector database and the target-level feature vector database are obtained by processing each image to be stored based on the multimodal large model, the global text features of the image and the local text features corresponding to each detected target object on the image; From the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors, K target images matching the text information are selected and output; K is greater than 1.
2. The method according to claim 1, characterized in that: Obtaining the scene feature vector database and the target-level feature vector database is achieved by the following steps: For each image to be stored, the image is input into an open object detection model for object detection to obtain an object detection result; Inputting the image and the target detection result of the image into the trained multimodal large model to obtain the global text feature corresponding to the image and the local text feature corresponding to each detected target object on the image; Extracting scene feature information from the global text features, and generating a feature scene feature vector of the image based on the scene feature information; The characteristic scene feature vectors of each image constitute the scene feature vector database; Target-level feature information associated with the target object is extracted from each local text feature, and each target-level feature vector is generated based on each target-level feature information; each target-level feature vector of each image constitutes the target-level feature vector database.
3. The method according to claim 2, characterized in that The target detection result of any image includes at least the position information of each detected target object in the image; The multimodal large model processes the image and the target detection result of the image through the following steps: According to the position information of each target object, a local image corresponding to each target object is obtained; Extracting features from the image and the local images of the target objects respectively to obtain global image features corresponding to the image and local image features corresponding to the target objects; Modality conversion is performed on the global image features and the local image features to obtain global text features corresponding to the image and local text features corresponding to each target object on the image.
4. The method according to claim 2, characterized in that: The scene feature information includes at least: OSD information displayed on the screen of the image acquisition device that acquires the image in the global text feature, and context text within a specified range centered on the scene-type keyword in the global text feature.
5. The method according to claim 2, characterized in that: After obtaining the characteristic scene feature vector and the target level feature vector, obtaining the scene feature vector database and the target level feature vector database further comprises the following steps: Storing the corresponding relationship between the image identifier of the image and the scene feature vector in the scene feature vector database; The correspondence between the plurality of target-level text features, the image identifier of the image, and the object identifier of each target object in the image is stored in the target-level feature vector database.
6. The method according to claim 1, characterized in that The selecting K target images matching the text information from the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors comprises: For each image in the union of the images corresponding to the N reference feature vectors and the images corresponding to the M candidate feature vectors; If the image is one of the images corresponding to the N reference feature vectors and is not one of the images corresponding to the M candidate feature vectors, then a first similarity value between the reference feature vector corresponding to the image and the feature vector is determined as a fusion similarity value of the image; If the image is not one of the images corresponding to the N reference feature vectors, but is one of the images corresponding to the M candidate feature vectors, then obtaining a second similarity value between each candidate feature vector in at least one candidate feature vector corresponding to the image and the feature vector, and determining the largest second similarity value as the fusion similarity value of the image; If the image is one of the images corresponding to the N reference feature vectors and one of the images corresponding to the M candidate feature vectors, a first similarity value between the reference feature vector corresponding to the image and the feature vector, a second similarity value between each candidate feature vector in at least one candidate feature vector corresponding to the image and the feature vector is obtained, and a maximum similarity value between the first similarity value and the second similarity value is determined as a fusion similarity value of the image; The images in the union are sorted in descending order according to the fusion similarity values, and the images corresponding to the first K fusion similarity values are determined as the K target images.
7. An image retrieval device, characterized in that: The device comprises: An acquisition module, used for acquiring a feature vector of text information to be retrieved; A first retrieval module is used to search for a first type of feature vector matching the feature vector from a scene feature vector database; the first type of feature vector includes N reference feature vectors; N is greater than 1; Determine a corresponding image for each of the N reference feature vectors; A second retrieval module is used to search for a second type of feature vector matching the feature vector from a target-level feature vector database; the second type of feature vector includes M candidate feature vectors; M is greater than 1; Determine a corresponding image for each of the M candidate feature vectors; wherein the scene feature vector database and the target-level feature vector database are obtained by processing each image to be stored based on the multimodal large model, the global text features of the image and the local text features corresponding to each detected target object on the image; A determination module is used to select and output K target images matching the text information from the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors; K is greater than 1.
8. The device according to claim 7, characterized in that The scene feature vector database and the target level feature vector database are obtained by the following steps: For each image to be stored, the image is input into an open object detection model for object detection to obtain an object detection result; Inputting the image and the target detection result of the image into the trained multimodal large model to obtain the global text feature corresponding to the image and the local text feature corresponding to each detected target object on the image; Extracting scene feature information from the global text features, and generating a feature scene feature vector of the image based on the scene feature information; The characteristic scene feature vectors of each image constitute the scene feature vector database; Extracting target-level feature information associated with the target object from each local text feature, generating each target-level feature vector based on each target-level feature information; each target-level feature vector of each image constitutes the target-level feature vector database; and / or, The target detection result of any image includes at least the position information of each detected target object in the image; The multimodal large model processes the image and the target detection result of the image through the following steps: According to the position information of each target object, a local image corresponding to each target object is obtained; Extracting features from the image and the local images of the target objects respectively to obtain global image features corresponding to the image and local image features corresponding to the target objects; Performing modal conversion on the global image features and the local image features to obtain global text features corresponding to the image and local text features corresponding to each target object on the image; and / or, The scene feature information at least includes: OSD information displayed on the screen of the image acquisition device that acquires the image in the global text feature, and context text within a specified range centered on the scene-type keyword in the global text feature; and / or, After obtaining the characteristic scene feature vector and the target level feature vector, obtaining the scene feature vector database and the target level feature vector database further comprises the following steps: Storing the corresponding relationship between the image identifier of the image and the scene feature vector in the scene feature vector database; storing the correspondence between the plurality of target-level text features, the image identifier of the image, and the object identifier of each target object in the image in the target-level feature vector database; and / or, The selecting K target images matching the text information from the images corresponding to the N reference feature vectors and the images corresponding to the M candidate reference feature vectors comprises: For each image in the union of the images corresponding to the N reference feature vectors and the images corresponding to the M candidate feature vectors; If the image is one of the images corresponding to the N reference feature vectors and is not one of the images corresponding to the M candidate feature vectors, then a first similarity value between the reference feature vector corresponding to the image and the feature vector is determined as a fusion similarity value of the image; If the image is not one of the images corresponding to the N reference feature vectors, but is one of the images corresponding to the M candidate feature vectors, then obtaining a second similarity value between each candidate feature vector in at least one candidate feature vector corresponding to the image and the feature vector, and determining the largest second similarity value as the fusion similarity value of the image; If the image is one of the images corresponding to the N reference feature vectors and one of the images corresponding to the M candidate feature vectors, a first similarity value between the reference feature vector corresponding to the image and the feature vector, a second similarity value between each candidate feature vector in at least one candidate feature vector corresponding to the image and the feature vector is obtained, and a maximum similarity value between the first similarity value and the second similarity value is determined as a fusion similarity value of the image; The images in the union are sorted in descending order according to the fusion similarity values, and the images corresponding to the first K fusion similarity values are determined as the K target images.
9. An electronic device, characterized in that: The electronic device includes: Processor; and A computer-readable storage medium, wherein computer program instructions are stored in the computer-readable storage medium, and when the computer program instructions are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, which, when executed by a processor, enable the processor to perform the steps of any one of the methods of claims 1 to 6.