Image acquisition methods, devices, electronic devices, storage media and products

By extracting keywords and encoding features from user-input descriptions, and combining image labels and feature vector matching, the problem of quickly locating and selecting detailed target images in electronic devices is solved, achieving more accurate image search and selection.

CN123068673APending Publication Date: 2026-11-10BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202510592975.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-08
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

In electronic devices, it is difficult for users to quickly locate and obtain specific image content, especially when there is too much storage space. Existing technologies cannot accurately search for and select target images containing detailed attributes.

Method used

By extracting keywords and encoding features from the user-input description information, and combining image label matching and image feature vector matching, the first image that matches the image label with the keywords is determined, and the second image that matches the text feature vector in terms of features is determined. The intersection of these two images is then used as the target image.

Benefits of technology

This technology enables the accurate location and selection of target images that match the user's input description within an image set, improving search accuracy and user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123068673A_ABST
    Figure CN123068673A_ABST
Patent Text Reader

Abstract

This disclosure relates to an image acquisition method, apparatus, electronic device, storage medium, and product. The image acquisition method includes: extracting keywords from user-input descriptive information to obtain m keywords, and encoding the m keywords to obtain text feature vectors. From a set of images to be retrieved, i first images whose image labels match the m keywords are identified, and j second images from the set of images to be retrieved whose image feature vectors match the text feature vectors are identified, where i and j are positive integers. The image labels represent the image content of the corresponding image, and the image feature vectors represent the image features of the corresponding image. The intersection of the i first images and the j second images is determined as p target images, and these p target images are acquired, where p is a positive integer. This disclosure enables more accurate location of target images corresponding to user-input descriptive information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of computer application technology, and in particular to image acquisition methods, devices, electronic devices, storage media and products. Background Technology

[0002] As the storage space of smart electronic devices increases, users will store a large amount of information through the local storage space of electronic devices. Users can store a large amount of image content in the local space of electronic devices, which can include images taken by electronic devices and images downloaded by electronic devices.

[0003] However, when a user needs to retrieve a specific image from the local storage of an electronic device, but the device has too many images stored, the user cannot quickly locate and retrieve that specific image, thus affecting the user experience. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an image acquisition method, apparatus, electronic device, storage medium, and product.

[0005] According to a first aspect of the present disclosure, an image acquisition method is provided, comprising: extracting keywords from user-input descriptive information to obtain m keywords, and performing feature encoding on the m keywords to obtain text feature vectors, wherein the descriptive information is used to describe a target object in a target image to be acquired, and the m keywords respectively correspond to different image contents possessed by the target object, and m is a positive integer; determining i first images in a set of images to be retrieved whose image tags match the m keywords, and determining j second images in the set of images to be retrieved whose image feature vectors match the text feature vectors, wherein i and j are positive integers, wherein the image tags are used to characterize the image content of the corresponding image, and the image feature vectors are used to characterize the image features of the corresponding image; determining p target images from the intersection of the i first images and the j second images, and acquiring the p target images, wherein p is a positive integer.

[0006] According to some embodiments of this disclosure, determining i first images in the image set to be retrieved whose image tags match the m keywords includes: for each image in the image set to be retrieved, determining n keywords from the m keywords that match the corresponding image tag, where n is a positive integer; for each image in the image set to be retrieved, in response to n being greater than a quantity threshold, determining the corresponding image as the first image; wherein the quantity threshold is determined based on m.

[0007] According to some embodiments of this disclosure, the image set to be retrieved includes k images, where k is a positive integer. The step of determining j second images in the image set to be retrieved that match the text feature vector includes: for each image in the image set to be retrieved, determining the similarity between the image feature vector and the text feature vector of each image, obtaining k similarities corresponding to each of the k images; sorting the k similarities in descending order to obtain sorted k similarities; and determining the j images corresponding to the j most prominent similarities among the sorted k similarities as the j second images.

[0008] According to some embodiments of this disclosure, after acquiring the p target images, the method further includes: for each of the p target images, determining n image tags that match n keywords in the image tags corresponding to the target image; for each of the p target images, determining the image region where the target image is located in the target image according to the correspondence between the corresponding n image tags and the image content, and identifying the target object in the target image.

[0009] According to some embodiments of this disclosure, the image tag corresponding to each image in the image set to be retrieved is included in a pre-built image tag library, and the image feature vector corresponding to each image in the image set to be retrieved is included in a pre-built image feature vector library. The image tag library is pre-built in the following manner: for each image in the image set to be retrieved, a descriptive tag is generated to obtain the image tag corresponding to each image, and the image tag library is obtained. The image feature vector library is pre-built in the following manner: for each image in the image set to be retrieved, image feature encoding is performed to obtain the image feature vector corresponding to each image, and the image feature vector library is obtained.

[0010] According to a second aspect of the present disclosure, an image acquisition apparatus is provided, comprising: an acquisition unit, configured to extract keywords from description information input by a user to obtain m keywords, and to perform feature encoding on the m keywords to obtain a text feature vector, wherein the description information is used to describe a target object in a target image to be acquired, and the m keywords respectively correspond to different image contents possessed by the target object, wherein m is a positive integer; a processing unit, configured to determine i first images in a set of images to be retrieved whose image tags match the m keywords, and to determine j second images in the set of images to be retrieved whose image feature vectors match the text feature vectors, wherein i and j are positive integers, wherein the image tags are used to characterize the image content of the corresponding image, and the image feature vectors are used to characterize the image features of the corresponding image; and a determination unit, configured to determine p target images from the intersection of the i first images and the j second images, and to acquire the p target images, wherein p is a positive integer.

[0011] According to some embodiments of this disclosure, the processing unit determines i first images in the image set to be retrieved whose image tags match the m keywords in the following manner: for each image in the image set to be retrieved, n keywords from the m keywords that match the corresponding image tag are determined, where n is a positive integer; for each image in the image set to be retrieved, in response to n being greater than a quantity threshold, the corresponding image is determined as the first image; wherein, the quantity threshold is determined based on m.

[0012] According to some embodiments of this disclosure, the image set to be retrieved includes k images, where k is a positive integer. The processing unit determines j second images in the image set to be retrieved whose image feature vectors match the text feature vectors as follows: For each image in the image set to be retrieved, the similarity between the image feature vector and the text feature vector of each image is determined to obtain k similarities corresponding to the k images respectively; the k similarities are sorted in descending order to obtain sorted k similarities; the j images corresponding to the j similarities with the highest order among the sorted k similarities are determined as the j second images.

[0013] According to some embodiments of this disclosure, after acquiring the p target images, the determining unit is further configured to: for each of the p target images, determine n image tags that match n keywords in the image tags corresponding to the target image; for each of the p target images, determine the image region where the target image is located in the target image according to the correspondence between the corresponding n image tags and the image content, and identify the target object in the target image.

[0014] According to some embodiments of this disclosure, the image tag corresponding to each image in the image set to be retrieved is included in a pre-built image tag library, and the image feature vector corresponding to each image in the image set to be retrieved is included in a pre-built image feature vector library. The image tag library is pre-built in the following manner: for each image in the image set to be retrieved, a descriptive tag is generated to obtain the image tag corresponding to each image, and the image tag library is obtained. The image feature vector library is pre-built in the following manner: for each image in the image set to be retrieved, image feature encoding is performed to obtain the image feature vector corresponding to each image, and the image feature vector library is obtained.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: execute the image acquisition method described in the first aspect or any embodiment of the first aspect.

[0016] According to a fourth aspect of the present disclosure, a storage medium is provided, the storage medium storing instructions that, when executed by a processor, enable the processor to perform the image acquisition method described in the first aspect or any embodiment of the first aspect.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the image acquisition method as described in the first aspect or any embodiment of the first aspect.

[0018] The technical solution provided by the embodiments of this disclosure can include the following beneficial effects: m keywords corresponding to the image content of the target object are obtained from the description information input by the user. These keywords are then matched with image tags corresponding to different images in the image set to be retrieved. i first images in the image set that match the description information in terms of image content are identified, where the image tag represents the image content contained in the corresponding image. The m keywords are feature-encoded to obtain a text feature vector. This text feature vector is then matched with image feature vectors corresponding to different images in the image set to be retrieved, identifying j second images in the image set that match the description information in terms of image features. The intersection of the i first images and the j second images is then filtered to remove interference items from both the i first images and the j second images, resulting in p target images that match the description information in both image features and image content. Through this disclosure, by searching for target images containing the target object using both image tag matching and image feature vector matching, it is possible to locate target objects with detailed attributes in the image set to be retrieved, and to more accurately locate target images corresponding to the description information input by the user.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0021] Figure 1 This is a flowchart illustrating an image acquisition method according to an exemplary embodiment.

[0022] Figure 2 This is a flowchart illustrating a method for determining i first images in a set of images to be retrieved, according to an exemplary embodiment.

[0023] Figure 3 This is a flowchart illustrating a method for determining j second images in a set of images to be retrieved, according to an exemplary embodiment.

[0024] Figure 4 This is a flowchart illustrating an image acquisition method according to an exemplary embodiment.

[0025] Figure 5 This is a block diagram illustrating an image acquisition method according to an exemplary embodiment of the present disclosure.

[0026] Figure 6 This is a block diagram illustrating a method for constructing an image tag library and an image feature vector library according to an exemplary embodiment of this disclosure.

[0027] Figure 7 This is a block diagram illustrating an image acquisition device according to an exemplary embodiment.

[0028] Figure 8 This is a block diagram illustrating an apparatus for image acquisition according to an exemplary embodiment.

[0029] Figure 9 This is a block diagram illustrating an apparatus for image acquisition according to an exemplary embodiment. Detailed Implementation

[0030] Some embodiments of this disclosure will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. Various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein, but can be changed as will become apparent upon understanding this disclosure, except for operations that must be performed in a particular order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted.

[0031] The embodiments described in the following examples of this disclosure are not representative of all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0032] The image acquisition methods provided in some embodiments of this disclosure are applied to scenarios where a target image pointed to by a user input is retrieved from a set of images to be retrieved based on user input. For example, retrieving multiple images containing a specific object from the album corresponding to a camera application. Another example is retrieving multiple images containing a specific object from a specified image library. Yet another example is retrieving a series of video frames containing a specific object from a video file.

[0033] With the development of the Internet and smart electronic devices, the storage capacity of cloud storage and the storage capacity of local storage media of smart electronic devices have been greatly expanded, both capable of storing large amounts of image and text data. Given the extremely large scale of images stored locally or in the cloud, and the rich variety and semantic complexity of the image and text content, how to efficiently and accurately retrieve specific image content from the massive image data in local storage media or cloud image libraries is a research direction for those skilled in the art.

[0034] In related technologies, text-image matching algorithms, object detection and classification methods, and image text extraction are generally used to assist users in obtaining target images. However, these technologies can only perform rough searches and cannot locate target images containing detailed attributes, resulting in an excessively wide range of search results. Users still need to browse further within the search results to find the image they want. Furthermore, these technologies cannot select the target object from the retrieved images. For example, when a user obtains a target image using a text-image matching algorithm, if the user inputs the descriptive text "two children on the beach," the results will include images that only contain the beach but not the children, or images that contain one or more children but not the beach. In other words, the search results from these technologies contain a large number of distractors, making it impossible to directly locate the target image containing detailed attributes.

[0035] In view of this, some embodiments of this disclosure provide an image acquisition method. For a set of images to be retrieved, each image corresponds to an image feature vector and an image label. m keywords corresponding to the content of the target object image are obtained from the user-input description information, and the text feature vectors corresponding to the m keywords are also obtained. Target images including the target object are searched through two dimensions: image label matching and image feature vector matching, resulting in p target images that match the description information in both image features and image content. Through this disclosure, target objects with detailed attributes can be located in the set of images to be retrieved, and target images corresponding to the user-input description information can be located more accurately.

[0036] Figure 1 This is a flowchart illustrating an image acquisition method according to an exemplary embodiment. For example... Figure 1 As shown, the method includes the following steps.

[0037] In step S101, keywords are extracted from the description information input by the user to obtain m keywords, and feature encoding is performed on the m keywords to obtain a text feature vector.

[0038] The description information describes the target object in the target image to be acquired. The m keywords correspond to different image contents of the target object, where m is a positive integer.

[0039] In step S102, i first images whose image labels match m keywords are determined in the image set to be retrieved, and j second images whose image feature vectors match text feature vectors are determined in the image set to be retrieved.

[0040] Where i and j are positive integers, the image label is used to represent the image content of the corresponding image, and the image feature vector is used to represent the image features of the corresponding image.

[0041] In step S103, the intersection of i first images and j second images is determined as p target images, and p target images are obtained, where p is a positive integer.

[0042] In this embodiment, the user-input description information can be voice information, text information, or image and text information containing the target object. When the description information is voice information, it can be converted into text information and keyword extraction can be performed. When the user-input description information is image and text information, keyword extraction and image content recognition can be performed on the image and text information to obtain the description text, and then keyword extraction can be performed on the description text. The image acquisition method in this disclosure can be obtained through a pre-trained image acquisition model. The image acquisition model can be a multimodal large model including multiple functional modules, and the keyword extraction function can be implemented by a specific module in the image acquisition model or by an independent model. In one example, this disclosure obtains keywords in the description information through a slot filling model: the natural language search statement obtained by combining the user input information is denoted as Query. The slot filling model is used to extract keywords from Query to obtain m independent search keywords, and the h-th search keyword is denoted as Ph (i = 1, 2, ..., m), which constitutes the search keyword text S.

[0043] In this embodiment of the disclosure, after obtaining the m keywords included in the description information, the m keyword information is feature encoded to obtain a text feature vector.

[0044] In this embodiment, the image set to be retrieved is a preprocessed image set. Each image in the image set is assigned a corresponding image label and an image feature vector. The image label represents the image content of the corresponding image, and the image feature vector represents the image features of the corresponding image. This disclosure matches keywords with image labels to determine i first images in the image set whose image labels match m keywords. The image content of the i first images corresponds to the description information input by the user. This disclosure also matches text feature vectors with image feature vectors to determine j second images in the image set whose image feature vectors match text feature vectors. The j second images and the i first images have the same image content and also have different image content. This disclosure determines the intersection of the i first images and the j second images as p target images, and filters out interference items in the i first images and the j second images. Finally, the image labels of the p target images match the keywords in the user-input description information, and the feature vectors of the p target images match the text vector of the user-input description information. In one example, the set of images to be retrieved can be a set of images corresponding to the feature application, a set of images in a specific folder in the local storage space of an electronic device, or a set of images obtained by sampling video frames at a preset sampling interval.

[0045] The image acquisition method in this embodiment of the present disclosure uses two dimensions, image label matching and image feature vector matching, to search for target images containing target objects. This enables the location of target objects with detailed attributes in the image set to be retrieved, and more accurately locates the target image corresponding to the description information input by the user.

[0046] In this embodiment of the disclosure, each image in the image set to be retrieved is assigned multiple tags. For each image in the image set to be retrieved, based on a quantity threshold and the number of times each of the m keywords matches the corresponding image tag, it is determined whether the corresponding image is a second image that matches the description information. The method for determining the i first images is described below in this embodiment of the disclosure.

[0047] Figure 2 This is a flowchart illustrating a method for determining i first images in a set of images to be retrieved, according to an exemplary embodiment. Figure 2 As shown, the method includes the following steps.

[0048] In step S201, for each image in the image set to be retrieved, n keywords that match the corresponding image label are determined from m keywords, where n is a positive integer.

[0049] In step S202, for each image in the set of images to be retrieved, in response to n being greater than a quantity threshold, the corresponding image is determined as the first image, wherein the quantity threshold is determined based on m.

[0050] In this embodiment, the method determines whether a corresponding image is a second image that matches the description information based on a quantity threshold and the number of matching keywords among m keywords. It is understood that the number of keywords "m" obtained from the user-input description information is not a constant value; the number of keywords "m" obtained for different description information is not the same. Therefore, this disclosure extracts keywords from the user-input description information to obtain m keywords, and determines the corresponding quantity threshold based on the obtained keyword count m. In one example, this disclosure uses the following formula to determine the quantity threshold: Where L is a defined threshold value, and max[(A,B)] is a function used to obtain the maximum value among A and B. Then it is Round down to the nearest integer.

[0051] In this embodiment of the disclosure, for each image in the image set to be retrieved, n keywords from m keywords that match the corresponding image tag are determined. If n is greater than a threshold, the image content of the corresponding image is considered to match the description information input by the user, and thus the corresponding image is determined as the first image. It is understood that this disclosure uses synonym matching and intelligent semantic matching when performing keyword and tag matching. For example, a synonym mapping table is introduced in the matching algorithm, so that "sea", "ocean", and "sea" can be matched consistently, while "yellow sand" does not match "pink sand" but matches "sand".

[0052] In this embodiment, the m keywords corresponding to the description information are matched with the image tags of each image to be retrieved, resulting in i first images in the set of images to be retrieved whose image tags match the m keywords. This facilitates further filtering of the i first images in the subsequent retrieval process to obtain target images that better match the description information input by the user.

[0053] In this embodiment of the disclosure, each image in the image set to be retrieved is assigned a corresponding image feature vector. For each image in the image set to be retrieved, the image features of the corresponding image are determined to match the descriptive information input by the user based on the similarity between the image feature vector and the text feature vector. The method for determining the i-th first image is described below in this embodiment of the disclosure.

[0054] Figure 3 This is a flowchart illustrating a method for determining j second images in a set of images to be retrieved, according to an exemplary embodiment. Figure 3As shown, the method includes the following steps.

[0055] In step S301, for each image in the image set to be retrieved, the similarity between the image feature vector and the text feature vector of each image is determined, and k similarity values ​​are obtained for each of the k images.

[0056] In step S302, the k similarities are sorted in descending order to obtain the sorted k similarities.

[0057] In step S303, the j images corresponding to the j most similar in the sorted k similarity scores are determined as the j second images.

[0058] In this embodiment of the disclosure, for each image in the image set to be retrieved, the similarity between the image feature vector and the text feature vector of each image is determined, resulting in k similarity scores for each of the k images. The obtained similarity scores can be pre-defined similarity scores, and the method for obtaining the similarity scores can be as follows: Where T is the text feature vector, I is the image feature vector, and Sim(T,I) is the cosine similarity between the text feature vector and the image feature vector.

[0059] In this embodiment of the disclosure, the image set to be retrieved includes k images. After obtaining k similarities corresponding to each of the k images, the k similarities are sorted in descending order to obtain sorted k similarities. Among the sorted k similarities, the one with the highest similarity is the highest, and the one with the lowest similarity is the lowest. This disclosure determines the j images with the highest similarity among the sorted k similarities, considering the j images with the highest similarity to be images that match the user's input description information in the feature dimension, and determines the j images with the highest similarity as the j second images.

[0060] In one example, this disclosure uses efficient search algorithms such as hierarchical navigable small-world network algorithm to quickly retrieve the set of images to be retrieved, and retrieves the first j images with the highest similarity values.

[0061] In this embodiment, j second images whose image feature vectors match the text feature vectors are determined from the set of images to be retrieved; that is, j second images that match the user-input description information in terms of feature dimensions are obtained. This facilitates further filtering of the j first images in the subsequent retrieval process to obtain target images that better match the user-input description information.

[0062] In this embodiment of the disclosure, after acquiring the target image, the target object described by the user-input description information can be further identified in the target image. The image acquisition method of this disclosure will be further described below.

[0063] Figure 4 This is a flowchart illustrating an image acquisition method according to an exemplary embodiment. For example... Figure 4 As shown, the method includes the following steps.

[0064] In step S401, for each of the p target images, n image labels that match the n keywords are determined from the image labels corresponding to the target image.

[0065] In step S402, for each of the p target images, the image region where the target image is located is determined in the target image according to the correspondence between the corresponding n image labels and the image content, and the target object is identified in the target image.

[0066] In this embodiment, after acquiring p target images, for each of the p target images, n image tags matching n keywords are determined from the image tags corresponding to the target image. These n image tags are the image tags corresponding to the target object, and the location of the target object in the corresponding target image can be identified through these n image tags. This disclosure, for each of the p target images, determines the image region where the target image is located in the target image based on the correspondence between the corresponding n image tags and the image content, and identifies the target object in the target image. This disclosure allows for the identification of target objects in target images in various ways, such as using a bounding box (x1,y1,x2,y2), where (x1,y1,) and (x2,y2) are the two pixel coordinates of the bounding box containing the target object. For example, (x1,y1,) and (x2,y2) are the pixel coordinates of the top left and bottom right corners of the bounding box, respectively. Another example is a mask image, which involves pixel-level segmentation of the target object, setting the pixel values ​​of the target object to 1 and the pixel values ​​of the background to 0.

[0067] By identifying the target object in p target images, users can quickly observe the target object within the images. Furthermore, when the target image to be retrieved is a single, specific image, the markers in the p target images can help users quickly observe the content at the end of the image, enabling users to quickly find the target image.

[0068] In this embodiment of the disclosure, the image tag corresponding to each image in the image set to be retrieved is included in a pre-built image tag library, and the image feature vector corresponding to each image in the image set to be retrieved is included in a pre-built image feature vector library. The following embodiments of this disclosure illustrate the method of constructing the image tag library and the image feature vector library.

[0069] In some embodiments of this disclosure, the image tag library is pre-constructed as follows: for each image in the image set to be retrieved, a descriptive tag is generated to obtain the image tag corresponding to each image. The image feature vector library is pre-constructed as follows: for each image in the image set to be retrieved, image feature encoding is performed to obtain the image feature vector corresponding to each image.

[0070] In this embodiment of the disclosure, for a set of images to be retrieved, descriptive labels are generated for all images in the set using a preset model, resulting in multiple labels corresponding to each image. The labels corresponding to each image are then stored in an image label library, maintaining the correspondence between images and their corresponding labels. The preset model used to obtain image labels can be a large model for multimodal image description generation.

[0071] In this embodiment of the disclosure, for the set of images to be retrieved, image feature encoding is performed on all images in the set to be retrieved to obtain an image feature vector corresponding to each image. The image feature vector corresponding to each image is then stored in an image feature vector library, maintaining the correspondence between images and their corresponding image feature vectors. Here, the length of the image feature vectors stored in the feature vector library is the same as the length of the text feature vectors obtained during the search phase.

[0072] In this embodiment, the image feature vector library and the image tag library are established separately using a multimodal large model. This can be done using a single-task multimodal large model containing a single module, or simultaneously using a multi-task multimodal large model containing multiple modules. The multi-task multimodal large model mainly includes an image encoding module, a text encoding module, and a text decoding module.

[0073] In an exemplary embodiment of this disclosure, Figure 5 This is a block diagram illustrating an image acquisition method according to an exemplary embodiment of the present disclosure, such as... Figure 5As shown, this disclosure can obtain target images in the following manner: Keywords from user input are obtained through a slot extraction module, and matched with the keywords using an image tag library to obtain a set of candidate image sets A (i first images) matching the keywords. The text encoding vectors (text feature vectors) corresponding to the keywords are obtained through the text encoding module in the multimodal feature vector alignment model and matched with the image encoding vectors (image feature vectors) in the image vector library to obtain a set of candidate vector image sets B (j second images) corresponding to the text encoding vectors. The target vector library is obtained by feature encoding the image to be retrieved through the image encoding module in the multimodal vector alignment model. After obtaining the candidate image sets A and B, their intersection is determined as the target image set C (p target images). For each image in the target image set C, the target object (target feature) in the target image is selected using a target attribute localization multimodal model, resulting in a target image set with the selected target object.

[0074] In an exemplary embodiment of this disclosure, Figure 6 This is a block diagram illustrating a method for constructing an image tag library and an image feature vector library according to an exemplary embodiment of this disclosure. Figure 6 As shown, this disclosure constructs an image tag library and an image vector library in the following manner: For each image to be processed in the image set to be retrieved, a text description tag for the image is obtained through an image description generation mode, resulting in an image tag library. The image to be processed is then feature-encoded using the image encoding module in a multimodal vector alignment model to obtain an image encoding vector, thus obtaining an image vector library. The multimodal vector alignment model also includes a text encoding module, which is used to obtain a text encoding vector corresponding to the user input during the target image retrieval stage, and the length of the obtained text encoding vector is consistent with the length of the image encoding vector.

[0075] In this embodiment, keyword extraction is performed on the user-input description information to obtain m keywords, and feature encoding is performed on the m keywords to obtain text feature vectors. In the image set to be retrieved, i first images whose image labels match the m keywords are determined, where image labels represent the image content of the corresponding image, and the i first images are those whose image content matches the description information. In the image set to be retrieved, j second images whose image feature vectors match the text feature vectors are determined, where image feature vectors represent the image features of the corresponding image, and the j second images are those whose image features match the description information. The intersection of the i first images and the j second images is determined as p target images, and these p target images are obtained. Through this disclosure, by searching for target images including the target object through two dimensions—image label matching and image feature vector matching—target objects with detailed attributes can be located in the image set to be retrieved, and target images corresponding to the user-input description information can be located more accurately. Compared to related image search solutions, this disclosure introduces a novel search selection function, enabling more precise selection of search targets while searching for images, thus enhancing the user experience. Furthermore, compared to traditional visual model solutions, this disclosure can utilize multiple multimodal models, or even a single large multimodal model, to achieve description generation, vector encoding, and object detection and localization. Moreover, this disclosure leverages a large multimodal model to detect and localize targets with detailed attributes, overcoming the limitations of traditional object detection methods.

[0076] Based on the same concept, this disclosure also provides an image acquisition device 100.

[0077] It is understood that the image acquisition device 100 provided in this disclosure includes hardware structures and / or software modules corresponding to each function in order to achieve the above-mentioned functions. In conjunction with the units and algorithm steps of the various examples disclosed in this disclosure, this disclosure can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the technical solutions of this disclosure.

[0078] Figure 7 This is a block diagram illustrating an image acquisition apparatus 100 according to an exemplary embodiment. (Refer to...) Figure 7 The device includes an acquisition unit 101, a processing unit 102, and a determination unit 103.

[0079] The acquisition unit 101 is used to extract keywords from the description information input by the user to obtain m keywords, and to encode the m keywords to obtain a text feature vector. The description information is used to describe the target object in the target image to be acquired. The m keywords correspond to different image contents of the target object, and m is a positive integer.

[0080] The processing unit 102 is used to determine i first images in the image set to be retrieved whose image tags match m keywords, and to determine j second images in the image set to be retrieved whose image feature vectors match text feature vectors, where i and j are positive integers. The image tags are used to characterize the image content of the corresponding image, and the image feature vectors are used to characterize the image features of the corresponding image.

[0081] The determining unit 103 is used to determine the intersection of i first images and j second images into p target images, and to obtain p target images, where p is a positive integer.

[0082] According to some embodiments of this disclosure, the processing unit 102 determines i first images in the image set to be retrieved whose image tags match m keywords in the following manner: For each image in the image set to be retrieved, n keywords from the m keywords that match the corresponding image tag are determined, where n is a positive integer. For each image in the image set to be retrieved, in response to n being greater than a quantity threshold, the corresponding image is determined as the first image. The quantity threshold is determined based on m.

[0083] According to some embodiments of this disclosure, the image set to be retrieved includes k images, where k is a positive integer. The processing unit 102 determines j second images in the image set to be retrieved whose image feature vectors match the text feature vectors as follows: For each image in the image set to be retrieved, the similarity between the image feature vector and the text feature vector of each image is determined, resulting in k similarity values ​​corresponding to the k images respectively. The k similarity values ​​are sorted in descending order to obtain sorted k similarity values. The j images corresponding to the j similarity values ​​with the highest order among the sorted k similarity values ​​are determined as the j second images.

[0084] According to some embodiments of this disclosure, after acquiring p target images, the determining unit 103 is further configured to: for each target image in the p target images, determine n image tags that match n keywords from the image tags corresponding to the target image; for each target image in the p target images, determine the image region where the target image is located in the target image according to the correspondence between the corresponding n image tags and the image content, and identify the target object in the target image.

[0085] According to some embodiments of this disclosure, the image tag corresponding to each image in the image set to be retrieved is included in a pre-built image tag library, and the image feature vector corresponding to each image in the image set to be retrieved is included in a pre-built image feature vector library. The image tag library is pre-built as follows: for each image in the image set to be retrieved, a descriptive tag is generated to obtain the image tag corresponding to each image, and thus the image tag library is obtained. The image feature vector library is pre-built as follows: for each image in the image set to be retrieved, image feature encoding is performed to obtain the image feature vector corresponding to each image, and thus the image feature vector library is obtained.

[0086] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0087] Figure 8 This is a block diagram illustrating an image acquisition device 200 according to an exemplary embodiment. The device 200 can be provided as an electronic device. For example, the device 200 can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0088] Reference Figure 8 The device 200 may include one or more of the following components: processing component 202, memory 204, power component 206, multimedia component 208, audio component 210, input / output (I / O) interface 212, sensor component 214, and communication component 216.

[0089] Processing component 202 typically controls the overall operation of device 200, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 202 may include one or more modules to facilitate interaction between processing component 202 and other components. For example, processing component 202 may include a multimedia module to facilitate interaction between multimedia component 208 and processing component 202.

[0090] Memory 204 is configured to store various types of data to support the operation of device 200. Examples of such data include instructions for any application or method operating on device 200, contact data, phonebook data, messages, pictures, videos, etc. Memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0091] The power supply component 206 provides power to the various components of the device 200. The power supply component 206 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device 200.

[0092] Multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 208 includes a front-facing camera and / or a rear-facing camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0093] Audio component 210 is configured to output and / or input audio signals. For example, audio component 210 includes a microphone (MIC) configured to receive external audio signals when device 200 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 204 or transmitted via communication component 216. In some embodiments, audio component 210 also includes a speaker for outputting audio signals.

[0094] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0095] Sensor assembly 214 includes one or more sensors for providing status assessments of various aspects of device 200. For example, sensor assembly 214 may detect the on / off state of device 200, the relative positioning of components such as the display and keypad of device 200, changes in the position of device 200 or a component of device 200, the presence or absence of user contact with device 200, the orientation or acceleration / deceleration of device 200, and temperature changes of device 200. Sensor assembly 214 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 214 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 214 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0096] Communication component 216 is configured to facilitate wired or wireless communication between device 200 and other devices. Device 200 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 216 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 216 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0097] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0098] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, which can be executed by a processor 220 of the device 200 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0099] Figure 9 This is a block diagram illustrating an image acquisition apparatus 300 according to an exemplary embodiment. For example, apparatus 300 may be provided as a server. (Refer to...) Figure 9The device 300 includes a processing component 322, which further includes one or more processors, and memory resources represented by memory 332 for storing instructions, such as application programs, that can be executed by the processing component 322. The application programs stored in memory 332 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 322 is configured to execute instructions to perform the aforementioned image acquisition method.

[0100] Device 300 may also include a power supply component 326 configured to perform power management of device 300, a wired or wireless network interface 350 configured to connect device 300 to a network, and an input / output (I / O) interface 358. Device 300 may operate on an operating system stored in memory 332, such as Windows Server™, MacOSX™, Unix™, Linux™, FreeBSD™, or similar.

[0101] In some embodiments of this disclosure, a storage medium including instructions is also provided, such as a memory 332 including instructions, which can be executed by the processing component 322 of the device 300 to perform the above-described method. For example, the storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0102] In some embodiments of this disclosure, a storage medium is provided, which may be a non-transitory computer-readable storage medium.

[0103] In some embodiments of this disclosure, when instructions in the storage medium are executed by the processor of an electronic device, the electronic device is able to perform the image acquisition method described above.

[0104] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the image acquisition method involved in any of the above embodiments.

[0105] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, a first component, part, region, layer, or section mentioned in the examples may also be referred to as a second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature.

[0106] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0107] In this description, "multiple" means at least two, referring to two or more, such as two, three, etc., unless otherwise explicitly specified. Other quantifiers are similar. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, unless otherwise specified or clearly indicated from the context, the articles "a" and "an" as used in this application and the appended claims are generally understood to mean "one or more."

[0108] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term "and / or" includes any one of the related listed items and any combination of two or more; "and / or" describes the association relationship between related objects, indicating that three relationships may exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Similarly, "at least one of..." includes any one of the related listed items and any combination of two or more.

[0109] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0110] Furthermore, the term "exemplary" is used herein to indicate that it serves as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term "exemplary" is intended to present concepts in a concrete manner. As used herein, the term "or" is intended to indicate an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X applies A or B" is intended to indicate any of the natural inclusive permutations. That is, if X applies A; X applies B; or X applies both A and B, then applying A or B satisfies the condition under any of the foregoing instances.

[0111] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding the specification and drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if it is not structurally equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in this disclosure, such terms are intended to be inclusive in a manner similar to the term “including.”

[0112] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0113] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. An image acquisition method, characterized in that, include: The user-input description information is used to extract keywords to obtain m keywords, and the m keywords are then used for feature encoding to obtain a text feature vector. The description information is used to describe the target object in the target image to be acquired. The m keywords correspond to different image contents of the target object, and m is a positive integer. In the set of images to be retrieved, determine i first images whose image tags match the m keywords, and determine j second images whose image feature vectors match the text feature vectors, where i and j are positive integers. The image tags are used to characterize the image content of the corresponding image, and the image feature vectors are used to characterize the image features of the corresponding image. The intersection of the i first images and the j second images is determined as p target images, and the p target images are obtained, where p is a positive integer.

2. The method according to claim 1, characterized in that, The step of determining i first images in the set of images to be retrieved whose image tags match the m keywords includes: For each image in the image set to be retrieved, determine n keywords from the m keywords that match the corresponding image tags, where n is a positive integer; For each image in the set of images to be retrieved, in response to n being greater than the quantity threshold, the corresponding image is determined as the first image; The quantity threshold is determined based on m.

3. The method according to claim 1, characterized in that, The set of images to be retrieved includes k images, where k is a positive integer. The step of determining j second images in the set of images to be retrieved whose image feature vectors match the text feature vectors includes: For each image in the set of images to be retrieved, the similarity between the image feature vector and the text feature vector of each image is determined, and k similarity values ​​are obtained for each of the k images. The k similarities are sorted in descending order to obtain the sorted k similarities; The j images corresponding to the j most similar to the k sorted similarities are determined as the j second images.

4. The method according to claim 2, characterized in that, After acquiring the p target images, the method further includes: For each of the p target images, determine n image tags that match the n keywords from the image tags corresponding to the target image; For each of the p target images, based on the correspondence between the corresponding n image labels and image content, the image region where the target image is located is determined in the target image, and the target object is identified in the target image.

5. The method according to any one of claims 1 to 4, characterized in that, The image tag corresponding to each image in the image set to be retrieved is included in a pre-built image tag library, and the image feature vector corresponding to each image in the image set to be retrieved is included in a pre-built image feature vector library; The image tag library is pre-built in the following manner: For each image in the set of images to be retrieved, a descriptive label is generated to obtain the image label corresponding to each image, and the image label library is obtained. The image feature vector library is pre-constructed in the following manner: For each image in the set of images to be retrieved, image feature encoding is performed to obtain the image feature vector corresponding to each image, and the image feature vector library is obtained.

6. An image acquisition device, characterized in that, include: The acquisition unit is used to extract keywords from the description information input by the user to obtain m keywords, and to encode the m keywords to obtain a text feature vector. The description information is used to describe the target object in the target image to be acquired. The m keywords correspond to different image contents of the target object, and m is a positive integer. The processing unit is configured to determine i first images in the image set to be retrieved whose image tags match the m keywords, and to determine j second images in the image set to be retrieved whose image feature vectors match the text feature vectors, where i and j are positive integers, and the image tags are used to characterize the image content of the corresponding image, and the image feature vectors are used to characterize the image features of the corresponding image; The determining unit is used to determine the intersection of the i first images and the j second images as p target images, and to acquire the p target images, where p is a positive integer.

7. The apparatus according to claim 6, characterized in that, The processing unit determines i first images in the set of images to be retrieved whose image tags match the m keywords in the following manner: For each image in the image set to be retrieved, determine n keywords from the m keywords that match the corresponding image tags, where n is a positive integer; For each image in the set of images to be retrieved, in response to n being greater than the quantity threshold, the corresponding image is determined as the first image; The quantity threshold is determined based on m.

8. The apparatus according to claim 6, characterized in that, The set of images to be retrieved includes k images, where k is a positive integer. The processing unit determines j second images in the set of images to be retrieved whose image feature vectors match the text feature vectors in the following manner: For each image in the set of images to be retrieved, the similarity between the image feature vector and the text feature vector of each image is determined, and k similarity values ​​are obtained for each of the k images. The k similarities are sorted in descending order to obtain the sorted k similarities; The j images corresponding to the j most similar to the k sorted similarities are determined as the j second images.

9. The apparatus according to claim 7, characterized in that, After acquiring the p target images, the determining unit is further configured to: For each of the p target images, determine n image tags that match the n keywords from the image tags corresponding to the target image; For each of the p target images, based on the correspondence between the corresponding n image labels and image content, the image region where the target image is located is determined in the target image, and the target object is identified in the target image.

10. The apparatus according to any one of claims 6 to 9, characterized in that, The image tag corresponding to each image in the image set to be retrieved is included in a pre-built image tag library, and the image feature vector corresponding to each image in the image set to be retrieved is included in a pre-built image feature vector library; The image tag library is pre-built in the following manner: For each image in the set of images to be retrieved, a descriptive label is generated to obtain the image label corresponding to each image, and the image label library is obtained. The image feature vector library is pre-constructed in the following manner: For each image in the set of images to be retrieved, image feature encoding is performed to obtain the image feature vector corresponding to each image, and the image feature vector library is obtained.

11. An electronic device, characterized in that, include: processor: Memory used to store computer programs or instructions that can be executed by a processor; The processor is configured to execute the computer program or instructions to implement the steps of the image acquisition method according to any one of claims 1 to 5.

12. A storage medium, characterized in that, The storage medium stores a computer program or instructions that, when executed by a processor of an electronic device, enable the electronic device to perform the image acquisition method according to any one of claims 1 to 5.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the image acquisition method as described in any one of claims 1 to 5.