An object query method and apparatus
By using target query patterns based on retrieval text and multimodal latent space features, the limitations of existing object retrieval methods in terms of application scope and accuracy are solved, enabling efficient and accurate image querying under different conditions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- LENOVO (BEIJING) LTD
- Filing Date
- 2022-11-08
- Publication Date
- 2026-05-26
AI Technical Summary
Existing object retrieval methods rely on existing image or attribute information, which limits their application scope and makes them prone to retrieval bias when the information is incomplete or inaccurate.
By obtaining the search text, the object attribute features are determined, and a target query pattern is designed based on the multimodal latent space features. The matching target image is then queried in the image library using this pattern, including a first query pattern (using only attribute features), a second query pattern (combining attribute features and multimodal latent space features), and a third query pattern (using only multimodal latent space features).
It improves the accuracy and application scope of object query, ensuring that target images can be found efficiently and accurately under different conditions.
Smart Images
Figure CN115618044B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of retrieval technology, and more specifically to an object query method and apparatus. Background Technology
[0002] With the development of Internet technology, various video and image data have exploded. In order to quickly retrieve relevant information about a certain object (e.g., a target person) from massive amounts of video data, various retrieval methods have emerged.
[0003] Current object retrieval methods can search for related images of the object being searched, but this method requires that the image to be searched already exists, which limits the scope of application of the retrieval. It can also be used to search for the attributes of the object being searched, but this method is prone to retrieval bias when the attribute information of the object being searched is incomplete or the description is inaccurate, which also limits the scope of use for users. Summary of the Invention
[0004] In view of the above, this application provides the following technical solution:
[0005] An object query method, comprising:
[0006] Obtain the search text corresponding to the object to be queried;
[0007] Based on the search text, determine the object attribute features corresponding to the object to be queried;
[0008] Based on the degree of matching between the object attribute features and the attribute tags in the target attribute feature set, a target query pattern determined based on the object attribute features and / or the multimodal latent space features corresponding to the search text is obtained;
[0009] Using the target query mode, a target image query result matching the object to be queried is obtained from the target image library.
[0010] Optionally, the target query pattern includes at least one of the following:
[0011] If each of the object attribute features matches each attribute label in the target attribute feature set, the target pattern is determined as the first query pattern, and the first query pattern represents the pattern of querying using the object attribute features.
[0012] If some of the object attribute features match the attribute labels in the target attribute feature set, the target query mode is determined as the second query mode. The second query mode represents a mode of querying using the object attribute features and the multimodal latent space features corresponding to the search text.
[0013] If the object attribute features do not match any of the attribute labels in the target attribute feature set, the target query mode is determined as the third query mode, which represents the mode of querying using the multimodal latent space features corresponding to the search text;
[0014] The multimodal latent space feature representation maps the text space and image space to an intermediate modal space in which the two can interact directly.
[0015] Optionally, the target query mode is a first query mode, wherein the step of using the target query mode to query the target image library to obtain target image query results that match the object to be queried includes:
[0016] Based on the object attribute features, a query is performed in the target image library, and images in the target image library whose image attribute features match the object attribute features are determined as the initial query images;
[0017] The image features of each initial query image are obtained, and the initial query images are sorted based on the image features and the object attribute features to obtain the target image query results.
[0018] Optionally, the step of sorting the initial query image based on the image features and the object attribute features to obtain the target image query result includes:
[0019] Based on the image acquisition time in the image features, determine the time parameter corresponding to each initial query image;
[0020] Obtain the image region in each initial query image that corresponds to the object attribute features, and determine the image quality parameters based on the image features of the image region;
[0021] Based on the matching degree between the image features in each initial query image and the object attribute features, the image attribute parameters are determined;
[0022] The initial query images are sorted according to the time parameter, the image quality parameter, and the image attribute parameter to obtain the target image query results.
[0023] Optionally, the target query mode is a second query mode, wherein the step of using the target query mode to query the target image library to obtain target image query results that match the object to be queried includes:
[0024] Based on the object attribute features, a query is performed in the target image library, and images in the target image library whose image attribute features match the object attribute features are determined as the initial query images;
[0025] Obtain the multimodal latent space features corresponding to each initial query image;
[0026] Based on the degree of matching between the multimodal latent space features corresponding to each initial query image and the multimodal latent space features corresponding to the search text, the target image query result is determined in the initial query image.
[0027] Optionally, the target query mode is a third query mode, wherein the step of using the target query mode to query the target image library to obtain target image query results that match the object to be queried includes:
[0028] Determine the multimodal latent space features of the retrieved text and the matching parameters of each image in the target image library in the multimodal latent space;
[0029] Based on the matching parameters, the target image query result is determined in the target image library.
[0030] Optionally, the method further includes:
[0031] Based on the search text, determine the query time characteristics corresponding to the search text;
[0032] The step of using the target query mode to query the target image library and obtain target image query results that match the object to be queried includes:
[0033] Based on the query time feature, candidate image query results that match the query time feature are determined in the target image library;
[0034] Based on the target query pattern, a target image query result that matches the object to be queried is determined from the candidate image query results.
[0035] Optionally, the query time feature is the retrieval time parameter feature in the retrieval text, wherein the step of using the target query pattern to query the target image library to obtain target image query results that match the query object includes:
[0036] Using the target query pattern, the system queries the target query database to obtain which target matches the object to be queried.
[0037] Obtain the image region in each target image that corresponds to the object attribute features, and determine the image quality parameters based on the image features of the image region;
[0038] Based on the matching degree between the image features in each target image and the object attribute features, the image attribute parameters are determined;
[0039] Based on the retrieval time parameter features, the image quality parameters, and the image attribute parameters, the image sorting mode is determined;
[0040] The target image is sorted based on the image sorting mode to obtain the target image query results.
[0041] Optionally, the query time feature is determined based on the time of obtaining the search text, wherein the step of using the target query pattern to query the target image library to obtain target image query results that match the object to be queried includes:
[0042] Using the target query pattern, an initial image matching the object attribute features is obtained by querying the target image library;
[0043] Based on the target time period corresponding to the query time characteristics;
[0044] Based on the image quality parameters and image attribute parameters of the initial images, the initial images for each target time period are sorted.
[0045] Based on the sorting results, the target image corresponding to each target time period is determined;
[0046] The target images corresponding to each target time period are combined to obtain the target image query results.
[0047] An object query device, comprising:
[0048] The first acquisition unit is used to obtain the search text corresponding to the object to be queried;
[0049] The determining unit is used to determine the object attribute features corresponding to the object to be queried based on the search text;
[0050] The second acquisition unit is used to obtain a target query pattern determined based on the object attribute features and / or the multimodal latent space features corresponding to the search text, based on the degree of matching between the object attribute features and the attribute tags in the target attribute feature set.
[0051] The query unit is used to use the target query mode to query the target image library to obtain target image query results that match the object to be queried.
[0052] As can be seen from the above technical solutions, this application discloses an object query method and apparatus, comprising: obtaining retrieval text corresponding to the object to be queried; determining object attribute features corresponding to the object to be queried based on the retrieval text; obtaining a target query pattern determined based on the object attribute features and / or multimodal latent space features corresponding to the retrieval text based on the degree of matching between the object attribute features and attribute tags in the target attribute feature set; and using the target query pattern to query in a target image library to obtain target image query results that match the object to be queried. This application improves the accuracy of object query by performing object query based on object attribute features and corresponding multimodal latent space features. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0054] Figure 1 A flowchart illustrating an object query method provided in an embodiment of this application;
[0055] Figure 2 A schematic diagram illustrating a query-based processing method provided in an embodiment of this application;
[0056] Figure 3 This is a schematic diagram of the structure of an object query device provided in an embodiment of this application;
[0057] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0058] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0059] This application provides an object query method that can be applied to scenarios where a target image matching the object to be queried is found. For example, in scenarios where a person with specific characteristics is found in a large number of videos, this method has the advantages of wide application range and high query accuracy.
[0060] See Figure 1This is a flowchart illustrating an object query method provided in an embodiment of this application. The method may include the following steps:
[0061] S101. Obtain the search text corresponding to the object to be queried.
[0062] S102. Based on the retrieved text, determine the object attribute features corresponding to the object to be queried.
[0063] S103. Based on the degree of matching between the object attribute features and the attribute labels in the target attribute feature set, obtain the target query pattern determined based on the object attribute features and / or the multimodal latent space features corresponding to the searched text.
[0064] S104. Using the target query mode, query the target image library to obtain the target image query results that match the object to be queried.
[0065] In step S101, the object to be queried is the object to be retrieved, such as a person with specific characteristics. The retrieval text corresponding to the object to be queried can be the text entered by the user when querying the object. For example, it can be the retrieval text directly entered by the user for the object to be queried, such as "Search for males with a height of 1.7 to 1.8 meters carrying a backpack". Alternatively, it can be the retrieval text composed of query information selected by the user based on the current query user interface. For example, the query user interface includes age, gender, facial features, and clothing fields, and the retrieval text is composed of information selected or entered by the user in each of these fields. This retrieval text serves as the basis for subsequent querying of the object to be queried in the target image database.
[0066] In step S102, object attribute features corresponding to the query object can be determined based on the search text. Object attribute features characterize the descriptive features of the query object. For example, if the query object is a person, the object attribute features could be the person's age, gender, facial features, clothing features, gait features, etc. Specifically, object attribute features can be directly determined from the search text. For example, if the search text is "query a male with a height of 1.7 to 1.8 meters carrying a backpack," then the object attribute features would be height, gender, and attire. Alternatively, semantic analysis can be performed on the currently input search text to obtain object attribute features associated with the query object. For example, if the search text is "query a tall person with a beard," then the object attribute features obtained through semantic analysis would be male and a height greater than 1.7 meters.
[0067] In step S103, after obtaining the object attribute features of the object to be queried, they can be compared with the attribute labels in the target attribute feature set to determine the degree of matching between the object attribute features and the attribute labels in the target attribute feature set, thereby determining the corresponding target query pattern. This can meet the needs of querying based on object attribute features and can also expand the application scope of querying based on object attribute features while improving query accuracy. For example, if there are few object attribute features, the query pattern can be based on the multimodal latent space features corresponding to the application retrieval text to achieve the goal of obtaining query results.
[0068] The attribute labels in the target attribute feature set include various labels that can accurately retrieve the corresponding target image, such as age labels, height labels, facial feature labels, body feature labels, clothing feature labels, accessory feature labels, gait feature labels, and gender labels. Therefore, the target query pattern can be determined based on the degree of matching between the object attribute features and the attribute labels in the target attribute feature set. Furthermore, the determined target query pattern is based on the object attribute features and / or the multimodal latent space features corresponding to the search text.
[0069] Specifically, the target query pattern should include at least one of the following:
[0070] If each object attribute feature matches each attribute label in the target attribute feature set, the target query pattern is determined as the first query pattern, which represents the pattern of querying using object attribute features.
[0071] If some object attribute features match the attribute labels in the target attribute feature set, the target query mode is determined as the second query mode. The second query mode represents the query mode that uses object attribute features and multimodal latent space features corresponding to the search text.
[0072] If the object attribute features do not match any of the attribute labels in the target attribute feature set, the target query mode is determined as the third query mode. The third query mode represents the mode of querying using the multimodal latent space features corresponding to the retrieved text.
[0073] Multimodal latent space feature representation maps text space and image space to an intermediate modal space where the two can directly interact. Specifically, based on multimodal technology, a processing network is designed to map text space and image space to an intermediate modal latent space where they can directly interact, allowing direct querying of the corresponding object image from free text describing object attributes. The establishment of multimodal latent space features includes: establishing an image visual feature space and establishing a text feature space describing the image; obtaining the latent representations of the final image visual features and text features for each object based on the multimodal feature space; and fusing the latent representations of the final image visual features and text representations for each object to obtain the fused latent representation of each object, thus obtaining the multimodal latent space features.
[0074] The main processing steps for subsequent object querying using multimodal latent space features include: inputting the fused latent representation of each object into a neural network to interact with image and text features; mapping the interaction result, object identifier, and user query text for that object into the latent space using transformation matrices to obtain the mapped latent identifier vector, object identifier representation vector, and query text representation vector for each object; based on the mapped object identifier representation vector and query text representation vector, obtaining the latent representation vector of the object corresponding to the query text; calculating the distance between the latent identifier vector of the object corresponding to the query text and the latent representation vector of each object in the target image database; sorting the objects according to this distance; and using the top-ranked objects as the object query results.
[0075] See Figure 2 This diagram illustrates a query-based processing method provided in this application. After obtaining the search text, the search text is parsed to obtain object attribute features. Then, the degree of matching between the object attribute features and the attribute tags in the target attribute feature set is determined, i.e., whether the object attribute tag belongs to the target attribute feature set. If the target attribute feature set contains all of the object attribute tag, it is determined to be a first query mode based on attribute query; if the target attribute feature set partially contains the object attribute features, it is determined to be a second query mode based on attribute filtering and multimodal latent space features, where the process of multimodal latent space feature query can be simply referred to as multimodal query; if the target attribute feature set does not contain the object attribute feature at all, it is determined to be a third query mode based on multimodal latent space features.
[0076] The process of querying based on the target query mode in step S104 is described in detail below. In one embodiment, the target query mode is a first query mode, wherein the step of using the target query mode to query the target image library to obtain target image query results that match the object to be queried includes: querying the target image library based on object attribute features, determining images in the target image library whose image attribute features and object attribute features match the matching condition as initial query images; obtaining the image features of each initial query image, and sorting the initial query images based on the image features and object attribute features to obtain target image query results.
[0077] Since each image in the target image library is labeled with a corresponding attribute tag, the object attribute features are matched with the attribute tags labeled with each image. For example, when the queried object is a person, the relevant features such as age, height, and gender contained in the object attribute features can be matched with each image. The matching condition refers to the condition that the matching degree is greater than a certain threshold. For example, if the matching degree of each object attribute feature is higher than 90%, the image is determined as the initial query image.
[0078] Furthermore, to enable users to obtain query results more accurately and quickly, the initial query images can be sorted based on their image features and object attribute features to obtain the target image query results. In other words, by combining image and attribute dimensions, the image with the highest query accuracy can be placed at the top of the displayed results.
[0079] Specifically, the step of sorting the initial query images based on image features and object attribute features to obtain target image query results includes: determining the time parameter corresponding to each initial query image based on the image acquisition time in the image features; obtaining the image region in each initial query image that corresponds to the object attribute features, and determining the image quality parameter based on the image features of the image region; determining the image attribute parameter based on the matching degree between the image features and the object attribute features in each initial query image; and sorting the initial query images according to the time parameter, image quality parameter, and image attribute parameter to obtain target image query results.
[0080] The image acquisition time can be the time it takes to acquire each image in the target image library, or the time it takes to add the image to the target image library. The image quality parameter is determined based on the image state of the image region corresponding to the object attribute features. For example, if the object attribute feature is a facial feature, and the image region containing the facial feature is a frontal face, then the image quality score corresponding to that image will be higher than the image quality score if the image region containing the facial feature is a side profile. The image attribute parameter is a parameter based on the matching degree between the image feature and the object attribute feature. For example, if the object attribute feature includes a height of 1.7 meters to 1.8 meters, then if the query image clearly matches this feature, the image attribute score corresponding to the image attribute parameter can be 95; if the initial query image does not clearly match the height feature, the image attribute score can be 65. Furthermore, a ranking function can be constructed based on the time parameter, image quality parameter, and image attribute parameter. After obtaining the initial query image, this ranking function can be used to sort the initial query image to obtain the target image query result. Alternatively, the sorting function can be determined at the same time as the target query pattern, and then the query can be performed based on the object attribute features and the sorting function to obtain the target image query results.
[0081] In another implementation, when the target query mode is the second query mode, the step of using the target query mode to query the target image library to obtain target image query results that match the object to be queried includes: querying the target image library based on object attribute features, determining images in the target image library whose image attribute features and object attribute features match the matching condition as initial query images; obtaining the multimodal latent space features corresponding to each initial query image; and determining the target image query results in the initial query images based on the degree of matching between the multimodal latent space features corresponding to each initial query image and the multimodal latent space features corresponding to the search text.
[0082] In the second query mode, the object attribute features are part of the existing target attribute set. These partial object attribute features can be used to filter the images in the target image library, initially selecting some initial query images that meet the object attribute features. Then, multimodal latent space features are extracted from the search text, and the matching score between these multimodal latent space features and the initially selected initial query images in the multimodal latent space is calculated. The query list is returned according to the matching score, and the images in this query list are the target image query results. Specifically, the multimodal query mode has been described in the aforementioned embodiments and will not be detailed here.
[0083] In another embodiment, when the target query mode is the third query mode, the step of using the target query mode to query the target image library to obtain the target image query result that matches the object to be queried includes: determining the multimodal latent space features of the retrieved text and the matching parameters of each image in the target image library in the multimodal latent space; and determining the target image query result in the target image library based on the matching parameters.
[0084] In this implementation, if none of the object attribute features are within the target attribute set, then multimodal latent space features are directly extracted from the search text, and the matching score between this multimodal latent space and the images in the target image library is calculated. A query list is then returned based on this matching score, and the images in this query list are the target image query results. Specifically, the multimodal query mode has been described in the preceding embodiments and will not be detailed here. Furthermore, the target image library can also be initially filtered based on application features, but the main processing method of this mode is still implemented through multimodal querying.
[0085] In this embodiment, existing object attribute features in the search text are used for querying. The corresponding target query pattern is determined based on the quantity and coverage of the object attribute features, thus expanding the application scope of object queries. Furthermore, predefined attribute tags from the target image library and relevant image features of the initial queried image, such as quality parameters, attribute parameters, and time parameters, are utilized to optimize the accuracy of the object query results. This achieves broad applicability, further enhancing the convenience of object queries for users and improving the accuracy of object attribute query results.
[0086] Correspondingly, this application embodiment also provides a method for generating a target image library. First, multiple acquired images are obtained. The attribute category library of object attribute labels for each acquired image and the attribute value corresponding to each attribute are determined. The attribute value corresponds to the attribute; for example, if the attribute is height, the attribute value is the height value. When establishing the corresponding target image library, an object attribute database also needs to be established. This involves performing object detection on each image, performing attribute recognition and quality analysis on each detected object image, extracting multimodal latent space features from each detected object image, and storing the multimodal latent space features, attribute features, quality parameters, and timestamp information of each object image. The object images containing this information are then stored in the target image library.
[0087] Furthermore, to more accurately retrieve images matching the target object, this embodiment further includes: determining the query time characteristics corresponding to the search text based on the search text. Specifically, the step of using a target query mode to obtain target image query results matching the target object from the target image library includes: determining candidate image query results matching the query time characteristics in the target image library based on the query time characteristics; and determining the target image query result matching the target object from the candidate query results based on the target query mode.
[0088] Specifically, the query time feature can be explicit time feature information included in the search text. For example, if the user inputs the search text "search for a woman with long hair, about 1.6 meters tall, carrying a white bag, who was this morning," then the time feature is "this morning." This query time feature can be used to initially filter images in the target image library. Then, the candidate image search results obtained from the initial screening can be further filtered based on the object attribute features in the search text. Specifically, filtering can be based on the query pattern determined by the object attribute features, a process that has been described in detail in the aforementioned embodiments and will not be elaborated here. The corresponding query time feature can also be time feature information indirectly obtained based on the search text, such as the system time when the search text was obtained. Specifically, text parsing can be performed on the input search text to extract the text sequence in the time dimension and the text sequence in the attribute dimension; parsing the text sequence in the time dimension yields the query time feature, and parsing the text sequence in the attribute dimension yields the object attribute feature.
[0089] In one implementation, the query time feature is the retrieval time parameter feature in the retrieved text. The step of using a target query mode to retrieve target image query results matching the target object from a target image library includes: using the target query mode to retrieve target objects matching the target object from the target image library; obtaining image regions in each target image corresponding to object attribute features, and determining image quality parameters based on the image features of the image regions; determining image attribute parameters based on the matching degree between image features in each target image and object attribute features; determining an image sorting mode based on the retrieval time parameter feature, image quality parameters, and image attribute parameters; and sorting the target images based on the image sorting mode to obtain the target image query results.
[0090] In this embodiment, the time parameter feature is obtained directly from the search text, such as "searching for this morning's...". Correspondingly, the process for determining the image quality parameter and image attribute parameter has been described in the previous embodiments and will not be detailed here. For example, based on the retrieved image, two vector values representing the image attribute parameter and image quality parameter are obtained: an attribute score vector and a quality score vector. For each attribute type, the image can obtain two values:
[0091] A attribute =score attribute (1)
[0092] Q attribute =quality attribute (2)
[0093] Equation (1) above represents the attribute score vector, and equation (2) represents the quality score.
[0094] If information about time period T can be parsed from the retrieved text, a list of images corresponding to each time period can be obtained:
[0095] List(T i ) = sort({S j}) (3)
[0096] in, q in the formula k Let a be the mass score vector in equation (2) above. k Let be the attribute score vector in equation (3) above.
[0097] In another implementation, the query time feature is determined based on the time when the search text is obtained, meaning that a specific time parameter cannot be directly obtained from the search text. Specifically, the step of using a target query pattern to retrieve target image query results matching the target object from a target image library includes: retrieving initial images matching the object's attribute features from the target image library using the target query pattern; determining target time periods based on the query time features; sorting the initial images for each target time period based on the image quality and image attribute parameters of the initial images; determining the target images corresponding to each target time period based on the sorting results; and combining the target images corresponding to each target time period to obtain the target image query results.
[0098] Specifically, if the time T information cannot be directly parsed from the search text, the corresponding query list can be represented as follows:
[0099] List = {Top k (List(T i))} (4)
[0100] In order to reflect the diversity of image acquisition time distribution in the target image query results, it is necessary to select the Top value for each target time period corresponding to the query time feature. k The results are then merged to form the query results, which are returned as the target image query result. Furthermore, image quality parameters and image attribute parameters can be used for sorting. This ensures both the accuracy of the query image results and the diversity of distribution across different time periods, meeting practical application needs.
[0101] In another embodiment of this application, an object query device is also provided, see [link to relevant documentation]. Figure 3 The device may include:
[0102] The first acquisition unit 301 is used to acquire the search text corresponding to the object to be queried;
[0103] The determining unit 302 is used to determine the object attribute features corresponding to the object to be queried based on the search text;
[0104] The second acquisition unit 303 is used to obtain a target query pattern determined based on the object attribute features and / or the multimodal latent space features corresponding to the search text, based on the degree of matching between the object attribute features and the attribute tags in the target attribute feature set.
[0105] The query unit 304 is used to use the target query mode to query the target image library to obtain the target image query result that matches the object to be queried.
[0106] This application discloses an object query device, comprising: a first acquisition unit obtaining search text corresponding to a query object; a determination unit determining object attribute features corresponding to the query object based on the search text; a second acquisition unit obtaining a target query pattern determined based on the object attribute features and / or multimodal latent space features corresponding to the search text, based on the degree of matching between the object attribute features and attribute tags in a target attribute feature set; and a query unit using the target query pattern to query a target image library to obtain target image query results matching the query object. This application improves the accuracy of object query by performing object query based on object attribute features and corresponding multimodal latent space features.
[0107] Optionally, the target query pattern includes at least one of the following:
[0108] If each of the object attribute features matches each attribute label in the target attribute feature set, the target query pattern is determined as the first query pattern, and the first query pattern represents the pattern of querying using the object attribute features.
[0109] If some of the object attribute features match the attribute labels in the target attribute feature set, the target query mode is determined as the second query mode. The second query mode represents a mode of querying using the object attribute features and the multimodal latent space features corresponding to the search text.
[0110] If the object attribute features do not match any of the attribute labels in the target attribute feature set, the target query mode is determined as the third query mode, which represents the mode of querying using the multimodal latent space features corresponding to the search text;
[0111] The multimodal latent space feature representation maps the text space and image space to an intermediate modal space in which the two can interact directly.
[0112] In one implementation, the target query mode is a first query mode, wherein the query unit includes:
[0113] The first determining subunit is used to query the target image library based on the object attribute features, and determine the images in the target image library whose image attribute features match the object attribute features as initial query images.
[0114] The sorting subunit is used to obtain the image features of each initial query image, and sort the initial query images based on the image features and the object attribute features to obtain the target image query results.
[0115] Furthermore, the sorting subunit is specifically used for:
[0116] Based on the image acquisition time in the image features, determine the time parameter corresponding to each initial query image;
[0117] Obtain the image region in each initial query image that corresponds to the object attribute features, and determine the image quality parameters based on the image features of the image region;
[0118] Based on the matching degree between the image features in each initial query image and the object attribute features, the image attribute parameters are determined;
[0119] The initial query images are sorted according to the time parameter, the image quality parameter, and the image attribute parameter to obtain the target image query results.
[0120] In another implementation, the target query mode is a second query mode, wherein the query unit is specifically used for:
[0121] Based on the object attribute features, a query is performed in the target image library, and images in the target image library whose image attribute features match the object attribute features are determined as the initial query images;
[0122] Obtain the multimodal latent space features corresponding to each initial query image;
[0123] Based on the degree of matching between the multimodal latent space features corresponding to each initial query image and the multimodal latent space features corresponding to the search text, the target image query result is determined in the initial query image.
[0124] In another implementation, the target query mode is a third query mode, wherein the query unit is specifically used for:
[0125] Determine the multimodal latent space features of the retrieved text and the matching parameters of each image in the target image library in the multimodal latent space;
[0126] Based on the matching parameters, the target image query result is determined in the target image library.
[0127] Optionally, the device further includes:
[0128] The time determination unit is used to determine the query time characteristics corresponding to the search text based on the search text;
[0129] The query unit includes:
[0130] The second determining subunit is used to determine, based on the query time feature, candidate image query results that match the query time feature in the target image library;
[0131] The third determining subunit is used to determine, based on the target query pattern, a target image query result that matches the object to be queried from the candidate image query results.
[0132] Correspondingly, the query time feature is the retrieval time parameter feature in the retrieved text, wherein the query unit includes:
[0133] The first acquisition subunit is used to use the target query mode to query and obtain a target image that matches the object to be queried in the target image library;
[0134] The fourth determining subunit is used to obtain the image region in each target image corresponding to the object attribute features, and to determine the image quality parameters based on the image features of the image region;
[0135] The fifth determining subunit is used to determine image attribute parameters based on the matching degree between image features in each target image and the object attribute features;
[0136] The sixth determining subunit is used to determine the image sorting mode based on the retrieval time parameter features, the image quality parameters, and the image attribute parameters;
[0137] The sorting subunit is used to sort the target image based on the image sorting mode to obtain the target image query result.
[0138] Furthermore, the query time feature is determined based on the time when the retrieved text is obtained, wherein the query unit is specifically used for:
[0139] Using the target query pattern, an initial image matching the object attribute features is obtained by querying the target image library;
[0140] Based on the target time period corresponding to the query time characteristics;
[0141] Based on the image quality parameters and image attribute parameters of the initial images, the initial images for each target time period are sorted.
[0142] Based on the sorting results, the target image corresponding to each target time period is determined;
[0143] The target images corresponding to each target time period are combined to obtain the target image query results.
[0144] It should be noted that the specific implementation of each unit and subunit in this embodiment can be referred to the corresponding content above, and will not be described in detail here.
[0145] In another embodiment of this application, a readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the various steps of the object query method as described in any of the preceding claims.
[0146] In another embodiment of this application, an electronic device is also provided, see [link to relevant documentation]. Figure 4 The electronic device may include:
[0147] Memory 401 is used to store the application and the data generated by the application during its operation;
[0148] Processor 402 is configured to execute the application program to implement the object query method as described in any of the above.
[0149] It should be noted that the specific implementation of the processor in this embodiment can be referred to the corresponding content above, and will not be described in detail here.
[0150] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0151] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0152] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0153] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. An object query method, comprising: Obtain the search text corresponding to the object to be queried; Based on the search text, determine the object attribute features corresponding to the object to be queried; Based on the degree of matching between the object attribute features and the attribute tags in the target attribute feature set, a target query pattern determined based on the object attribute features and / or the multimodal latent space features corresponding to the search text is obtained; Using the target query mode, a target image query result matching the object to be queried is obtained from the target image library; The target query pattern includes at least one of the following: If each of the object attribute features matches each attribute label in the target attribute feature set, the target query pattern is determined as the first query pattern, and the first query pattern represents the pattern of querying using the object attribute features. If some of the object attribute features match the attribute labels in the target attribute feature set, the target query mode is determined as the second query mode. The second query mode represents a mode of querying using the object attribute features and the multimodal latent space features corresponding to the search text. If the object attribute features do not match any of the attribute labels in the target attribute feature set, the target query mode is determined as the third query mode, which represents the mode of querying using the multimodal latent space features corresponding to the search text; The multimodal latent space feature representation maps the text space and image space to an intermediate modal space in which the two can interact directly.
2. The method according to claim 1, wherein the target query mode is a first query mode, wherein, The step of using the target query pattern to query the target image library and obtain target image query results that match the object to be queried includes: Based on the object attribute features, a query is performed in the target image library, and images in the target image library whose image attribute features match the object attribute features are determined as the initial query images; The image features of each initial query image are obtained, and the initial query images are sorted based on the image features and the object attribute features to obtain the target image query results.
3. The method according to claim 2, wherein sorting the initial query image based on the image features and the object attribute features to obtain the target image query result includes: Based on the image acquisition time in the image features, determine the time parameter corresponding to each initial query image; Obtain the image region in each initial query image that corresponds to the object attribute features, and determine the image quality parameters based on the image features of the image region; Based on the matching degree between the image features in each initial query image and the object attribute features, the image attribute parameters are determined; The initial query images are sorted according to the time parameter, the image quality parameter, and the image attribute parameter to obtain the target image query results.
4. The method according to claim 1, wherein the target query mode is a second query mode, wherein, The step of using the target query pattern to query the target image library and obtain target image query results that match the object to be queried includes: Based on the object attribute features, a query is performed in the target image library, and images in the target image library whose image attribute features match the object attribute features are determined as the initial query images; Obtain the multimodal latent space features corresponding to each initial query image; Based on the degree of matching between the multimodal latent space features corresponding to each initial query image and the multimodal latent space features corresponding to the search text, the target image query result is determined in the initial query image.
5. The method according to claim 1, wherein the target query mode is a third query mode, wherein, The step of using the target query pattern to query the target image library and obtain target image query results that match the object to be queried includes: Determine the multimodal latent space features of the retrieved text and the matching parameters of each image in the target image library in the multimodal latent space; Based on the matching parameters, the target image query result is determined in the target image library.
6. The method according to claim 1, further comprising: Based on the search text, determine the query time characteristics corresponding to the search text; The step of using the target query mode to query the target image library and obtain target image query results that match the object to be queried includes: Based on the query time feature, candidate image query results that match the query time feature are determined in the target image library; Based on the target query pattern, a target image query result that matches the object to be queried is determined from the candidate image query results.
7. The method according to claim 6, wherein the query time feature is the retrieval time parameter feature in the retrieved text, wherein, The step of using the target query pattern to query the target image library and obtain target image query results that match the object to be queried includes: Using the target query mode, a target image matching the object to be queried is obtained from the target image library; Obtain the image region in each target image that corresponds to the object attribute features, and determine the image quality parameters based on the image features of the image region; Based on the matching degree between the image features in each target image and the object attribute features, the image attribute parameters are determined; Based on the retrieval time parameter features, the image quality parameters, and the image attribute parameters, the image sorting mode is determined; The target image is sorted based on the image sorting mode to obtain the target image query results.
8. The method according to claim 7, wherein the query time feature is determined based on the time of obtaining the retrieved text, wherein, The step of using the target query pattern to query the target image library and obtain target image query results that match the object to be queried includes: Using the target query pattern, an initial image matching the object attribute features is obtained by querying the target image library; Based on the target time period corresponding to the query time characteristics; Based on the image quality parameters and image attribute parameters of the initial images, the initial images for each target time period are sorted. Based on the sorting results, the target image corresponding to each target time period is determined; The target images corresponding to each target time period are combined to obtain the target image query results.
9. An object query device, comprising: The first acquisition unit is used to obtain the search text corresponding to the object to be queried; The determining unit is used to determine the object attribute features corresponding to the object to be queried based on the search text; The second acquisition unit is used to obtain a target query pattern determined based on the object attribute features and / or the multimodal latent space features corresponding to the search text, based on the degree of matching between the object attribute features and the attribute tags in the target attribute feature set. The query unit is used to use the target query mode to query the target image library to obtain target image query results that match the object to be queried; The target query pattern includes at least one of the following: If each of the object attribute features matches each attribute label in the target attribute feature set, the target query pattern is determined as the first query pattern, and the first query pattern represents the pattern of querying using the object attribute features. If some of the object attribute features match the attribute labels in the target attribute feature set, the target query mode is determined as the second query mode. The second query mode represents a mode of querying using the object attribute features and the multimodal latent space features corresponding to the search text. If the object attribute features do not match any of the attribute labels in the target attribute feature set, the target query mode is determined as the third query mode, which represents the mode of querying using the multimodal latent space features corresponding to the search text; The multimodal latent space feature representation maps the text space and image space to an intermediate modal space in which the two can interact directly.