Multi-target image query method and device and storage medium

Abstract

The invention discloses a multi-target image query method and device and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: taking a plurality of associated first targets as a whole (i.e., a composite target), segmenting a single first target in an alternative image, and segmenting the composite target, so as to obtain a plurality of candidate images; and obtaining a single image and a first composite image. And then feature extraction is carried out on the single image and the first composite image, so that the obtained features comprise information of the first targets and information of the composite targets, and the extracted features can describe the alternative images more accurately. The alternative images are converted into more accurate feature descriptions, so that the detection precision is higher and the false detection rate is lower during subsequent image query. Therefore, according to the embodiment of the invention, the technical problem of false detection in a manner of returning the query result only by matching the features of the single or multiple first targets when the multiple targets are in contact with one another is solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a multi-target image query method, device and storage medium. Background Art

[0002] With the rapid development of visual Internet technology, camera devices play an important role in security monitoring, traffic management and other fields. These devices generate a huge number of warning images every day, and users' demand for retrieving these images is increasing. It is particularly important to retrieve these images quickly and accurately, which requires not only traditional query methods based on warning message attributes, but also direct search based on image content.

[0003] The existing technical solutions mainly include: tag-based image retrieval, which uses the tags pre-marked for images and allows users to search by selecting corresponding tags as keywords. Image retrieval based on feature extraction, which extracts features from images and then completes image retrieval based on feature vector comparison.

[0004] The above image retrieval scheme has the problem of low retrieval precision and recall rate when the image contains multiple targets. Specifically, when there are multiple targets in the image library, there are multiple target feature values, and the feature of the input image is a single target feature value, the feature matching result is low similarity and difficult to detect. If the detection threshold is increased, the precision rate is maintained, but the recall rate is reduced. If the detection threshold is lowered, the recall rate is maintained, but the precision rate is reduced. In addition, when multiple targets are related to each other, there will be a false detection problem. When multiple targets are divided into single targets, there will be a situation where a whole target is mistaken for two targets. For example, if a person rides a bicycle, it will be mistakenly detected that all images with people and bicycles will be retrieved. If a woman holding a child, images with both children and women will be retrieved.

[0005] In summary, facing the multi-target image retrieval scenario in the visual Internet, the current image retrieval scheme still has shortcomings in terms of retrieval accuracy and false detection rate. Summary of the invention

[0006] The embodiments of the present application provide a multi-target image query method, device and storage medium to solve the technical problem of low retrieval accuracy in the image retrieval scheme of the prior art in the multi-target image retrieval scenario.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a multi-target image query method, the method comprising:

[0009] Obtain query information and multiple candidate images;

[0010] Performing target segmentation processing on each of the candidate images to obtain each single image corresponding to each first target segmented from the candidate images and first composite images corresponding to the associated multiple first targets;

[0011] Extracting feature information from the single image and the first composite image respectively to obtain a first feature vector corresponding to the single image and a second feature vector corresponding to the first composite image;

[0012] Constructing a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and storing the first feature vector and the second feature vector in a vector database;

[0013] The vector database is queried according to the query information, and candidate images matching the query information are output.

[0014] In combination with the first aspect, in a possible design manner, performing target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and a first composite image corresponding to the associated multiple first targets includes:

[0015] Performing target recognition processing on each candidate image to obtain a first target region of each first target in the candidate image;

[0016] In the case where the candidate image includes a plurality of first targets, identifying association information between the plurality of first targets, and if the association information indicates that the plurality of first targets are associated, acquiring a first composite region of the associated plurality of first targets in the candidate image;

[0017] A single image corresponding to the first target area and a first composite image corresponding to the first composite area are segmented from the candidate image.

[0018] In combination with the first aspect, in a possible design manner, identifying association information between the multiple first targets, and if the association information indicates that the multiple first targets are associated, obtaining a first composite area of ​​the associated multiple first targets in the candidate image includes:

[0019] Identifying overlapping information between first target areas corresponding to a plurality of first targets;

[0020] The association information between the multiple first targets is represented by overlapping information between the multiple first target regions. If the overlapping information indicates that the first target regions corresponding to the multiple first targets overlap, a first composite region of the associated multiple first targets in the candidate image is obtained.

[0021] In combination with the first aspect, in a possible design manner, acquiring a first composite area of ​​the associated multiple first targets in the candidate image includes:

[0022] Obtaining a detection bounding box corresponding to each first target among the associated multiple first targets;

[0023] An area enclosed by a minimum rectangle that can contain each of the detection frames is used as a first composite area of ​​the multiple first objects in the candidate image.

[0024] In combination with the first aspect, in a possible design manner, the characterizing the association information between the multiple first targets by using the overlapping information between the multiple first target regions includes:

[0025] If it is identified that the multiple first target areas overlap with each other or the multiple first target areas overlap with each other, determining that the multiple first targets are associated with each other;

[0026] If it is identified that the multiple first target areas do not overlap with each other, it is determined that the multiple first targets are not associated with each other.

[0027] In combination with the first aspect, in a possible design manner, when the query information is a query image, querying the vector database according to the query information and outputting candidate images matching the query information includes:

[0028] performing target recognition processing on the query image, and marking each second target recognized in the query image;

[0029] After detecting a user's selection operation on the plurality of second objects, identifying association information between the plurality of second objects;

[0030] If the association information indicates that the plurality of second objects are associated with each other, obtaining a second composite region of the plurality of associated second objects in the query image;

[0031] Segmenting a second composite image corresponding to the second composite area from the query image;

[0032] extracting feature information from the second composite image to obtain a third feature vector corresponding to the second composite image;

[0033] The vector database is queried according to the third feature vector, and candidate images matching the third feature vector are output.

[0034] In combination with the first aspect, in a possible design manner, when the query information is a query text, querying the vector database according to the query information and outputting candidate images matching the query information includes:

[0035] Vectorizing the query text through a multimodal large model, and aligning characteristics with the first feature vector in the image-text feature vector space of the multimodal large model to obtain a fourth feature vector;

[0036] The vector database is queried according to the fourth feature vector, and a candidate image matching the fourth feature vector is output.

[0037] In combination with the first aspect, in a possible design manner, constructing a correspondence between the candidate image and the first feature vector and the second feature vector, and storing the first feature vector and the second feature vector in a vector database includes:

[0038] Generate a target label for the candidate image based on the first feature vector and the second feature vector;

[0039] storing the first feature vector and the second feature vector corresponding to the candidate images of the same image category in the same partition of a vector database, wherein the image category includes a target label, a shooting device, and a shooting time of the candidate images;

[0040] The querying the vector database according to the query information and outputting candidate images matching the query information includes:

[0041] The image category to be queried by the query information is obtained, the query information is input into a partition of the vector database matching the image category to be queried for query, and candidate images matching the query information in the partition are output.

[0042] In a second aspect, an embodiment of the present application provides a multi-target image query device, comprising:

[0043] A data acquisition module, used to acquire query information and multiple candidate images;

[0044] An image segmentation module, used for performing target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and a first composite image corresponding to the associated multiple first targets;

[0045] A feature recognition module, used to extract feature information from the single image and the first composite image respectively, to obtain a first feature vector corresponding to the single image and a second feature vector corresponding to the first composite image;

[0046] A feature storage module, used to construct a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and store the first feature vector and the second feature vector in a vector database;

[0047] The image query module is used to query the vector database according to the query information and output candidate images matching the query information.

[0048] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method of the first aspect and possible design methods thereof.

[0049] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method of the first aspect and its possible design method when running, and the display screen is configured to display the second video and audio processed by the processor.

[0050] Compared with the prior art, the embodiment of the present application provides a multi-target image query method, device and storage medium, first obtaining query information and multiple candidate images; performing target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and the first composite image corresponding to the associated multiple first targets; then extracting feature information from the single image and the first composite image respectively to obtain the first feature vector corresponding to the single image and the second feature vector corresponding to the first composite image; constructing the corresponding relationship between the candidate image and the first feature vector and the second feature vector, storing the first feature vector and the second feature vector in a vector database; then querying the vector database according to the query information, and outputting the candidate image matching the query information. The present application detects the composite target and also extracts the features of the first composite image corresponding to the composite target, thereby achieving a more accurate feature description of the candidate image. Then, in the subsequent query process for the multi-target query scene, a more accurate matching image can be returned based on the more accurately described features, thereby enhancing the recognition ability of multiple targets and composite targets, improving the retrieval accuracy and reducing the false detection rate.

[0051] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0053] Figure 1 A hardware structure block diagram of a computer device provided in an embodiment of the present application is shown;

[0054] Figure 2 A schematic diagram of a query scenario provided by an embodiment of the present application is shown;

[0055] Figure 3 A flowchart of a multi-target image query method provided by an embodiment of the present application is shown;

[0056] Figure 4 A flow chart of a target segmentation method provided by an embodiment of the present application is shown;

[0057] Figure 5A A schematic diagram of a single target segmentation method provided in an embodiment of the present application is shown;

[0058] Figure 5B A schematic diagram of a non-overlapping multi-target segmentation method provided in an embodiment of the present application is shown;

[0059] Figure 5C A schematic diagram of a mutually overlapping multi-target segmentation method provided in an embodiment of the present application is shown;

[0060] Fig. 6A A schematic diagram showing a non-overlapping multiple target positions provided by an embodiment of the present application is shown;

[0061] Figure 6B A schematic diagram showing overlapping of multiple target positions provided by an embodiment of the present application is shown;

[0062] Figure 6C A schematic diagram showing a pairwise overlap of multiple target positions provided by an embodiment of the present application is shown;

[0063] Figure 7 A schematic diagram of a multi-target image query method provided by an embodiment of the present application is shown;

[0064] Figure 8 A structural block diagram of a multi-target image query device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0065] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0066] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0067] The embodiment of the present application provides a multi-target image query method, which regards multiple associated first targets as a whole (i.e., a composite target), and in the candidate image, not only the single first target is segmented, but also the composite target is segmented to obtain a single image and a first composite image. Then, feature extraction is performed on the single image and the first composite image respectively, so that the obtained features contain both the information of each first target and the information of the composite target, so that the extracted features describe the candidate image more accurately. By converting the alternative image into a more accurate feature description, the detection accuracy is higher and the false detection rate is lower when querying the image subsequently. Therefore, the embodiment of the present application solves the technical problem of false detection in the method of returning the query result only by matching the features of a single or multiple first targets when multiple targets are related to each other.

[0068] The method can be executed in a terminal, a computer or a similar test system. For example, running on a computer device, Figure 1 FIG. 1 shows a hardware structure block diagram of a computer device provided in an embodiment of the present application. Figure 1 As shown, the computer device may include one or more ( Figure 1The computer device may further include a transmission device 106 and an input / output device 108 for communication functions. It will be appreciated by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0069] The memory 104 can be used to store computer programs, for example, software programs and modules of application software. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, to implement the above method. The memory 104 can be used to store data, such as query information, alternative images, feature vectors, etc. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0070] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0071] The embodiments of the present application can be applied to scenarios such as searching for images by text and searching for images by images. Taking searching for images by images as an example, Figure 2 A schematic diagram of a query scenario provided by an embodiment of the present application is shown. Figure 2 As shown, the user inputs a query image 208, and the query image 208 is subjected to target recognition, and the single targets are obtained as a man 201, a woman 202, a child 203, a football 204, and a tree 205, and the composite targets are a man and a woman holding hands 206, and a child playing football 207. The computer device displays the query image 208 and annotates the identified targets, such as Figure 2Each target is marked with a rectangular frame. When the user selects the child playing football 207, it means that the user needs to search for images that are highly correlated with the child playing football 207. The computer device crops the area where the child playing football 207 is located from the query image 208 to obtain a second composite image 209, matches the feature vector of the second composite image 209 with the feature vector in a pre-constructed vector database, and returns the target image 210 corresponding to the matched feature vector to the user. The target image 210 shows a group of children playing football, which matches the child playing football 207, so the query result is accurate. The embodiment of the present application can also be applied to the video intelligent search scenario of cloud playback in the visual network by searching images by images and searching images by texts, to realize frame analysis of video data, data vectorization, open custom feature retrieval, and retrieval of target videos by images.

[0072] Figure 3 A flowchart of a multi-target image query method provided by an embodiment of the present application is shown. Figure 3 As shown, the method includes steps S301 to S305.

[0073] Step S301: Obtain query information and multiple candidate images.

[0074] In this step, the computer device receives the user's query instruction and obtains query information. The query information can be a complete image, such as a query image uploaded by the user, or an image cropped from the complete image after the computer device detects that the user has selected several targets in the image.

[0075] The query information can be represented in a variety of ways, such as images, text, voice, audio and video, etc. If the query information is a query image, the computer device outputs candidate images that match the image, and if the query information is text, the computer device outputs candidate images that match the text.

[0076] The candidate images may be derived from a video stream, a network database, or a memory of a user's camera device, etc. Some of the candidate images are pre-labeled to train the target detection model. The labeling may be performed in the candidate images by software, and the labeling content includes labeling the category and the coordinates of a single target.

[0077] The category refers to the type of object, usually a label such as "person" or "ball" or "tree" or "car". The coordinates are the location of the target in the candidate image, usually represented by a bounding box, which contains the pixel coordinates of the upper left and lower right corners of the rectangular box where the target is located. These two pixel coordinates determine the exact location and size of the target in the candidate image.

[0078] The computer device stores the candidate images in the form of feature vectors to reduce the amount of storage. For the specific process of converting the candidate images into feature vectors, refer to the description of steps S302 to S304 below.

[0079] Step S302: perform target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and a first composite image corresponding to the associated multiple first targets.

[0080] In this step, when considering the association between multiple targets, if only features are extracted for multiple targets separately, the obtained feature vectors will not accurately describe the candidate images. For example, a person riding a bicycle should be considered as a whole, but the person and the vehicle are identified through the target detection model, and the image is queried based on the person and the vehicle. As long as there are images of people and vehicles, they will be retrieved; for example, if there is a woman holding a child, images of both children and women will be retrieved. Therefore, when retrieving multi-target images, only retrieving a single target will result in too low retrieval accuracy. Based on this, in addition to segmenting the single image, this step also segments the multiple associated targets as a whole from the candidate image as the first composite image. Continuing with the example of a person riding a bicycle, the image of the person, the image of the vehicle, and the image of the person riding a bicycle will be segmented in step S302.

[0081] As an example, the candidate image can be identified by the target detection model, and the target can be segmented from the candidate image based on the recognition result. The recognition result includes the category and the coordinates of a single target. The training process of the target detection model is not described in detail here.

[0082] Figure 4 A flow chart of a target segmentation method provided in an embodiment of the present application is shown. Figure 4 The step S302 further includes:

[0083] Step S401: Perform target recognition processing on each candidate image to obtain a first target region of each first target in the candidate image.

[0084] Step S402: when the candidate image includes multiple first targets, identifying association information between the multiple first targets, and if the association information indicates that the multiple first targets are associated, obtaining a first composite region of the associated multiple first targets in the candidate image.

[0085] Step S403: Segment the candidate images into a single image corresponding to the first target region and a first composite image corresponding to the first composite region.

[0086] This embodiment segments the image based on the first target area and the first composite area. It is understandable that if the alternative image contains only one first target, then only the first target area where the first target is located is segmented. The first target area is the bounding box of the first target, and the size of the first target area is equal to the size of the bounding box of the first target. If the alternative image contains multiple first targets, then the first target areas corresponding to each first target and the first composite area where the composite target is located are segmented separately. The first composite area contains the key features of the composite target, and the size of the first composite area is based on the ability to identify the composite target. Exemplarily, the first composite area is a rectangular box of the first target area that includes all associated first targets.

[0087] Figure 5A A schematic diagram of a single target segmentation method provided in an embodiment of the present application is shown. Figure 5A As shown, target recognition processing is performed on the first alternative image 501, and the first target "car" is identified from the first alternative image 501. The area of ​​the car in the first alternative image 501 is the car area 502. There is only one first target, so the image at the position of the car area 502 is segmented from the first alternative image 501 to obtain the car image 503.

[0088] Figure 5B A schematic diagram of a non-overlapping multi-target segmentation method provided in an embodiment of the present application is shown. Figure 5B As shown, target recognition processing is performed on the second alternative image 504, and the first targets "cat" and "grass" are identified from the second alternative image 504. The area of ​​the cat in the second alternative image 504 is the cat area 505, and the area of ​​the grass in the second alternative image 504 is the grass area 506. There are two unrelated first targets, so the image at the position of the cat area 505 is segmented out from the second alternative image 504 to obtain the cat image 507, and the image at the position of the grass area 506 is segmented out from the second alternative image 504 to obtain the grass image 508.

[0089] Figure 5C A schematic diagram of a mutually overlapping multi-target segmentation method provided by an embodiment of the present application is shown, such as Figure 5CAs shown, target recognition processing is performed on the third candidate image 509, and the first targets "pedestrian" and "sidewalk" are identified from the third candidate image 509. The area of ​​the pedestrian in the third candidate image 509 is the pedestrian area 510, and the area of ​​the sidewalk in the third candidate image 509 is the sidewalk area 511. There are two associated first targets, so the first composite area 512 is obtained. It can be seen that the first composite area 512 includes the pedestrian area 510 corresponding to the associated pedestrian and the sidewalk area 511 corresponding to the sidewalk. The image at the position of the pedestrian area 510 is segmented from the third candidate image 509 to obtain the pedestrian image 513, the image at the position of the sidewalk area 511 is segmented from the third candidate image 509 to obtain the sidewalk image 514, and the image at the position of the first composite area 512 is segmented from the third candidate image 509 to obtain the first composite image 515.

[0090] In some of the embodiments, step S402 includes: in the case where the alternative image includes multiple first targets, identifying overlapping information between first target areas corresponding to the multiple first targets; characterizing association information between the multiple first targets with the overlapping information between the multiple first target areas, and if the overlapping information indicates that the first target areas corresponding to the multiple first targets overlap, obtaining a first composite area of ​​the associated multiple first targets in the alternative image.

[0091] In this embodiment, the overlapping information includes the degree of overlap and the nature of overlap, wherein the degree of overlap can be determined by the overlapping area. The more overlapping areas there are, the greater the degree of overlap is, indicating a stronger association. If the overlapping area is 0, the degree of overlap is 0, indicating no association.

[0092] Specifically, when the overlapping information is the overlap degree, if the overlap degree between the multiple first target areas is greater than or equal to a threshold, it is determined that the multiple first targets are associated with each other; otherwise, if the overlap degree between the multiple first targets is less than the threshold, it is determined that the multiple first targets are associated with each other.

[0093] When the overlapping information is overlapping, if the multiple first target areas overlap, it is determined that the multiple first targets are associated with each other. Otherwise, if the multiple first targets do not overlap, it is determined that the multiple first targets are associated with each other.

[0094] The above-mentioned embodiment uses overlapping information between multiple first target areas to represent association information between multiple first targets, including: if it is identified that multiple first target areas overlap with each other or multiple first target areas overlap with each other, it is determined that the multiple first targets are associated with each other; if it is identified that the multiple first target areas do not overlap with each other, it is determined that the multiple first targets are not associated with each other.

[0095] Combine the following FIG. 6A to FIG. 6C This embodiment is described. Fig. 6A A schematic diagram of a non-overlapping multiple target positions provided by an embodiment of the present application is shown, such as Fig. 6A As shown, the first targets (A1 to An) of the candidate images are first segmented independently to obtain n single images. If there is overlap between the positions of multiple first targets, they are identified as composite targets. The overlap includes multiple target positions overlapping each other (such as Figure 6B As shown), multiple target positions overlap each other (as shown Figure 6C As shown), the segmentation principle is to take the union of the target positions. For example, when the positions of two targets m1 (x1_min, y1_min, x1_max, y1_max) and m2 (x2_min, y2_min, x2_max, y2_max) overlap, the union of the overall target m is (min(x1_min, x2_min), min(y1_min, y2_min), max(x1_max, x2_max), max(y1_max, y2_max)), that is, the area enclosed by the minimum matrix containing the detection border is taken as the area of ​​the overall target m.

[0096] When the segmentation principle is to take the union of the target positions, the above-mentioned obtaining the first composite area of ​​the multiple associated first targets in the alternative image includes: obtaining the detection frame corresponding to each first target in the multiple associated first targets; and taking the area enclosed by the minimum rectangle that can contain each detection frame as the first composite area of ​​the multiple first targets in the alternative image.

[0097] The minimum rectangle of this embodiment includes the pixel features of the associated multiple first targets while eliminating as few pixel features that are not the first targets as possible, so as to improve the accuracy of subsequent recognition of composite target features.

[0098] In other embodiments, the associated information can be determined by a pre-trained target detection model. Specifically, the associated first target is annotated when annotating the alternative image, and the target detection model is trained with the alternative image. The target detection model is trained with the gap between the target position and associated information detected by the target detection model and the annotated target position and associated information as the target, so that the target detection model has the ability to identify the target and output the associated information label. Then, in addition to annotating the bounding box of the single first target, the target detection model also annotates the bounding box of the composite target.

[0099] Step 303 : extracting feature information from the single image and the first composite image respectively, to obtain a first feature vector corresponding to the single image and a second feature vector corresponding to the first composite image.

[0100] In this step, the feature vector of the image can be extracted by a multimodal large model, which refers to a large-scale deep learning model that can process and understand multiple types of data (such as text, images, videos, voice, etc.). The multimodal large model can perform multi-task learning by combining information from different modalities, showing stronger reasoning and understanding capabilities than a single modality model. Exemplary, multimodal large models include CLIP (Contrastive Language-Image Pretraining), BLIP (Bootstrapping Language-Image Pretraining), etc.

[0101] Step 304: construct a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and store the first feature vector and the second feature vector in a vector database.

[0102] Among them, the vector database can be a milvus vector database. This step realizes the conversion of the matching result of the feature vector into the matching result of the query information and the alternative image based on the corresponding relationship in a way of establishing a corresponding relationship. For example, if a feature vector 1 whose matching degree of the feature vector corresponding to the query information is higher than the threshold is queried, the alternative image 1 corresponding to the feature vector 1 is output. The feature vector can be directly stored in the vector database through the corresponding relationship to reduce the storage space. As an example, a corresponding relationship between the feature vector and the image address of the alternative image can be established. When a feature vector 1 whose matching degree is higher than the threshold is queried, the alternative image is downloaded from the image address corresponding to the feature vector 1 based on the corresponding relationship, and the alternative image is displayed to the user.

[0103] Step 305: query the vector database according to the query information, and output candidate images matching the query information.

[0104] The query information is vectorized and converted into a feature vector to achieve matching between feature vectors, and the candidate image corresponding to the feature vector that meets the condition of matching degree with the feature vector of the query information is output. The condition of meeting the condition may be that the matching degree is higher than a preset similarity threshold.

[0105] The embodiments of the present application can realize image query for multimodal input, that is, the user can input query information such as text, image, video, voice, etc., and the computer device vectorizes the query information through the multimodal large model, maps it to the same vector space as the feature vector of the alternative image, and then compares the matching degree between the feature vector of the query information and the vector of the alternative image, and outputs the alternative image to the user based on the matching degree.

[0106] The query method of this embodiment is described below by taking the query information being a query text or a query image as an example.

[0107] When the query information is a query text, the query text is vectorized through the multimodal large model, and the characteristics are aligned with the first feature vector in the image-text feature vector space of the multimodal large model to obtain a fourth feature vector; based on the fourth feature vector and the vector database, a query is performed to output an alternative image matching the fourth feature vector.

[0108] In the case where the query information is a query image, target recognition processing is performed on the query image, and each recognized second target is marked in the query image; after detecting the user's selection operation on multiple second targets, the association information between the multiple second targets is identified; if the association information indicates that the multiple second targets are associated, a second composite area of ​​the associated multiple second targets in the query image is obtained; a second composite image corresponding to the second composite area is segmented from the query image; feature information is extracted from the second composite image to obtain a third feature vector corresponding to the second composite image; a vector database is queried according to the third feature vector, and an alternative image matching the third feature vector is output.

[0109] It should be noted that in order to improve the recall rate and precision rate of the query, when the user provides the query image, the computer device annotates each second target identified through target recognition processing, with the purpose of providing multiple second targets for the user to choose from. When the user selects several of the multiple second targets, it is equivalent to refining the query information, so the query range will be further narrowed and the matching results will be more accurate. In addition, when the multiple second targets selected by the user are related, the computer device will also crop the image of the area where the multiple related second targets are located (the second composite area), so as to obtain a second composite image that can describe the composite target. By recognizing the second composite image, the characteristics of the query image can be understood more accurately, so the accuracy of subsequent queries will be higher.

[0110] In some of the embodiments, by storing feature vectors in partitions, the query partition is first determined during query, and then each feature vector in the query partition is matched with the query information, thereby achieving efficient query of billions of images and reducing response delay.

[0111] Specifically, when constructing a vector database, a target label of an alternative image is generated based on the first feature vector and the second feature vector; the first feature vector and the second feature vector corresponding to the alternative images of the same image category are stored in the same partition of the vector database, where the image category includes the target label, shooting device, and shooting time of the alternative image.

[0112] When querying, the image category to be queried by the query information is obtained, the query information is input into the partition of the vector database matching the image category to be queried for query, and the candidate images matching the query information in the partition are output.

[0113] This embodiment extracts feature vectors from billions of images (equivalent to the single image and the first composite image mentioned above) through a multimodal large model, and stores the feature vectors in a vector database according to business characteristics, including multiple dimensions such as shooting equipment, target tags, and time range. First, scalar filtering is used to narrow the search scope, and then vector matching is used to match in the partitions. This hybrid search method of scalar filtering combined with vector matching improves the vector search efficiency and reduces the query result response delay.

[0114] In the embodiment described in the above steps S301 to S305, query information and multiple candidate images are first obtained; target segmentation processing is performed on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and the first composite image corresponding to the associated multiple first targets; feature information is then extracted from the single image and the first composite image respectively to obtain the first feature vector corresponding to the single image and the second feature vector corresponding to the first composite image; a corresponding relationship between the candidate image and the first feature vector and the second feature vector is constructed, and the first feature vector and the second feature vector are stored in a vector database; then the vector database is queried according to the query information to output the candidate image matching the query information. The present application achieves a more accurate feature description of the candidate image by detecting a composite target and also extracting features from the first composite image corresponding to the composite target. Then, in the subsequent query process for a multi-target query scene, a more accurate matching image can be returned based on the more accurately described features, thereby enhancing the recognition capability of multiple targets and composite targets, improving the retrieval accuracy, and reducing the false detection rate. In addition, this application also uses a multimodal large model to extract features from query information of different modalities, integrates open custom features and closed label retrieval, and in the case where the query information is a query image, vectorizes user-defined targets and performs retrieval based on user-defined targets to improve the openness, ease of use and flexibility of the retrieval. In addition, by extracting the feature vectors of the segmented image through a multimodal large model, the similarity of image matching can be greatly improved.

[0115] The method provided in the embodiment of the present application is illustrated below with a specific example. The method is applied to the server. The server and the client interact in a microservice http api protocol manner. A unified text search and image search interface is defined externally, and the https api interface specification is uniformly used. The relevant functional modules are implemented based on the Kubernetes+Springboot microservice framework. The functional modules include image reading, image segmentation, label extraction and image vectorization, vector data storage, text search and image search modules.

[0116] Image reading is mainly based on the image address to download and obtain image information, solve the problems such as repeated downloading or breakpoint resumption when image download fails, and improve image reading performance; image segmentation mainly combines the rules of target recognition and location information to complete the segmentation and storage of images, and divides large images into small images; label extraction and image vectorization are based on the image provided by the segmentation image service to realize label extraction and call the multimodal large model capability to extract feature vectors; vector data storage depends on the feature vectorization information and label information provided by the image vectorization module, and stores the target data in the milvus vector database; the text-to-image and image-to-image modules vectorize the text and image respectively, and then request the vector database to complete the similarity matching, combined with multi-dimensional retrieval, and return the target image results to the user.

[0117] Figure 7 A schematic diagram of a multi-target image query method provided by an embodiment of the present application is shown. Figure 7As shown, in this method, image reading is triggered on demand in combination with user needs, wherein the image is equivalent to the above-mentioned alternative image. The server extracts the user's video or image resources on demand in combination with the user's database building needs to prepare for subsequent vectorized storage. Then, the image is segmented into multiple targets and composite targets, the image is read, the image target is identified, and the target and composite target are segmented separately to obtain the segmented single image and the first composite image. Then, the server extracts labels from the image, specifically for the segmented target objects, including independent targets and composite targets, and extracts target labels through large model capabilities, such as extracting people, cars, pets, children, the elderly, cats, dogs, etc. The target label can provide a reference for the subsequent feature partitioning and warehousing. Then the image is vectorized and stored in the database. Specifically, the segmented small images are targeted at individual targets or composite targets. The multimodal large model capability is called for image vectorization for the original image before segmentation and the small images after segmentation. If an original image has n independent small targets and m composite targets, an original image will generate 1+n+m vectorized feature values. Its advantage is that it can achieve refined matching, and the retrieved image can maximize the image matching accuracy and avoid missed detection and false detection of the same object. In addition, the feature values ​​of each image after vectorization are combined with label information and partitioned according to business characteristics (such as user account, shooting device, shooting time) to narrow the subsequent search scope and improve retrieval efficiency under billions of data volumes. At the query time, search by image, search by custom features, etc. In the image search mode, the user uploads an image, and the server identifies the image target and identifies multiple single targets or composite targets. The user selects the target object to be searched, which can be a single target, a composite target or a combination of multiple targets. The server performs image segmentation based on the selected target and position, and then calls the multimodal large model capability for vectorization. Combined with the matching combination of multiple vector feature values, the target image is located and searched. This method can greatly improve the precision and recall rate of image retrieval. The more image targets and composite targets are searched, the more obvious the improvement in precision and recall rate. In the custom feature search mode, open input of custom feature text can be used. The server vectorizes the text information through the multimodal large model, and combines the spatial alignment characteristics of the image and text feature vectors of the multimodal large model to achieve the purpose of flexible image search through text feature vector search and matching image feature vectors. This method also provides multi-dimensional fusion retrieval. Specifically, based on the performance of large-scale image retrieval, partition retrieval is first performed to minimize the retrieval scope. Within the partition range, a hybrid retrieval combining vector matching and scalar filtering is performed based on image or text vectors and label fields, as well as time range constraints. By adopting this multi-dimensional fusion retrieval method, the image retrieval performance can be maximized and the latency can be reduced.

[0118] Figure 8A structural block diagram of a multi-target image query device provided by an embodiment of the present application is shown. Figure 8 As shown, the device comprises:

[0119] The data acquisition module 81 is used to acquire query information and multiple candidate images.

[0120] The image segmentation module 82 is used to perform target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and a first composite image corresponding to the associated multiple first targets.

[0121] The feature recognition module 83 is used to extract feature information from the single image and the first composite image respectively, and obtain a first feature vector corresponding to the single image and a second feature vector corresponding to the first composite image.

[0122] The feature storage module 84 is used to construct a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and store the first feature vector and the second feature vector in a vector database.

[0123] The image query module 85 is used to query the vector database according to the query information and output candidate images matching the query information.

[0124] In some of the embodiments, the image segmentation module 82 is further used to perform target recognition processing on each alternative image to obtain a first target area of ​​each first target in the alternative image; in the case where the alternative image includes multiple first targets, identify association information between the multiple first targets, and if the association information indicates that the multiple first targets are associated, obtain a first composite area of ​​the associated multiple first targets in the alternative image; and segment a single image corresponding to the first target area and a first composite image corresponding to the first composite area from the alternative image.

[0125] In some of the embodiments, the image segmentation module 82 is further used to identify overlapping information between first target areas corresponding to multiple first targets; the overlapping information between the multiple first target areas is used to represent the association information between the multiple first targets; if the overlapping information indicates that the first target areas corresponding to the multiple first targets overlap, then the first composite area of ​​the associated multiple first targets in the alternative image is obtained.

[0126] In some of the embodiments, the image segmentation module 82 is further used to obtain a detection frame corresponding to each first target in the associated multiple first targets; and to use the area enclosed by the smallest rectangle that can contain each detection frame as the first composite area of ​​the multiple first targets in the alternative image.

[0127] In some of the embodiments, the image segmentation module 82 is further used to determine that the multiple first targets are associated with each other if it is identified that the multiple first target areas overlap with each other or overlap with each other; and to determine that the multiple first targets are not associated with each other if it is identified that the multiple first target areas do not overlap with each other.

[0128] In some of the embodiments, the feature storage module 84 is also used to generate a target label for the alternative image based on the first feature vector and the second feature vector; the first feature vector and the second feature vector corresponding to the alternative images of the same image category are stored in the same partition of the vector database, wherein the image category includes the target label, shooting device, and shooting time of the alternative image.

[0129] The image query module 85 is further used to obtain the image category to be queried in the query information, input the query information into the partition of the vector database matching the image category to be queried for query, and output the candidate images matching the query information in the partition.

[0130] In some of the embodiments, the image query module 85 is also used to perform target recognition processing on the query image, and to mark each recognized second target in the query image; after detecting the user's selection operation on multiple second targets, identify the association information between the multiple second targets; if the association information indicates that the multiple second targets are associated, obtain a second composite area of ​​the associated multiple second targets in the query image; segment a second composite image corresponding to the second composite area from the query image; extract feature information from the second composite image to obtain a third feature vector corresponding to the second composite image; query the vector database according to the third feature vector, and output an alternative image matching the third feature vector.

[0131] In some of the embodiments, the image query module 85 is also used to vectorize the query text through the multimodal large model, and align the characteristics with the first feature vector in the image-text feature vector space of the multimodal large model to obtain a fourth feature vector; based on the fourth feature vector and the vector database, a query is performed to output an alternative image matching the fourth feature vector.

[0132] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0133] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0134] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0135] Optionally, in this embodiment, the processor may be configured to execute the following steps S1 to S5 through a computer program.

[0136] S1. Obtain query information and multiple candidate images.

[0137] S2. Performing target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and a first composite image corresponding to the associated multiple first targets.

[0138] S3. Extract feature information from the single image and the first composite image respectively to obtain a first feature vector corresponding to the single image and a second feature vector corresponding to the first composite image.

[0139] S4. Construct a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and store the first feature vector and the second feature vector in a vector database.

[0140] S5. Query the vector database according to the query information, and output candidate images matching the query information.

[0141] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0142] In addition, in combination with the method provided in the above embodiment, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any multi-target image query method in the above embodiment is implemented.

[0143] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes each function or step executed by the processor in the above method embodiment.

[0144] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0145] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0146] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0147] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A multi-target image query method, characterized in that: The method comprises: Obtain query information and multiple candidate images; Performing target segmentation processing on each of the candidate images to obtain each single image corresponding to each first target segmented from the candidate images and first composite images corresponding to the associated multiple first targets; Extracting feature information from the single image and the first composite image respectively to obtain a first feature vector corresponding to the single image and a second feature vector corresponding to the first composite image; Constructing a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and storing the first feature vector and the second feature vector in a vector database; The vector database is queried according to the query information, and candidate images matching the query information are output.

2. The multi-target image query method according to claim 1, characterized in that: The step of performing target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and a first composite image corresponding to the associated plurality of first targets includes: Performing target recognition processing on each candidate image to obtain a first target region of each first target in the candidate image; In the case where the candidate image includes a plurality of first targets, identifying association information between the plurality of first targets, and if the association information indicates that the plurality of first targets are associated, acquiring a first composite region of the associated plurality of first targets in the candidate image; A single image corresponding to the first target area and a first composite image corresponding to the first composite area are segmented from the candidate image.

3. The multi-target image query method according to claim 2, characterized in that: The identifying the association information between the plurality of first targets, and if the association information indicates that the plurality of first targets are associated, acquiring a first composite area of ​​the associated plurality of first targets in the candidate image, comprises: Identifying overlapping information between first target areas corresponding to a plurality of first targets; The association information between the multiple first targets is represented by overlapping information between the multiple first target regions. If the overlapping information indicates that the first target regions corresponding to the multiple first targets overlap, a first composite region of the associated multiple first targets in the candidate image is obtained.

4. The multi-target image query method according to claim 2 or claim 3, characterized in that: The step of acquiring a first composite area of ​​the associated plurality of first targets in the candidate image comprises: Obtaining a detection bounding box corresponding to each first target among the associated multiple first targets; An area enclosed by a minimum rectangle that can contain each of the detection frames is used as a first composite area of ​​the multiple first objects in the candidate image.

5. The multi-target image query method according to claim 3, characterized in that: The method of representing the association information between the multiple first targets by using the overlapping information between the multiple first target regions includes: If it is identified that the multiple first target areas overlap with each other or the multiple first target areas overlap with each other, determining that the multiple first targets are associated with each other; If it is identified that the multiple first target areas do not overlap with each other, it is determined that the multiple first targets are not associated with each other.

6. The multi-target image query method according to claim 1, characterized in that: In the case where the query information is a query image, querying the vector database according to the query information and outputting candidate images matching the query information includes: performing target recognition processing on the query image, and marking each second target recognized in the query image; After detecting a user's selection operation on the plurality of second objects, identifying association information between the plurality of second objects; If the association information indicates that the plurality of second objects are associated with each other, obtaining a second composite region of the plurality of associated second objects in the query image; Segmenting a second composite image corresponding to the second composite area from the query image; extracting feature information from the second composite image to obtain a third feature vector corresponding to the second composite image; The vector database is queried according to the third feature vector, and candidate images matching the third feature vector are output.

7. The multi-target image query method according to claim 1, characterized in that: In the case where the query information is a query text, querying the vector database according to the query information and outputting candidate images matching the query information includes: Vectorizing the query text through a multimodal large model, and aligning characteristics with the first feature vector in the image-text feature vector space of the multimodal large model to obtain a fourth feature vector; The vector database is queried according to the fourth feature vector, and a candidate image matching the fourth feature vector is output.

8. The multi-target image query method according to claim 6 or claim 7, characterized in that: The step of constructing a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and storing the first feature vector and the second feature vector in a vector database includes: Generate a target label for the candidate image based on the first feature vector and the second feature vector; storing the first feature vector and the second feature vector corresponding to the candidate images of the same image category in the same partition of a vector database, wherein the image category includes a target label, a shooting device, and a shooting time of the candidate images; The querying the vector database according to the query information and outputting candidate images matching the query information includes: The image category to be queried by the query information is obtained, the query information is input into a partition of the vector database matching the image category to be queried for query, and candidate images matching the query information in the partition are output.

9. A multi-target image query device, characterized in that: include: A data acquisition module, used to acquire query information and multiple candidate images; An image segmentation module, used for performing target segmentation processing on each candidate image to obtain each single image corresponding to each first target segmented from the candidate image and a first composite image corresponding to the associated multiple first targets; A feature recognition module, used to extract feature information from the single image and the first composite image respectively, to obtain a first feature vector corresponding to the single image and a second feature vector corresponding to the first composite image; A feature storage module, used to construct a corresponding relationship between the candidate image and the first feature vector and the second feature vector, and store the first feature vector and the second feature vector in a vector database; The image query module is used to query the vector database according to the query information and output candidate images matching the query information.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the multi-target image query method according to any one of claims 1 to 8 when running.

Images