Image target detection method and device, electronic equipment and storage medium

Abstract

The invention provides an image target detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a multimodal cue word with complex semantics, carrying out the semantic extraction of the multimodal cue word to obtain a plurality of entity relation phrases with low-order semantics, obtaining a picture set related to the multimodal cue word, and obtaining a plurality of entity relation phrases with low-order semantics; marking the picture set according to the entity relationship phrase to obtain a first image target frame, calculating the image-text similarity between the first image target frame and the entity relationship phrase, screening the first image target frame according to a predetermined lower limit of the image-text similarity to obtain a to-be-processed second image target frame, and processing the to-be-processed second image target frame according to the to-be-processed second image target frame. And fusing the second image target frames to obtain a plurality of image target clusters, generating fusion target frames corresponding to the plurality of image target clusters, ranking the fusion target frames according to the confidence of the image target clusters and the number of the image target frames in the image target clusters, and determining a target image target frame corresponding to the multi-modal cue word. According to the method, the accuracy of determining the complex semantic description target is improved.

Description

Technical Field

[0001] The present application belongs to the field of image technology, and specifically relates to an image target detection method, an image target detection device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In real life, there are a large number of application scenarios that require detection and positioning of targets with complex semantic descriptions. Traditional target detection models can handle coarse-grained targets such as "pedestrians", "vehicles", and "animals". When positioning targets with complex semantic descriptions, it is difficult to determine whether the selected target matches the complex semantic description because the subject involves multiple modifiers or attachments.

[0003] At present, when traditional target detection models detect targets with complex semantic descriptions, it is difficult to determine the scope of negative samples due to the fuzzy definition of the boundaries of negative samples. In addition, it is difficult to collect positive samples with complex semantic descriptions for recognition. The ratio of positive and negative samples is seriously unbalanced, resulting in the inability of traditional target detection models to recognize targets with complex semantic descriptions, further affecting the accuracy and recall rate of targets with complex semantic descriptions. Summary of the invention

[0004] In view of this, the present invention aims to propose an image target detection method, device, electronic device and storage medium to solve the problem that traditional target detection models cannot recognize targets with complex semantic descriptions, further affecting the accuracy and recall rate of targets with complex semantic descriptions.

[0005] According to a first aspect of the present invention, there is provided an image object detection method, the method comprising:

[0006] Acquire multimodal prompt words with complex semantics, perform semantic extraction on the multimodal prompt words, and obtain multiple entity relationship phrases with low-order semantics;

[0007] Acquire a picture set associated with the multimodal prompt word, and annotate the picture set according to the entity relationship phrase to obtain a first image target frame;

[0008] Calculating the image-text similarity between the first image target frame and the entity relationship phrase, and screening the first image target frame according to a predetermined image-text similarity lower limit to obtain a second image target frame to be processed;

[0009] Fusing the second image object frames to obtain multiple image object clusters, and generating multiple fused object frames corresponding to the image object clusters;

[0010] The fused target frame is ranked according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and the target image target frame corresponding to the multimodal prompt word is determined.

[0011] Optionally, the acquiring of multimodal prompt words with complex semantics and performing semantic extraction on the multimodal prompt words to obtain a plurality of entity relationship phrases with low-order semantics include:

[0012] Acquire multimodal prompt words with complex semantics, perform word segmentation and part-of-speech tagging on the multimodal prompt words, and identify entities in the multimodal prompt words;

[0013] Extracting relationships between entities in the multimodal prompt words to obtain entity relationships;

[0014] The entities and the entity relationships are combined to obtain a plurality of entity relationship phrases with low-order semantics.

[0015] Optionally, the acquiring a picture set associated with the multimodal prompt word, and annotating the picture set according to the entity relationship phrase to obtain a first image target frame includes:

[0016] Obtaining a set of pictures associated with the multimodal prompt words;

[0017] Using a pre-trained image segmentation model to segment the images in the image set, and identifying segmented images corresponding to low-order semantic phrases;

[0018] The segmented image is annotated with a boundary box to obtain a first image target box.

[0019] Optionally, the calculating the image-text similarity between the first image target frame and the entity relationship phrase, and screening the first image target frame according to a predetermined image-text similarity lower limit to obtain a second image target frame to be processed includes:

[0020] Using a pre-trained contrastive language image model to calculate the image-text similarity between the first image target frame and the entity relationship phrase, to obtain the image-text similarity value of the first image target frame;

[0021] Comparing the image-text similarity value of the first image target frame with a predetermined image-text similarity lower limit to obtain a comparison result;

[0022] The image target frames whose image-text similarity values ​​in the comparison results are less than the image-text similarity lower limit are screened to obtain a second image target frame to be processed.

[0023] Optionally, the predetermined lower limit of image-text similarity is obtained by the following steps:

[0024] Acquire a pre-collected picture set and the entity relationship phrase;

[0025] Inputting the picture set and the entity relationship phrase into a pre-trained contrastive language image model, and outputting a set of image-text similarity values ​​between the entity relationship phrase and the picture set;

[0026] The minimum value among the image-text similarity values ​​is determined as the image-text similarity lower limit.

[0027] Optionally, fusing the second image object frame to obtain a plurality of image object clusters, and generating a plurality of fused object frames corresponding to the image object clusters, comprises:

[0028] Mapping the second image target frame to a preset region map, performing target cluster fusion in the preset region map, and obtaining a plurality of image target clusters; wherein any two image target frames in the image target clusters overlap, and the overlap degree of any two image target frames is greater than a preset threshold;

[0029] The boundary coordinates of each of the image object clusters are determined, and a fused object frame corresponding to each of the image object clusters is generated.

[0030] Optionally, ranking the fused target frames according to the confidence of the image target cluster and the number of image target frames in the image target cluster to determine the target image target frame corresponding to the multimodal prompt word includes:

[0031] Determining the confidence of the image object cluster and the number of image object boxes in the image object cluster;

[0032] Ranking the fused target frames corresponding to the image target clusters according to preset priorities of confidence and the number of image target frames to obtain a ranking order;

[0033] A fusion target frame whose ranking order is higher than a preset ranking is determined as a target image target frame corresponding to the multimodal prompt word.

[0034] According to a second aspect of the present invention, there is provided an image object detection device, the device comprising:

[0035] A semantic extraction module is used to obtain multimodal prompt words with complex semantics, perform semantic extraction on the multimodal prompt words, and obtain multiple entity relationship phrases with low-order semantics;

[0036] A picture annotation module, used to obtain a picture set associated with the multimodal prompt word, annotate the picture set according to the entity relationship phrase, and obtain a first image target frame;

[0037] A target screening module, used for calculating the image-text similarity between the first image target frame and the entity relationship phrase, screening the first image target frame according to a predetermined image-text similarity lower limit, and obtaining a second image target frame to be processed;

[0038] An object fusion module, used to fuse the second image object frame to obtain multiple image object clusters, and generate fused object frames corresponding to the multiple image object clusters;

[0039] The target determination module is used to rank the fused target frame according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determine the target image target frame corresponding to the multimodal prompt word.

[0040] According to another aspect of the present invention, there is also provided an electronic device, comprising:

[0041] processor;

[0042] a memory for storing instructions executable by the processor;

[0043] The processor is configured to execute the instructions to implement the image target detection method as described above.

[0044] According to another aspect of the present invention, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image object detection method described above are implemented.

[0045] The image target detection method provided by the embodiment of the present invention obtains multimodal prompt words with complex semantics, performs semantic extraction on the multimodal prompt words, obtains multiple entity relationship phrases with low-order semantics, obtains a picture set associated with the multimodal prompt words, annotates the picture set according to the entity relationship phrases, obtains a first image target frame, calculates the image-text similarity between the first image target frame and the entity relationship phrase, screens the first image target frame according to a predetermined image-text similarity lower limit, obtains a second image target frame to be processed, fuses the second image target frame to obtain multiple image target clusters, generates fused target frames corresponding to the multiple image target clusters, ranks the fused target frames according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determines the target image target frame corresponding to the multimodal prompt word. The embodiment of the present invention decomposes complex semantic prompt words into multiple low-order semantic entity relationship phrases, performs image segmentation based on the low-order semantic phrases, and identifies target frames related to the low-order semantic phrases, so that the model can cut in from the low-order semantics to recall the target, which is beneficial to improving the model's recall ability for targets corresponding to complex semantic descriptions, and uses image-text similarity to screen out target frames with high correlation with low-order semantic phrases, thereby filtering out irrelevant targets and improving the accuracy of the model's recall results. By fusing multiple low-order semantic recall results, targets that meet the multimodal prompt words are determined, thereby improving the accuracy of detecting targets with complex semantic descriptions.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0048] Figure 1 is a flowchart of a method for detecting an image target provided by an embodiment of the present invention;

[0049] Figure 2 yes Figure 1 Flow chart of step 101 in the image target detection method provided by the embodiment of the present invention;

[0050] Figure 3 yes Figure 1 Flow chart of step 102 in the image target detection method provided by the embodiment of the present invention;

[0051] Figure 4 yes Figure 1 Flow chart of step 103 in the image target detection method provided by the embodiment of the present invention;

[0052] Figure 5 yes Figure 1 Flow chart of step 104 in the image target detection method provided by the embodiment of the present invention;

[0053] Figure 6 yes Figure 1 Flow chart of step 105 in the image target detection method provided by the embodiment of the present invention;

[0054] Figure 7 is a scene schematic diagram of the image target detection method provided by an embodiment of the present invention;

[0055] Figure 8 is a structural schematic diagram of an image target detection device provided by an embodiment of the present invention;

[0056] Fig. 9 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] To make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can be implemented. The division of the following embodiments is for the convenience of description, and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0058] Reference Figure 1 , shows a flowchart of the steps of the image target detection method provided by an embodiment of the present invention, and the method may include:

[0059] Step 101, obtaining multimodal prompt words with complex semantics, performing semantic extraction on the multimodal prompt words, and obtaining a plurality of entity relationship phrases with low-order semantics.

[0060] The embodiment of the present invention realizes accurate and efficient detection of image targets under complex semantic descriptions based on an image segmentation model and a contrastive language image model. The image segmentation model of this embodiment takes the SAM model as an example, and the contrastive language image model takes CLIP as an example. It should be noted that SAM is a method for solving image segmentation tasks, which recognizes and segments various objects in images based on a deep learning architecture, without the need to train independent models for specific scenes or objects. CLIP is a multimodal text and image pre-training model, which achieves cross-modal understanding by learning how to compare text and images, and can learn the semantic relationship between text and images without any supervised labels.

[0061] It should be noted that complex semantic description means that the subject in a multimodal model prompt word Prompt will be limited by multiple modifiers, or there will be multiple attachments interacting with the subject, such as "a lady wearing a white hat and a black and white checkered top riding a white bicycle". Since most of the complex semantics are not in the semantic space that can be understood by the large model, it is challenging for the image segmentation model, i.e., the SAM model, to recall targets with complex semantic descriptions. After obtaining the multimodal prompt word with complex semantics, this embodiment first performs low-order semantic extraction. The phrases extracted from the low-order semantics are usually in the semantic space that can be understood by the SAM model, and the targets corresponding to the recalled low-order semantics include the targets matched by the original prompt.

[0062] Specifically, we first obtain multimodal prompt words with complex semantics, perform semantic extraction on the multimodal prompt words, extract entities and entity relationships in the multimodal prompt words, and obtain multiple low-order semantic entity relationship phrases, where a multimodal prompt word (Prompt) is a natural language description used to guide a multimodal model to perform a specific task. A multimodal prompt word is an instruction used to guide the model to perform specific operations or tasks on the data. An entity refers to a word or phrase with a specific meaning in the text, usually representing a specific object such as a person, place, organization, or object. An entity relationship refers to the semantic connection between entities, describing the actions, states, and positions between entities.

[0063] Step 102: Obtain a picture set associated with the multimodal prompt word, annotate the picture set according to the entity relationship phrase, and obtain a first image target frame.

[0064] In an embodiment of the present invention, a picture set associated with a multimodal prompt word is obtained. A large number of unlabeled pictures related to the prompt scene can be collected or gathered as the picture set, and the picture set and low-order semantic entity relationship phrases are input into the image segmentation model SAM model. The SAM model is used to annotate the picture set, and the output image target frame is the first image target frame, wherein the first image target frame is the initial target frame obtained by annotating the picture set with the entity relationship phrase.

[0065] Step 103 , calculating the image-text similarity between the first image target frame and the entity relationship phrase, and screening the first image target frame according to a predetermined image-text similarity lower limit to obtain a second image target frame to be processed.

[0066] In an embodiment of the present invention, a pre-trained contrastive language image model is used to calculate the image-text similarity between a first image target frame and an entity relationship phrase to obtain the image-text similarity value of the first image target frame. Specifically, a pre-trained contrastive language image model (such as CLIP) is used to calculate the image-text similarity between each first image target frame and the entity relationship phrase, and the minimum similarity value is taken as the image-text similarity value of the first image target frame. The image-text similarity value of each first image target frame is compared with a predetermined image-text similarity lower limit, and the comparison result is recorded. It is determined whether the image-text similarity value of each target frame is less than the image-text similarity lower limit. After comparing the image-text similarity value with the predetermined image-text similarity lower limit, a target frame that meets the conditions is screened out to obtain a second image target frame to be processed, and the second image target frame is an image target frame screened from the first image target frame according to the image-text similarity value being less than the image-text similarity lower limit.

[0067] Step 104 , fusing the second image object frames to obtain multiple image object clusters, and generating fused object frames corresponding to the multiple image object clusters.

[0068] In an embodiment of the present invention, the second image target frame is mapped to a preset area map, which is a blank area. The preset area map can include all second image target frames, so that the image target frames are mapped and merged into the same picture. Target cluster fusion is performed in the preset area map to obtain multiple image target clusters, wherein image target cluster fusion is performed under the condition that any two image target frames overlap and the overlap degree of any two image target frames is greater than a preset threshold. For each image target cluster, the coordinates of the minimum left boundary, the minimum upper boundary, the maximum right boundary and the maximum lower boundary of all image target frames in the image target cluster are calculated to form a new image target frame as a fused target frame. The fused target frame is a maximum circumscribed rectangle containing all image target frames in the image target cluster.

[0069] Step 105 , ranking the fused target frames according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determining the target image target frame corresponding to the multimodal prompt word.

[0070] In an embodiment of the present invention, the confidence of each image target cluster is calculated. The confidence can be calculated by the average confidence or weighted average confidence of the image target boxes, which is not specifically limited here. The number of image target boxes in each image target cluster is counted, and the preset priority is that the confidence priority is higher than the image target box number priority. According to the preset priorities of the confidence and the number of image target boxes, the fused target boxes corresponding to the image target clusters are ranked, and the ranking order of the image target clusters is determined. The fused target boxes with a ranking order higher than the preset ranking are determined as the target image target boxes corresponding to the multimodal prompt words. The preset ranking is set according to the target detection accuracy, which is not specifically limited here.

[0071] The image target detection method provided by the embodiment of the present invention obtains multimodal prompt words with complex semantics, performs semantic extraction on the multimodal prompt words, obtains multiple entity relationship phrases with low-order semantics, obtains a picture set associated with the multimodal prompt words, annotates the picture set according to the entity relationship phrases, obtains a first image target frame, calculates the image-text similarity between the first image target frame and the entity relationship phrase, screens the first image target frame according to a predetermined image-text similarity lower limit, obtains a second image target frame to be processed, fuses the second image target frame to obtain multiple image target clusters, generates fused target frames corresponding to the multiple image target clusters, ranks the fused target frames according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determines the target image target frame corresponding to the multimodal prompt word. The embodiment of the present invention decomposes complex semantic prompt words into multiple low-order semantic entity relationship phrases, performs image segmentation based on the low-order semantic phrases, and identifies target frames related to the low-order semantic phrases, so that the model can cut in from the low-order semantics to recall the target, which is beneficial to improving the model's recall ability for targets corresponding to complex semantic descriptions, and uses image-text similarity to screen out target frames with high correlation with low-order semantic phrases, thereby filtering out irrelevant targets and improving the accuracy of the model's recall results. By fusing multiple low-order semantic recall results, targets that meet the multimodal prompt words are determined, thereby improving the accuracy of detecting targets with complex semantic descriptions.

[0072] Further, see Figure 2 , showing Figure 1 A flowchart of step 101 in a method for detecting an image target is provided. The method is substantially the same as the method for detecting an image target provided in the first embodiment of the present invention. Step 101 may include:

[0073] Step 1011, obtain multimodal prompt words with complex semantics, perform word segmentation and part-of-speech tagging on the multimodal prompt words, and identify entities in the multimodal prompt words.

[0074] Step 1012, extracting the relationship between entities in the multimodal prompt words to obtain entity relationships.

[0075] Step 1013, combining entities and entity relationships to obtain multiple entity relationship phrases with low-level semantics.

[0076] It should be noted that, in an embodiment of the present invention, multimodal prompt words with complex semantics are obtained, and the multimodal prompt words are segmented and tagged with parts of speech. Specifically, a segmentation tool can be used to segment the prompt words, and a part-of-speech tagging tool can be used to tag the segmentation results with parts of speech, to identify entities in the prompt words, and to identify the relationships such as actions, positions, and attributes between entities, to combine entities and relationships into low-order semantic entity relationship phrases, thereby decomposing the prompt words with complex semantics into smaller units, facilitating subsequent entity recognition and relationship extraction, identifying key entities in the prompt words, and providing a basis for subsequent relationship extraction.

[0077] For example, taking the multimodal prompt word "A girl carrying a pink backpack and wearing a white shirt is reading on the grass" as an example, the word segmentation results include A, girl, carrying, a, pink, backpack, and, wearing, a, white, shirt, is, reading, on, the, grass; the part-of-speech tagging results include DET, NOUN, VERB, DET, ADJ, NOUN, CONJ, VERB, DET, ADJ, NOUN, AUX, VERB, ADP, DET, NOUN; the entity recognition results include girl, pink backpack, white shirt, grass; the relationship extraction results include carrying, wearing, reading on, and the entities and relations are combined into low-order semantic entity relationship phrases to obtain a set of low-order semantic entity relationship phrases, for example: "A girl carrying a pink backpack", "A girl wearing a white shirt", "A girl reading on the grass"

[0078] The embodiment of the present invention decomposes the complex semantic prompt words into multiple low-order semantic entity relationship phrases. The low-order semantic entity relationship phrases are easier to understand and process, which reduces the difficulty of complex semantic description and facilitates subsequent image segmentation and target positioning.

[0079] Further, see Figure 3 , showing Figure 1A flowchart of step 102 in a method for detecting an image target is provided. The method is substantially the same as the method for detecting an image target provided in the first embodiment of the present invention. Step 102 may include:

[0080] Step 1021, obtaining a set of pictures associated with the multimodal prompt words.

[0081] Step 1022: Use a pre-trained image segmentation model to segment the images in the image set, and identify segmented images corresponding to low-order semantic phrases.

[0082] Step 1023 , annotate the segmented image with a boundary box to obtain a first image target box.

[0083] It should be noted that, in the embodiment of the present invention, a picture set is obtained according to a multimodal prompt word, and a picture set associated with the multimodal prompt word can be obtained from a database or a network. The picture set should contain pictures related to the scene described by the prompt word. Specifically, a pre-trained image segmentation model (such as SAM) can be used to perform image segmentation on each picture in the picture set, identify the segmented image corresponding to the low-order semantic phrase, annotate the segmented image with a bounding box, and output a first image target box. The first image target box is the initial target box obtained by annotating the picture set with the entity relationship phrase.

[0084] For example, the picture set includes 100 pictures related to the scenes described by the prompt words, and the set of low-order semantic entity relationship phrases includes "A girl carrying a pink backpack", "A girl wearing a white shirt", and "A girl reading on the grass". A pre-trained image segmentation model (such as SAM) is used to perform image segmentation on each picture, and the segmented images corresponding to the low-order semantic phrases are identified. For example, the SAM model identifies the girl carrying a pink backpack and generates a segmented image. The SAM model identifies the girl wearing a white shirt and generates a segmented image. A bounding box is annotated for each segmented image to generate a first image target box. The bounding box annotation can be achieved by calculating the minimum bounding rectangle of the segmented image. Specifically, based on the minimum bounding rectangle of the segmented image, a target box [x1, y1, x2, y2] is generated, which will not be described one by one here.

[0085] The embodiment of the present invention segments the image through a pre-trained image segmentation model, identifies the targets corresponding to the low-order semantic phrases, annotates the segmented image with bounding boxes, and generates an initial image target box to facilitate subsequent image-text similarity calculation and target screening.

[0086] Further, see Figure 4 , showing Figure 1A flowchart of step 103 in a method for detecting an image target is provided. The method is substantially the same as the method for detecting an image target provided in the first embodiment of the present invention. Step 103 may include:

[0087] Step 1031 , using a pre-trained contrastive language image model to calculate the image-text similarity between the first image target frame and the entity relationship phrase, to obtain the image-text similarity value of the first image target frame.

[0088] Step 1032: Compare the image-text similarity value of the first image target frame with a predetermined image-text similarity lower limit to obtain a comparison result.

[0089] Step 1033 , filter the image target frames whose image-text similarity values ​​in the comparison results are less than the image-text similarity lower limit to obtain a second image target frame to be processed.

[0090] It should be noted that, in an embodiment of the present invention, a pre-trained contrastive language image model is used to calculate the image-text similarity between the first image target box and the entity relationship phrase to obtain the image-text similarity value of the first image target box. Specifically, a pre-trained contrastive language image model (such as CLIP) is used to calculate the image-text similarity between each first image target box and the entity relationship phrase, and the minimum similarity value is taken as the image-text similarity value of the first image target box. The image-text similarity is calculated by the pre-trained contrastive language image model to evaluate the correlation between the target box and the low-order semantic phrase.

[0091] Specifically, the image-text similarity value of each first image target frame is compared with a predetermined image-text similarity lower limit, the comparison result is recorded, and it is determined whether the image-text similarity value of each target frame is less than the image-text similarity lower limit. After comparing the image-text similarity value with the predetermined image-text similarity lower limit, the target frames that meet the conditions are screened out, and the comparison result includes the comparison result of the image-text similarity value of each first image target frame with the image-text similarity lower limit. The image target frames whose image-text similarity values ​​are less than the image-text similarity lower limit in the comparison result are screened out to obtain a set of second image target frames to be processed. Through the comparison operation, the target frames with higher correlation with low-order semantic phrases are screened out to improve the accuracy of target positioning.

[0092] The embodiment of the present invention calculates the image-text similarity through a pre-trained contrastive language image model, evaluates the relevance of a target frame with a low-order semantic phrase, and uses the image-text similarity value to perform a screening operation on the image target frame, thereby screening out target frames with a high relevance to the low-order semantic phrase, reducing the interference of irrelevant target frames, and improving the accuracy of target positioning.

[0093] Specifically, the predetermined lower limit of image-text similarity is obtained by the following steps:

[0094] First, obtain a pre-collected picture set and the entity relationship phrase;

[0095] Secondly, the picture set and the entity relationship phrase are input into a pre-trained contrastive language image model, and a set of image-text similarity values ​​between the entity relationship phrase and the picture set are output;

[0096] Secondly, the minimum value among the image-text similarity values ​​is determined as the image-text similarity lower limit.

[0097] It should be noted that in the above steps, a pre-collected picture set and the entity relationship phrases are obtained, wherein the pre-collected picture set can be a small number of pictures related to the scene described by the multimodal prompt words provided by the user, and the low-order semantic entity relationship phrases extracted based on the multimodal prompt words, the picture set and the entity relationship phrases are input into a pre-trained contrastive language image model, and the pre-trained contrastive language image model (such as CLIP) is used to calculate the image-text similarity between each entity relationship phrase and each picture. For each entity relationship phrase, its similarity value with each picture is calculated, and a set of image-text similarity values ​​between the entity relationship phrase and the picture set are output, that is, the similarity value between each entity relationship phrase and each picture. For each entity relationship phrase, the minimum value of the similarity values ​​calculated with all pictures and the minimum value of the minimum values ​​of all entity relationship phrases are taken as the lower limit of the image-text similarity.

[0098] Specifically, the LLM large semantic model is first used to obtain the entity relationship phrase set corresponding to the multimodal prompt word. The entity relationship phrase set is represented by S. At the same time, the user provides a small number of pre-collected picture sets that are similar to the multimodal prompt word descriptions. The pre-collected picture set is represented by I. Each sample in S and I is input into the CLIP model to obtain a set of similarity thresholds. The minimum value is taken as the lower limit of similarity. The lower limit of image-text similarity Determined by the following formula:

[0099]

[0100] in, is the lower limit of image-text similarity, S i is a sample in the entity relationship phrase set, I j are samples from a pre-collected image set.

[0101] Further, see Figure 5 , showing Figure 1 A flowchart of step 104 in a method for detecting an image target is provided. The method is substantially the same as the method for detecting an image target provided in the first embodiment of the present invention. Step 104 may include:

[0102] Step 1041, mapping the second image target frame to a preset area map, performing target cluster fusion in the preset area map, and obtaining multiple image target clusters; wherein any two image target frames in the image target cluster overlap, and the overlap degree of any two image target frames is greater than a preset threshold.

[0103] Step 1042, determining the boundary coordinates of each image object cluster, and generating a fused object box corresponding to each image object cluster.

[0104] It should be noted that, in an embodiment of the present invention, the second image target frame is mapped to a preset area map, which is a blank area. The preset area map can include all second image target frames, so that the image target frame mappings are merged into the same picture, and target cluster fusion is performed in the preset area map to obtain multiple image target clusters; wherein, any two image target frames in the image target cluster overlap, and the overlap degree of any two image target frames is greater than a preset threshold, and the preset threshold value of the overlap degree IoU (Intersection Over Union) is 0.5 for illustration.

[0105] Specifically, for each image target cluster, its boundary coordinates are calculated, and the boundary coordinates include the minimum left boundary, the minimum upper boundary, the maximum right boundary and the maximum lower boundary. According to the boundary coordinates, a fused target frame corresponding to each image target cluster is generated, wherein the fused target frame is, for each image target cluster, the coordinates of the minimum left boundary, the minimum upper boundary, the maximum right boundary and the maximum lower boundary of all image target frames in the image target cluster are calculated to form a new image target frame, and the fused target frame is a maximum circumscribed rectangle containing all image target frames in the image target cluster. Specifically, target cluster fusion is performed in a preset region map to obtain multiple image target clusters. For example, the IoU of target frame 1 and target frame 3 is calculated. If the IoU is greater than a preset threshold (for example, 0.5), target frame 1 and target frame 3 are merged into one image target cluster. Through the target cluster fusion operation, target frames with higher overlap are merged into one image target cluster. By fusing multiple target frames with low-order semantic recall, the recall rate of target positioning is improved.

[0106] It should be noted that, refer to Figure 7, shows a scene schematic diagram of the image target detection method provided by an embodiment of the present invention, maps all low-order semantically recalled image target frames to a large image of a preset area, and searches for all image target clusters that meet preset conditions, wherein the preset conditions include that any two frames in the cluster overlap, and the IoU between any two frames is greater than a preset threshold. The image target cluster search process is as follows: select one with the highest confidence from all low-order semantically recalled image target frames as the initial seed to start the search. If there are multiple image target frames with the highest confidence, randomly select one as the initial seed; continuously add the searched image target frames to the seed set until no new image target frames are found, forming a new image target cluster, and judging whether there are image target frames that have not been used as seeds. If not, all image target clusters have been found, otherwise select new seeds from image target frames that have not been searched to start a new round of search until all image target clusters are found.

[0107] The embodiment of the present invention fuses the filtered image target frames, merges the target frames with high overlap into an image target cluster, generates a fused target frame, reduces the interference of irrelevant target frames, improves the accuracy of target positioning, and further improves the recall rate of target positioning.

[0108] Further, see Figure 6 , showing Figure 1 A flowchart of step 105 in a method for detecting an image target is provided. The method is substantially the same as the method for detecting an image target provided in the first embodiment of the present invention. Step 105 may include:

[0109] Step 1051 , determining the confidence of the image object cluster and the number of image object boxes in the image object cluster.

[0110] Step 1052: rank the fused target frames corresponding to the image target clusters according to the preset priorities of confidence and the number of image target frames to obtain a ranking order.

[0111] Step 1053: determine the fusion target frame whose ranking order is higher than the preset ranking as the target image target frame corresponding to the multimodal prompt word.

[0112] It should be noted that, in an embodiment of the present invention, for each image target cluster, its confidence is calculated. The confidence can be calculated by the average confidence or weighted average confidence of the image target boxes, which is not specifically limited here. For each image target cluster, the number of image target boxes therein is counted, and the reliability of the target cluster is evaluated by calculating the confidence of the image target cluster. The number of image target boxes in the image target cluster is counted to provide a basis for subsequent ranking.

[0113] Specifically, the preset priority is that the confidence priority is higher than the image target box quantity priority. According to the preset priorities of confidence and the number of image target boxes, the fused target boxes corresponding to the image target cluster are ranked. First, they are sorted according to the confidence, and the ones with higher confidence are ranked in front. If the confidence is the same, they are sorted according to the number of image target boxes, and the ones with more are ranked in front. The ranking order of the image target cluster is determined, and the fused target box with a ranking higher than the preset ranking is determined as the target image target box corresponding to the multimodal prompt word. The preset ranking is set according to the target detection accuracy, which will not be elaborated here.

[0114] The embodiment of the present invention evaluates the reliability of the image target cluster by calculating the confidence of the image target cluster and the number of image target boxes in the image target cluster, determines the priority of the image target cluster, and determines the target image target box corresponding to the multimodal prompt word through the ranking order.

[0115] Reference Figure 8 , shows a schematic diagram of the structure of an image target detection device provided by an embodiment of the present invention, the device comprising:

[0116] The semantic extraction module 201 is used to obtain multimodal prompt words with complex semantics, perform semantic extraction on the multimodal prompt words, and obtain multiple entity relationship phrases with low-order semantics;

[0117] A picture annotation module 202 is used to obtain a picture set associated with the multimodal prompt word, annotate the picture set according to the entity relationship phrase, and obtain a first image target frame;

[0118] The target screening module 203 is used to calculate the image-text similarity between the first image target frame and the entity relationship phrase, and screen the first image target frame according to a predetermined image-text similarity lower limit to obtain a second image target frame to be processed;

[0119] An object fusion module 204 is used to fuse the second image object frame to obtain multiple image object clusters, and generate fused object frames corresponding to the multiple image object clusters;

[0120] The target determination module 205 is used to rank the fused target frames according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determine the target image target frame corresponding to the multimodal prompt word.

[0121] Furthermore, the semantic extraction module 201 includes:

[0122] The recognition submodule is used to obtain multimodal prompt words with complex semantics, perform word segmentation and part-of-speech tagging on the multimodal prompt words, and recognize entities in the multimodal prompt words;

[0123] An extraction submodule, used to extract the relationship between entities in the multimodal prompt words to obtain entity relationships;

[0124] The combination submodule is used to combine the entity and the entity relationship to obtain a plurality of entity relationship phrases with low-order semantics.

[0125] Furthermore, the picture annotation module 202 includes:

[0126] A first acquisition submodule is used to acquire a picture set associated with the multimodal prompt word;

[0127] A segmentation submodule, configured to segment the images in the image set using a pre-trained image segmentation model, and identify segmented images corresponding to low-order semantic phrases;

[0128] The labeling submodule is used to label the segmented image with a boundary box to obtain a first image target box.

[0129] Furthermore, the target screening module 203 includes:

[0130] A calculation submodule, configured to calculate the image-text similarity between the first image target frame and the entity relationship phrase using a pre-trained contrastive language image model, and obtain the image-text similarity value of the first image target frame;

[0131] A comparison submodule, used for comparing the image-text similarity value of the first image target frame with a predetermined image-text similarity lower limit to obtain a comparison result;

[0132] The screening submodule is used to screen the image target frames whose image-text similarity values ​​in the comparison results are less than the image-text similarity lower limit, and obtain the second image target frame to be processed.

[0133] Furthermore, in the target screening module 203, the predetermined lower limit of the image-text similarity is obtained by the following steps:

[0134] A second acquisition submodule is used to acquire a pre-collected picture set and the entity relationship phrase;

[0135] A processing submodule, configured to input the picture set and the entity relationship phrase into a pre-trained contrastive language image model, and output a set of image-text similarity values ​​between the entity relationship phrase and the picture set;

[0136] The first determining submodule is used to determine the minimum value among the image-text similarity values ​​as the image-text similarity lower limit.

[0137] Furthermore, the target fusion module 204 includes:

[0138] A fusion submodule, used for mapping the second image target frame to a preset area map, performing target cluster fusion in the preset area map, and obtaining a plurality of image target clusters; wherein any two image target frames in the image target clusters overlap, and the overlap degree of any two image target frames is greater than a preset threshold;

[0139] The generation submodule is used to determine the boundary coordinates of each of the image object clusters and generate a fused object frame corresponding to each of the image object clusters.

[0140] Furthermore, the target determination module 205 includes:

[0141] A second determination submodule, used to determine the confidence of the image object cluster and the number of image object boxes in the image object cluster;

[0142] A ranking submodule, used to rank the fused target frames corresponding to the image target clusters according to preset priorities of confidence and the number of image target frames to obtain a ranking order;

[0143] The third determination submodule is used to determine the fusion target frame whose ranking order is higher than the preset ranking as the target image target frame corresponding to the multimodal prompt word.

[0144] The image target detection device provided by the embodiment of the present invention obtains multimodal prompt words with complex semantics, performs semantic extraction on the multimodal prompt words, obtains multiple entity relationship phrases with low-order semantics, obtains a picture set associated with the multimodal prompt words, annotates the picture set according to the entity relationship phrases, obtains a first image target frame, calculates the image-text similarity between the first image target frame and the entity relationship phrase, screens the first image target frame according to a predetermined image-text similarity lower limit, obtains a second image target frame to be processed, fuses the second image target frame to obtain multiple image target clusters, generates fused target frames corresponding to the multiple image target clusters, ranks the fused target frames according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determines the target image target frame corresponding to the multimodal prompt word. The embodiment of the present invention decomposes complex semantic prompt words into multiple low-order semantic entity relationship phrases, performs image segmentation based on the low-order semantic phrases, and identifies target frames related to the low-order semantic phrases, so that the model can cut in from the low-order semantics to recall the target, which is beneficial to improving the model's recall ability for targets corresponding to complex semantic descriptions, and uses image-text similarity to screen out target frames with high correlation with low-order semantic phrases, thereby filtering out irrelevant targets and improving the accuracy of the model's recall results. By fusing multiple low-order semantic recall results, targets that meet the multimodal prompt words are determined, thereby improving the accuracy of detecting targets with complex semantic descriptions.

[0145] Reference Fig. 9, an embodiment of the present invention further provides an electronic device, such as Fig. 9 As shown, it includes a processor 301, a communication interface 302, a memory 303 and a communication bus 304, wherein the processor 301, the communication interface 302, and the memory 303 communicate with each other through the communication bus 304.

[0146] A processor 301, a memory 303 for storing processor executable instructions;

[0147] The processor 301 is configured to execute the instructions to implement the image target detection method as described above:

[0148] Acquire multimodal prompt words with complex semantics, perform semantic extraction on the multimodal prompt words, and obtain multiple entity relationship phrases with low-order semantics;

[0149] Acquire a picture set associated with the multimodal prompt word, and annotate the picture set according to the entity relationship phrase to obtain a first image target frame;

[0150] Calculating the image-text similarity between the first image target frame and the entity relationship phrase, and screening the first image target frame according to a predetermined image-text similarity lower limit to obtain a second image target frame to be processed;

[0151] Fusing the second image object frames to obtain multiple image object clusters, and generating multiple fused object frames corresponding to the image object clusters;

[0152] The fused target frame is ranked according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and the target image target frame corresponding to the multimodal prompt word is determined.

[0153] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0154] The communication interface is used for communication between the above terminal and other devices.

[0155] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0156] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0157] In another embodiment provided by the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, the image target detection method described in any of the above embodiments is implemented.

[0158] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0159] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0160] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0161] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for detecting an image target, characterized in that: The method comprises: Acquire multimodal prompt words with complex semantics, perform semantic extraction on the multimodal prompt words, and obtain multiple entity relationship phrases with low-order semantics; Acquire a picture set associated with the multimodal prompt word, and annotate the picture set according to the entity relationship phrase to obtain a first image target frame; Calculating the image-text similarity between the first image target frame and the entity relationship phrase, and screening the first image target frame according to a predetermined image-text similarity lower limit to obtain a second image target frame to be processed; Fusing the second image object frames to obtain multiple image object clusters, and generating multiple fused object frames corresponding to the image object clusters; The fused target frame is ranked according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and the target image target frame corresponding to the multimodal prompt word is determined.

2. The method according to claim 1, characterized in that: The step of acquiring a multimodal prompt word with complex semantics and performing semantic extraction on the multimodal prompt word to obtain a plurality of entity relationship phrases with low-order semantics includes: Acquire multimodal prompt words with complex semantics, perform word segmentation and part-of-speech tagging on the multimodal prompt words, and identify entities in the multimodal prompt words; Extracting relationships between entities in the multimodal prompt words to obtain entity relationships; The entities and the entity relationships are combined to obtain a plurality of entity relationship phrases with low-order semantics.

3. The method according to claim 1, characterized in that: The step of obtaining a picture set associated with the multimodal prompt word, and labeling the picture set according to the entity relationship phrase to obtain a first image target frame includes: Obtaining a set of pictures associated with the multimodal prompt words; Using a pre-trained image segmentation model to segment the images in the image set, and identifying segmented images corresponding to low-order semantic phrases; The segmented image is annotated with a boundary box to obtain a first image target box.

4. The method according to claim 1, characterized in that: The calculating the image-text similarity between the first image target frame and the entity relationship phrase, and screening the first image target frame according to a predetermined image-text similarity lower limit to obtain a second image target frame to be processed, includes: Using a pre-trained contrastive language image model to calculate the image-text similarity between the first image target frame and the entity relationship phrase, to obtain the image-text similarity value of the first image target frame; Comparing the image-text similarity value of the first image target frame with a predetermined image-text similarity lower limit to obtain a comparison result; The image target frames whose image-text similarity values ​​in the comparison results are less than the image-text similarity lower limit are screened to obtain a second image target frame to be processed.

5. The method according to claim 4, characterized in that: The predetermined lower limit of image-text similarity is obtained by the following steps: Acquire a pre-collected picture set and the entity relationship phrase; Inputting the picture set and the entity relationship phrase into a pre-trained contrastive language image model, and outputting a set of image-text similarity values ​​between the entity relationship phrase and the picture set; The minimum value among the image-text similarity values ​​is determined as the image-text similarity lower limit.

6. The method according to claim 1, characterized in that: The step of fusing the second image object frame to obtain a plurality of image object clusters and generating a plurality of fused object frames corresponding to the image object clusters includes: Mapping the second image target frame to a preset region map, performing target cluster fusion in the preset region map, and obtaining a plurality of image target clusters; wherein any two image target frames in the image target clusters overlap, and the overlap degree of any two image target frames is greater than a preset threshold; The boundary coordinates of each of the image object clusters are determined, and a fused object frame corresponding to each of the image object clusters is generated.

7. The method according to claim 1, characterized in that: The step of ranking the fused target frames according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determining the target image target frame corresponding to the multimodal prompt word, includes: Determining the confidence of the image object cluster and the number of image object boxes in the image object cluster; Ranking the fused target frames corresponding to the image target clusters according to preset priorities of confidence and the number of image target frames to obtain a ranking order; A fusion target frame whose ranking order is higher than a preset ranking is determined as a target image target frame corresponding to the multimodal prompt word.

8. An image target detection device, characterized in that: The device comprises: A semantic extraction module is used to obtain multimodal prompt words with complex semantics, perform semantic extraction on the multimodal prompt words, and obtain multiple entity relationship phrases with low-order semantics; A picture annotation module, used to obtain a picture set associated with the multimodal prompt word, annotate the picture set according to the entity relationship phrase, and obtain a first image target frame; A target screening module, used for calculating the image-text similarity between the first image target frame and the entity relationship phrase, screening the first image target frame according to a predetermined image-text similarity lower limit, and obtaining a second image target frame to be processed; An object fusion module, used to fuse the second image object frame to obtain multiple image object clusters, and generate fused object frames corresponding to the multiple image object clusters; The target determination module is used to rank the fused target frame according to the confidence of the image target cluster and the number of image target frames in the image target cluster, and determine the target image target frame corresponding to the multimodal prompt word.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images