Target detection method, device and equipment

Abstract

The invention provides a target detection method, device and equipment, and the method comprises the steps: determining a target entity based on a cue word text, and determining an equivalent entity and a misinformation entity corresponding to the target entity; obtaining a first detection frame corresponding to the target entity, a second detection frame corresponding to the equivalent entity and a third detection frame corresponding to the misinformation entity from the to-be-detected image; determining a target detection frame based on the overlapping degree among the first detection frame, the second detection frame and the third detection frame; and intercepting a detection sub-image of the target detection frame from the to-be-detected image, determining target attribute information based on the detection sub-image, and outputting the target attribute information and the target detection frame. According to the technical scheme of the invention, the false alarm detection box can be effectively eliminated, and the modeling capability of semantic consistency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a target detection method, device and equipment. BACKGROUND

[0002] Target detection refers to locating and classifying all entity objects (i.e. target of interest) in a to-be-detected image, and outputting the detection box coordinates and class label of the entity object. The detection box is also called a bounding box. For example, the detection box coordinates of the entity object can be output, such as (x1, y1) and (x2, y2), where (x1, y1) represents the coordinates of the upper left corner of the detection box, and (x2, y2) represents the coordinates of the lower right corner of the detection box, or (x, y, w, h), where (x, y) represents the center point coordinates of the detection box, w represents the width of the detection box, and h represents the height of the detection box. The class label of the entity object can be output, such as "car", "pedestrian", etc. The confidence of the entity object can also be output, which represents the probability of the existence of the entity object.

[0003] In order to realize target detection, the to-be-detected image can be input into a target detection model, and the target detection model can process the to-be-detected image to output the detection box coordinates and class label of the entity object. However, the target detection model can only detect entity objects with fixed class labels, and cannot detect entity objects with arbitrary class labels. For example, the target detection model supports detecting entity objects with class labels such as "car" and "pedestrian", but does not support detecting entity objects with class labels such as "small car" and "human body". SUMMARY

[0004] The present application provides a target detection method, which comprises: determining a target entity based on an acquired prompt word text, determining an equivalent entity and a false positive entity corresponding to the target entity; wherein the equivalent entity and the target entity correspond to the same entity object, and the false positive entity is an entity object that produces false positives for the target entity; acquiring a first detection box corresponding to the target entity, a second detection box corresponding to the equivalent entity, and a third detection box corresponding to the false positive entity from an acquired to-be-detected image; determining a target detection box based on the overlapping degree between the first detection box, the second detection box and the third detection box; cutting a detection sub-image of the target detection box from the to-be-detected image, determining target attribute information based on the detection sub-image, and outputting the target attribute information and the target detection box.

[0005] The application provides a target detection device, the device comprising: a determination module configured to determine a target entity based on an obtained prompt word text, determine an equivalent entity and a false positive entity corresponding to the target entity; wherein the equivalent entity and the target entity correspond to the same entity object, and the false positive entity is an entity object that produces false positives for the target entity; an acquisition module configured to acquire a first detection box corresponding to the target entity, a second detection box corresponding to the equivalent entity, and a third detection box corresponding to the false positive entity from a to-be-detected image; a determination module configured to determine a target detection box based on an overlapping degree between the first detection box, the second detection box, and the third detection box; a processing module configured to cut a detection sub-image of the target detection box from the to-be-detected image, determine target attribute information based on the detection sub-image, and output the target attribute information and the target detection box.

[0006] The application provides an electronic device, comprising: a processor and a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by the processor; the processor is configured to execute the machine executable instructions to implement the target detection method of the above examples.

[0007] The application provides a computer program product, which can comprise a computer program, the computer program being executed by a processor to implement the target detection method of the above examples.

[0008] The application provides a machine readable storage medium, the machine readable storage medium storing machine executable instructions capable of being executed by a processor; wherein the processor is configured to execute the machine executable instructions, and the machine executable instructions, when executed, implement the target detection method of the above examples.

[0009] As can be seen from the above technical solutions, in the embodiments of the present application, the equivalent entity corresponding to the target entity can be determined. The equivalent entity and the target entity belong to the same entity object. The target detection frame is determined based on the degree of overlap between the first detection frame of the target entity and the second detection frame of the equivalent entity, thereby performing target recognition based on the target entity and the equivalent entity. The category of the equivalent entity can be unified with the category of the target entity, and synonymous semantics are integrated to supplement the category, thereby improving the modeling capability of semantic consistency and solving the problem of attenuation of the recognition capability of unknown category attributes. It is particularly suitable for entity recognition of unknown category targets in dynamic open environments and the joint determination of their multi-dimensional attributes (such as color, action, spatial position), ensuring cross-modal and synonymous semantic performance. In addition, the false positive entity corresponding to the target entity is determined. The false positive entity is an entity object that generates a false positive for the target entity. The target detection frame is determined based on the degree of overlap between the first detection frame of the target entity and the third detection frame of the false positive entity, thereby performing target recognition based on the target entity and the false positive entity, effectively eliminating the false positive detection frame, effectively reducing the false detection rate of synonymous prompt words, effectively reducing the probability of entity recognition errors, significantly reducing the false positive problem of visually similar objects, and improving detection accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a flow chart of a target detection method in one embodiment of the present application; Figure 2 is a flow chart of a target detection method in one embodiment of the present application; Figure 3 Schematic diagram of a target detection framework in one embodiment of the present application; Figure 4 is a schematic diagram of a target detection process in one embodiment of the present application; Figure 5 This is a flow chart of automatic threshold adjustment in one embodiment of the present application; Figure 6 This is a schematic structural diagram of a target detection device in one embodiment of the present application; Figure 7 It is a hardware structure diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0011] In the embodiment of the present application, a target detection method is proposed, which can be applied to electronic devices. Figure 1 FIG. 1 is a flow chart of the target detection method, which may include: Step 101, determining a target entity based on the obtained prompt word text, determining an equivalent entity and a false positive entity corresponding to the target entity; the equivalent entity and the target entity correspond to the same entity object (i.e. different expressions for the same entity object), and the false positive entity is an entity object that produces false positives for the target entity.

[0012] Step 102, obtaining a first detection box corresponding to the target entity, a second detection box corresponding to the equivalent entity, and a third detection box corresponding to the false positive entity from the obtained image to be detected.

[0013] Step 103, determining a target detection box based on the overlapping degree between the first detection box, the second detection box and the third detection box.

[0014] Exemplarily, a candidate detection box can be determined based on the overlapping degree between the first detection box and the second detection box; wherein the candidate detection box includes the first detection box, or the candidate detection box includes the second detection box, or the candidate detection box includes the first detection box and the second detection box. The target detection box is determined based on the overlapping degree between the candidate detection box and the third detection box; wherein the target detection box includes the remaining candidate detection box filtered based on the overlapping degree, that is, for each candidate detection box, the candidate detection box can be taken as the target detection box, or the candidate detection box is filtered and not taken as the target detection box.

[0015] Step 104, cutting a detection sub-image of the target detection box from the image to be detected, determining target attribute information based on the detection sub-image, and outputting the target attribute information and the target detection box, wherein the target attribute information can represent the attribute of the target detection box, and the target attribute information and the target detection box represent the target recognition result.

[0016] Exemplarily, determining the target detection box based on the overlapping degree between the first detection box, the second detection box and the third detection box can include but is not limited to: determining a first overlapping degree between the first detection box and the second detection box; if the first overlapping degree is less than a first threshold, the first detection box and the second detection box can be determined as candidate detection boxes; if the first overlapping degree is not less than the first threshold, and the confidence of the first detection box is not less than the confidence of the second detection box, the first detection box can be determined as a candidate detection box; if the first overlapping degree is not less than the first threshold, and the confidence of the first detection box is less than the confidence of the second detection box, the second detection box can be determined as a candidate detection box. A second overlapping degree between the candidate detection box and the third detection box is determined, and a weighted operation is performed on the second overlapping degree and the confidence of the third detection box to obtain a false positive score; if the false positive score is not greater than a second threshold, the candidate detection box is determined as the target detection box; if the false positive score is greater than the second threshold, the candidate detection box is prohibited from being determined as the target detection box.

[0017] Exemplarily, before determining the equivalent entity and the false positive entity corresponding to the target entity, the obtained multiple sample texts can be clustered to obtain multiple clustering clusters, and for each clustering cluster, the clustering cluster can include at least one sample text belonging to the same entity object; a dynamic semantic enhancement vocabulary table is constructed, and the dynamic semantic enhancement vocabulary table includes multiple first vocabulary items corresponding to the multiple clustering clusters; for each first vocabulary item, the first vocabulary item can include the entity corresponding to each sample text in the clustering cluster. In addition, the obtained multiple sample images can be recognized to obtain the predicted entity of the entity object in each sample image; a false positive suppression vocabulary table can be constructed, and the false positive suppression vocabulary table can include multiple second vocabulary items corresponding to multiple entity objects; for each second vocabulary item, the second vocabulary item can include the corresponding relationship between the real entity of the entity object and the predicted entity of the entity object.

[0018] On this basis, determining the equivalent entity and the false positive entity corresponding to the target entity can include but is not limited to: querying the first vocabulary item corresponding to the target entity from the dynamic semantic enhancement vocabulary table, and determining the entity in the first vocabulary item as the equivalent entity; wherein the similarity between at least one entity in the first vocabulary item and the target entity is greater than a third threshold. In addition, the second vocabulary item corresponding to the target entity can be queried from the false positive suppression vocabulary table, and the predicted entity in the second vocabulary item is determined as the false positive entity; wherein the similarity between the real entity in the second vocabulary item and the target entity can be greater than a fourth threshold.

[0019] Exemplarily, based on the detection sub-image, the target attribute information can be determined, which can include but is not limited to: performing feature extraction on the detection sub-image to obtain a first image feature; performing feature extraction on the target attribute to obtain a first text feature, and the target attribute can be determined based on the prompt word text. Determine the first cross-modal similarity based on the first image feature and the first text feature; determine whether there is a positive sample image corresponding to the target attribute; if yes, the first similarity threshold corresponding to the target attribute can be determined based on the positive sample image corresponding to the target attribute, and if not, the configured default similarity threshold can be determined as the first similarity threshold corresponding to the target attribute; if the first cross-modal similarity is greater than the first similarity threshold, the target attribute is determined as the target attribute information corresponding to the target detection frame.

[0020] Exemplarily, determining the first similarity threshold corresponding to the target attribute based on the positive sample image can include but is not limited to: performing augmentation processing on the positive sample image corresponding to the target attribute to obtain a plurality of augmented positive sample images; performing augmentation processing on the negative sample image corresponding to the target attribute to obtain a plurality of augmented negative sample images; wherein the entity object in each positive sample image corresponds to the target attribute, and the entity object in each negative sample image does not correspond to the target attribute. Based on this, the positive sample cross-modal similarity is determined based on the image feature and the first text feature of the positive sample image, and the negative sample cross-modal similarity is determined based on the image feature and the first text feature of the negative sample image; the first similarity threshold is determined based on the positive sample cross-modal similarity (a plurality of positive sample cross-modal similarities) and the negative sample cross-modal similarity (a plurality of negative sample cross-modal similarities), and the first similarity threshold is used to distinguish the positive sample cross-modal similarity and the negative sample cross-modal similarity, so that the positive sample cross-modal similarity corresponding to all positive sample images is greater than the first similarity threshold, and the negative sample cross-modal similarity corresponding to all negative sample images is less than the first similarity threshold.

[0021] Exemplarily, determining the first similarity threshold based on the positive sample cross-modal similarity and the negative sample cross-modal similarity can include but is not limited to: determining a current distribution similarity threshold based on the positive sample cross-modal similarity and the negative sample cross-modal similarity; if there is no historical distribution similarity threshold, the current distribution similarity threshold can be determined as the first similarity threshold, and the first similarity threshold is used as the historical distribution similarity threshold of the next period; if there is a historical distribution similarity threshold, a divergence value can be determined based on the current distribution similarity threshold and the historical distribution similarity threshold, if the divergence value is greater than a fifth threshold, the current distribution similarity threshold can be determined as the first similarity threshold, if the divergence value is not greater than the fifth threshold, the historical distribution similarity threshold used in the last period can be determined as the first similarity threshold.

[0022] Exemplarily, the divergence value can be determined using the following formula: ; wherein, the current distribution similarity threshold is represented by the s-th historical distribution similarity threshold is represented by S, and the set S represents all historical distribution similarity thresholds, the divergence value is represented by D.

[0023] Exemplarily, the first detection box corresponding to the target entity can be obtained from the acquired image to be detected, which can include but is not limited to: for any candidate box in the image to be detected, performing feature extraction on the candidate box to obtain a second image feature, performing feature extraction on the target entity to obtain a second text feature, and determining a second cross-modal similarity based on the second image feature and the second text feature; determining whether there is a positive sample image corresponding to the target entity; if yes, determining a second similarity threshold corresponding to the target entity based on the positive sample image, and if no, determining a default similarity threshold configured as the second similarity threshold corresponding to the target entity; if the second cross-modal similarity is greater than the second similarity threshold, the candidate box is determined as the first detection box corresponding to the target entity; and if the second cross-modal similarity is not greater than the second similarity threshold, the candidate box is prohibited from being determined as the first detection box corresponding to the target entity.

[0024] Exemplarily, the second similarity threshold corresponding to the target entity can be determined based on the positive sample image, which can include but is not limited to: performing augmentation processing on the positive sample image corresponding to the target entity to obtain a plurality of positive sample images, performing augmentation processing on a negative sample image corresponding to the target entity to obtain a plurality of negative sample images; determining a positive sample cross-modal similarity based on the image feature of the positive sample image and the second text feature, and determining a negative sample cross-modal similarity based on the image feature of the negative sample image and the second text feature; and determining the second similarity threshold based on the positive sample cross-modal similarity and the negative sample cross-modal similarity.

[0025] As can be seen from the above technical solutions, in the embodiments of the present application, the equivalent entity corresponding to the target entity can be determined, the equivalent entity and the target entity belong to the same entity object, the target detection box is determined based on the overlapping degree between the first detection box of the target entity and the second detection box of the equivalent entity, so that the target recognition is performed based on the target entity and the equivalent entity, the classes of the equivalent entity and the target entity are unified, the synonymous semantics are integrated for class supplement, the modeling capability of semantic consistency is improved, the unknown class attribute recognition capability decay problem is solved, and the entity recognition and joint determination of multi-dimensional attributes (such as color, action, and spatial position) of unknown class targets in a dynamic and open environment are especially suitable, and the cross-modal and synonymous semantic performance is guaranteed. In addition, the false positive entity corresponding to the target entity is determined, the false positive entity is an entity object that produces false positives for the target entity, the target detection box is determined based on the overlapping degree between the first detection box of the target entity and the third detection box of the false positive entity, so that the target recognition is performed based on the target entity and the false positive entity, the false positive detection box is effectively eliminated, the false detection rate of synonymous prompt words is effectively reduced, the probability of entity recognition error is effectively reduced, the false positive problem of visually similar objects is significantly reduced, and the detection precision and robustness are improved.

[0026] The above technical solutions of the embodiments of the present application will be described below in combination with specific application scenarios.

[0027] Target detection refers to locating and classifying all entity objects in a to-be-detected image, and outputting the detection box coordinates and class label of the entity object. In order to realize target detection, the to-be-detected image can be input into a target detection model, and the target detection model is used to process the to-be-detected image, and output the detection box coordinates and class label of the entity object. However, the open target detection based on deep learning has the following problems: The semantic sensitivity problem of open target detection. The open target detection model based on text prompts needs to rely on the alignment training of text and image, but is limited by the discrete annotation of the training data, and lacks the ability to model semantic consistency for synonymous text prompts. For example, when inputting the prompt words "human", "people", "person", etc., although these prompt words represent the same semantic concept, the target detection model will regard these prompt words as independent class labels, resulting in deviation of the detection result and reducing the robustness.

[0028] False alarm problem caused by visual feature confusion. Elongated cylindrical objects (such as cigarettes, toothpicks, straws, pens, etc.) have high similarity in visual feature space, and the target detection model lacks explicit false alarm suppression capability, resulting in non-target entities with similar feature expressions being incorrectly identified, which significantly affects the detection accuracy.

[0029] Failure of cross-modal confidence range. The target detection model faces problems such as confidence mechanism misalignment and static threshold defects. The target detection model relies on a fixed confidence threshold (such as 0.5) preset by humans, which leads to the following problems: Modality difference sensitivity: the modality difference between image features and class text semantics causes confidence fluctuations, and the confidence threshold cannot adapt to new scenarios. Narrow adjustment range: fine-tuning of the confidence threshold will cause a sharp increase in false alarm rate or an increase in missed detection rate, and cannot balance the trade-off between precision and recall. Lack of dynamic response: lack of lightweight adaptive mechanism forces manual frequent adjustment of the confidence threshold, resulting in low operational efficiency.

[0030] In view of the above findings, an open target detection and attribute recognition method with scene self-adaptation is proposed in the embodiments of the present application, which has the functions of "open semantic dynamic enhancement" and "cross-modal threshold adaptive adjustment". The method can guarantee the joint recognition accuracy of target attributes in an open environment, and realize the dynamic and stable optimization of the confidence threshold. The embodiments belong to the cross-technology field of artificial intelligence and computer vision, and are especially suitable for entity recognition of unknown class targets and joint determination of multi-dimensional attributes (such as color, action, and spatial position) in a dynamic open environment. The method can guarantee the performance of cross-modal and synonymous semantics, and can be widely applied to automatic driving, security detection, industrial quality inspection, and other scenes that need to adapt to changes in open target categories and attributes in real time.

[0031] In this embodiment, a recognition architecture that decouples entities and attributes is designed, and a cascade topology of requirement parsing function, entity recognition function, and attribute recognition function is designed. Through a structured process, high-precision separation and recognition of target entities and attributes in open scenarios are achieved, rather than entity and attribute coupling training. This solves the problem of attenuated recognition ability of unknown category attributes, eliminates semantic interference between entities and attributes, and improves the performance of the open target detection algorithm.

[0032] In this embodiment, a collaborative mechanism of dynamic semantic enhancement and false positive suppression vocabulary (DS-FPSV) is designed to integrate a synonymous semantic input supplement mechanism (dynamically expanding entity prompt words through a synonym clustering library) and a false positive entity registration and elimination mechanism (constructing an adversarial set of easily confused entities to suppress visual feature confusion), thereby improving the modeling capability of semantic consistency.

[0033] In this embodiment, an adaptive adjustment mechanism for automated thresholds is designed to achieve adaptive optimization of cross-modal confidence based on extremely small samples (e.g., 10 labeled samples). Adaptive optimization of confidence thresholds is achieved through probability-driven amplification preprocessing, cross-modal feature collaborative projection, and closed-form decision boundary calculation, rather than static configuration of confidence thresholds, thereby achieving dynamic and stable optimization of cross-modal confidence.

[0034] In one possible implementation, to implement the "Open Semantic Dynamic Enhancement" feature, a pre-built DS-FPSV (Dynamic Semantic Enhanced & False-Positive Suppressed Vocabulary) is constructed. DS-FPSV includes a dynamic semantic enhancement vocabulary and a false-positive suppression vocabulary. The dynamic semantic enhancement vocabulary is called DS, and the false-positive suppression vocabulary is called FPSV. As a dynamic database, DS-FPSV supports online updates (with a construction complexity of O(N logN)) and seamlessly integrates with the object detection model.

[0035] In the process of constructing a dynamic semantic enhancement vocabulary, multiple sample texts can be obtained. For each sample text, the sample text can include descriptive information for an entity object. An entity object can be something that exists in the real world, has independent existence and characteristics, and can be observed, perceived, and measured, such as a car, a computer, a person, etc., and the descriptive information of the entity object is information that describes the entity object.

[0036] The multiple sample texts can be clustered to obtain multiple clustering clusters. For example, for each sample text, a text feature of the sample text can be extracted. The multiple sample texts can be clustered based on the text features of the different sample texts to obtain multiple clustering clusters. For each clustering cluster, the clustering cluster includes at least one sample text, and the similarity between the text features of the sample texts in the clustering cluster is greater than a threshold. For example, the multiple sample texts can be clustered by using a k-means algorithm (i.e., a k-means clustering algorithm) or an HDBSCAN algorithm (i.e., a density clustering algorithm), and the clustering algorithm is not limited in this embodiment.

[0037] When the multiple clustering clusters are obtained, for each clustering cluster, the clustering cluster can include at least one sample text, and all the sample texts in the clustering cluster belong to the same entity object, i.e., the sample texts are description information of the same entity object. For example, sample text 1 is description information of the entity object "human", sample text 2 is description information of the entity object "people", and sample text 3 is description information of the entity object "person". Obviously, these sample texts are all description information of the same entity object "person". When the sample texts are clustered, the sample texts can be clustered together to obtain a clustering cluster, i.e., all the sample texts in the clustering cluster belong to the same entity object "person".

[0038] A dynamic semantic enhanced vocabulary is constructed, and the dynamic semantic enhanced vocabulary includes multiple first vocabulary items corresponding to the multiple clustering clusters. For example, the dynamic semantic enhanced vocabulary includes first vocabulary item 1 corresponding to clustering cluster 1, first vocabulary item 2 corresponding to clustering cluster 2, and so on. For each first vocabulary item, the first vocabulary item can include an entity corresponding to each sample text in the clustering cluster (i.e., the clustering cluster corresponding to the first vocabulary item). Here, the entity can be information of an entity object, such as an object name, an object serial number, and the like.

[0039] For example, first vocabulary item 1 can include entities such as "human", "person", "people", "human being", "individual person", "human being", and "person". The entity "human" is obtained by analyzing sample text 1 (description information of the entity object "human") in clustering cluster 1, the entity "people" is obtained by analyzing sample text 2 in clustering cluster 1, and the entity "person" is obtained by analyzing sample text 3 in clustering cluster 1. Obviously, the first vocabulary item 1 includes multiple entities of the same entity object "person", and the entities correspond to the same entity object.

[0040] For example, the dynamic semantic enhancement vocabulary can also be referred to as a synonym clustering library (Φ syn) or a semantic equivalent set, and can be formed by clustering text features (such as using a k-means algorithm or an HDBSCAN algorithm). The dynamic semantic enhancement vocabulary can collect and merge semantic expressions with the same meaning across different data sets (such as COCO, Visual Genome, or open-image data sets, which are used to provide sample text), and each clustering cluster can represent a completely unified semantic concept.

[0041] For example, after forming the dynamic semantic enhancement vocabulary by clustering text features, the dynamic semantic enhancement vocabulary can also be manually checked and reviewed to obtain a final output dynamic semantic enhancement vocabulary. Alternatively, the dynamic semantic enhancement vocabulary can be directly provided by a human, and no limitation is made in this regard.

[0042] For the construction process of the false report suppression vocabulary, a plurality of sample images can be obtained. For each sample image, the sample image can include image information of an entity object, such as image information of a "car".

[0043] For each sample image, the sample image can be identified to obtain a predicted entity of an entity object in the sample image. The predicted entity is a predicted identification of the entity object in the sample image, such as a predicted object name. For example, the entity object in the sample image is a cigarette, and the true entity is "cigarette". When the sample image is identified, the predicted entity of the entity object can be "toothpick", or "straw", or "lollipop", or "pen".

[0044] For each entity object in all sample images, a true entity of the entity object and a predicted entity of the entity object can be obtained. On this basis, a false report suppression vocabulary can be constructed. The false report suppression vocabulary can include a plurality of second vocabulary items corresponding to a plurality of entity objects, such as a second vocabulary item 1 corresponding to an entity object 1, a second vocabulary item 2 corresponding to an entity object 2, and so on. For each second vocabulary item, the second vocabulary item can include a corresponding relationship between a true entity of an entity object (i.e., the entity object corresponding to the second vocabulary item) and a predicted entity of the entity object.

[0045] For example, the second vocabulary item 1 can include a corresponding relationship between the true entity "cigarette" of the entity object and the predicted entity "toothpick, straw, lollipop, pen" of the entity object. Obviously, the predicted entity is an entity object that produces a false report for the true entity "cigarette".

[0046] For example, the false positive suppression vocabulary can also be referred to as a confusing false positive suppression set (Ψ_fp), and a confusing object set can be identified through visual feature clustering (such as ViT or CLIP-based visual embedding). For example, for the entity "cigarette", its confusing set is Ψ_fp = {toothpick, straw, lollipop, pen}, indicating that these objects have high similarity in the visual feature space (such as RGB histogram or Edge feature). The false positive suppression vocabulary can collect and merge semantic expressions that produce false positives across datasets (such as COCO, Visual Genome, or open-image datasets, which are used to provide sample images).

[0047] For example, after forming the false positive suppression vocabulary through visual feature clustering, the false positive suppression vocabulary can also be manually checked and reviewed to obtain the final output false positive suppression vocabulary. Alternatively, the false positive suppression vocabulary can be directly provided by humans, and the acquisition method of the false positive suppression vocabulary is not limited.

[0048] In summary, based on the training data (such as sample text and sample images) of the open target detection model, the DS-FPSV can be constructed in advance, the text and visual features of the entity corpus are clustered, the synonymous semantic enhancement relationship and the confusing false positive suppression relationship are constructed, and the open target detection model is realized based on: Synonymous semantic input supplement mechanism. Establish a synonym clustering library The semantic equivalent set of the input entity prompt word can be automatically expanded, which can be realized through a dynamic semantic enhancement vocabulary. For example, when "human" is input, it is dynamically expanded to "human, person, people, human being, individual", and feature alignment is performed in a unified semantic space with the help of the model. Feature alignment is obtained through similarity retrieval of semantic representation.

[0049] False positive entity registration exclusion mechanism. Construct a confusing entity adversarial set This can be achieved through a false positive suppression vocabulary. For example, when the target entity "cigarette" is input, the negative sample suppression of its confusing set "toothpick, straw, lollipop, pen" is activated synchronously, and the negative sample is suppressed through feature decoupling and false positive response mask.

[0050] In the above application scenarios, an embodiment of the present application proposes a target detection method, which can be applied to electronic devices such as personal computers, notebook computers, Internet of Things devices, cameras, management devices, cloud devices, servers, etc. The type of electronic device is not limited and can be any device. Referring to Figure 2 The flowchart of the target detection method is shown in FIG. 1, which can include: Step 201, obtaining prompt text and an image to be detected.

[0051] For example, the prompt text can also be referred to as a demand text, such as detecting a person wearing a red dress in an image to be detected, and the prompt text is injected according to actual demand, which is not limited.

[0052] In step 202, the target entity and the target attribute are determined based on the prompt text, the target entity is the information of the entity object in the prompt text, such as "person", and the target attribute is the information of the attribute object in the prompt text, such as "wearing a red dress", or "red", or "red dress", etc.

[0053] For example, the demand analysis module can implement the demand analysis function, and the demand analysis module can be a demand analysis model. The demand analysis model can be a lightweight language model (such as a distilled version of the BERT model or the T5-Small model, etc.), or other models, which are not limited. The input data of the demand analysis model is the prompt text, and the output data of the demand analysis model is the target entity and the target attribute.

[0054] For example, the demand analysis model receives the input of the natural language demand, that is, the prompt text, such as "detecting a black truck in an image". The demand analysis model decouples the entity and the attribute in the prompt text through semantic dependency analysis and named entity recognition technology, such as the entity being "truck" and the attribute being "red". The entity is the information of the entity object in the prompt text, which is referred to as the target entity in the subsequent, and the attribute is the information of the attribute object in the prompt text, which is referred to as the target attribute in the subsequent. Through the demand analysis model, the object ontology and the modification attribute in the semantic can be explicitly separated, the object ontology is the target entity, the modification attribute is the target attribute, and the structured mapping of the semantic is realized.

[0055] In step 203, the equivalent entity corresponding to the target entity and the false positive entity are determined. For example, the equivalent entity and the target entity belong to the same entity object, that is, different description information for the same entity object, and the false positive entity is the entity object that produces false positives for the target entity.

[0056] For example, the similarity between the word feature vector of the target entity and the word feature vector of each entity in the dynamic semantic enhancement word table can be determined. For example, the word feature vector of the target entity is determined, and the word feature vector of each entity in the dynamic semantic enhancement word table is determined. In this way, the similarity (such as cosine similarity, etc.) between the word feature vector of the target entity and the word feature vector of each entity in the dynamic semantic enhancement word table can be calculated.

[0057] Then, the k entities with the largest similarity are selected from the dynamic semantic enhancement word table, that is, the top-k most similar entries are found through the dynamic semantic enhancement word table. Alternatively, entities with a similarity greater than a preset threshold (such as 0.8) are selected from the dynamic semantic enhancement word table, and the selection method of the entities is not limited.

[0058] Based on the selected entities, such as k entities, a first vocabulary item to which the k entities belong is determined, and a first vocabulary item with the most entities is taken as the first vocabulary item corresponding to the target entity. For example, 5 entities are selected in the first vocabulary item 1, and 2 entities are selected in the first vocabulary item 2, and the first vocabulary item 1 is taken as the first vocabulary item corresponding to the target entity.

[0059] As described above, the first vocabulary item corresponding to the target entity can be queried from the dynamic semantic enhanced vocabulary, and on this basis, the entities in the first vocabulary item can be determined as equivalent entities, that is, the entities in the first vocabulary item and the target entity belong to the same entity object. Obviously, the similarity between the entities in the first vocabulary item and the target entity is large, such as greater than a third threshold.

[0060] In this way, the semantic equivalent set can be output, and each entity in the semantic equivalent set is the equivalent entity corresponding to the target entity, and the semantic equivalent set is queried from the dynamic semantic enhanced vocabulary. For example, the target entity is "person", and the equivalent entity corresponding to the target entity can include but is not limited to: human, person, people, human being, individual.

[0061] For example, the similarity between the target entity and each real entity in the false alarm suppression vocabulary can be determined. For example, the word feature vector of the target entity is determined, and the word feature vector of each real entity in the false alarm suppression vocabulary is determined, so that the similarity (such as cosine similarity) between the word feature vector of the target entity and the word feature vector of each real entity in the false alarm suppression vocabulary can be calculated.

[0062] Then, the real entity with the largest similarity is selected from the false alarm suppression vocabulary, and if the largest similarity is greater than a fourth threshold, it means that the real entity is the same as the target entity, that is, the same entity, and the second vocabulary item to which the real entity belongs can be taken as the second vocabulary item corresponding to the target entity. If the largest similarity is not greater than the fourth threshold, it means that there is no second vocabulary item corresponding to the target entity in the false alarm suppression vocabulary, and at this time there is no false alarm entity.

[0063] Taking the case that the largest similarity is greater than the fourth threshold, the second vocabulary item corresponding to the target entity can be queried from the false alarm suppression vocabulary, and on this basis, the predicted entity in the second vocabulary item can be determined as the false alarm entity, that is, the predicted entity in the second vocabulary item is the entity object that produces false alarms for the target entity. Obviously, the similarity between the real entity in the second vocabulary item and the target entity can be greater than the fourth threshold, indicating that the real entity is the same as the target entity.

[0064] In this way, the false alarm set can be output, and the false alarm set Each entity in the confusion set (may also be referred to as the easily confused set) is a false positive entity corresponding to the target entity, and the false positive set is obtained by querying the false positive suppression vocabulary. For example, the target entity is "person", and the false positive entity corresponding to the target entity can include but is not limited to: tree stump, model, robot, doll, sculpture, monkey.

[0065] Step 204, obtaining the first detection box corresponding to the target entity, the second detection box corresponding to the equivalent entity, and the third detection box corresponding to the false positive entity from the image to be detected.

[0066] For example, the entity recognition module can implement the entity recognition function, and the entity recognition module can be a target detection model (i.e. entity detection model). The target detection model can be any model for implementing target detection, such as GroundingDINO model or MQ-Det model, etc. The input data of the target detection model is the image to be detected, the target entity, the equivalent entity and the false positive entity, and the output data of the target detection model is the first detection box corresponding to the target entity, the second detection box corresponding to the equivalent entity and the third detection box corresponding to the false positive entity.

[0067] For example, the image to be detected and the target entity can be input to the target detection model, and the target detection model can detect the detection box corresponding to the target entity (denoted as the first detection box) in the image to be detected, and output the first detection box (such as detection box coordinates) corresponding to the target entity, the class label of the first detection box and the confidence of the first detection box. The confidence represents the probability of the existence of the entity object.

[0068] For example, the image to be detected and each equivalent entity (hereinafter an equivalent entity will be taken as an example for description) can be input to the target detection model, and the target detection model can detect the detection box corresponding to the equivalent entity (denoted as the second detection box) in the image to be detected, and output the second detection box (such as detection box coordinates) corresponding to the equivalent entity, the class label of the second detection box and the confidence.

[0069] For example, the image to be detected and each false positive entity (hereinafter a false positive entity will be taken as an example for description) can be input to the target detection model, and the target detection model can detect the detection box corresponding to the false positive entity (denoted as the third detection box) in the image to be detected, and output the third detection box (such as detection box coordinates) corresponding to the false positive entity, the class label of the third detection box and the confidence.

[0070] In one possible implementation, obtaining the first detection box corresponding to the target entity from the image to be detected (such as the target detection model detecting the first detection box in the image to be detected) can include: The multiple candidate boxes are divided from the to-be-detected image, and the division manner of the candidate boxes is not limited. For any candidate box in the to-be-detected image, the second image feature (i.e., the image feature of the candidate box region in the to-be-detected image) is obtained by performing feature extraction on the candidate box. The second text feature (such as the word feature of the target entity) is obtained by performing feature extraction on the target entity. The second cross-modal similarity is determined based on the second image feature and the second text feature (since it is the similarity of the image feature and the text feature, it is called the cross-modal similarity). How to determine the second cross-modal similarity is not limited in this embodiment.

[0071] The second similarity threshold corresponding to the target entity is determined. The second similarity threshold can be a preconfigured fixed similarity threshold, or can be an adaptively dynamically adjusted similarity threshold. How to adaptively dynamically adjust the similarity threshold can be referred to subsequent embodiments.

[0072] On this basis, if the second cross-modal similarity is greater than the second similarity threshold corresponding to the target entity, the candidate box is determined as the first detection box corresponding to the target entity; if the second cross-modal similarity is not greater than the second similarity threshold, the candidate box is prohibited to be determined as the first detection box corresponding to the target entity. Apparently, after the above processing is performed on all candidate boxes in the to-be-detected image, the first detection box corresponding to the target entity can be obtained, and the first detection box can include at least one candidate box.

[0073] In a possible implementation, for each equivalent entity, the second detection box corresponding to the equivalent entity is obtained from the to-be-detected image (such as the target detection model detecting the second detection box), including: The multiple candidate boxes are divided from the to-be-detected image, and for any candidate box in the to-be-detected image, the second image feature (i.e., the image feature of the candidate box region in the to-be-detected image) is obtained by performing feature extraction on the candidate box. The second text feature (such as the word feature of the equivalent entity) is obtained by performing feature extraction on the equivalent entity. The second cross-modal similarity is determined based on the second image feature and the second text feature.

[0074] The second similarity threshold corresponding to the equivalent entity is determined. The second similarity threshold can be a preconfigured fixed similarity threshold, or can be an adaptively dynamically adjusted similarity threshold. When the similarity threshold is adaptively dynamically adjusted, the second similarity threshold corresponding to the equivalent entity and the second similarity threshold corresponding to the target entity are independently determined, that is, the second similarity threshold is determined for each entity, and therefore the second similarity thresholds of different entities are the same or different.

[0075] On this basis, if the second cross-modal similarity is greater than or equal to the second similarity threshold corresponding to the equivalent entity, the candidate box is determined as the second detection box corresponding to the equivalent entity; if the second cross-modal similarity is not greater than the second similarity threshold, the candidate box is prohibited to be determined as the second detection box corresponding to the equivalent entity. Obviously, after the above processing is performed on all candidate boxes in the to-be-detected image, the second detection box corresponding to the equivalent entity can be obtained, and the second detection box can include at least one candidate box.

[0076] In a possible implementation, for each false positive entity, a third detection box corresponding to the false positive entity is obtained from the to-be-detected image (for example, the target detection model detects the third detection box), including: A plurality of candidate boxes are divided from the to-be-detected image, for any candidate box in the to-be-detected image, a second image feature (that is, an image feature of a candidate box region in the to-be-detected image) is obtained by performing feature extraction on the candidate box. A second text feature (for example, a word feature of the false positive entity) is obtained by performing feature extraction on the false positive entity. A second cross-modal similarity is determined based on the second image feature and the second text feature. A second similarity threshold corresponding to the false positive entity is determined, which can be a preconfigured fixed similarity threshold, or a self-adaptive dynamically adjusted similarity threshold.

[0077] On this basis, if the second cross-modal similarity is greater than or equal to the second similarity threshold corresponding to the false positive entity, the candidate box is determined as the third detection box corresponding to the false positive entity; if the second cross-modal similarity is not greater than the second similarity threshold, the candidate box is prohibited to be determined as the third detection box corresponding to the false positive entity. Obviously, after the above processing is performed on all candidate boxes in the to-be-detected image, the third detection box corresponding to the false positive entity can be obtained, and the third detection box can include at least one candidate box.

[0078] Step 205, determining a target detection box based on an overlapping degree between the first detection box, the second detection box, and the third detection box.

[0079] For example, a candidate detection box is determined based on an overlapping degree between the first detection box and the second detection box; the candidate detection box includes the first detection box, or the candidate detection box includes the second detection box, or the candidate detection box includes the first detection box and the second detection box; a target detection box is determined based on an overlapping degree between the candidate detection box and the third detection box; the target detection box includes the candidate detection box filtered based on the overlapping degree. For example, the target detection box corresponding to the target entity can be determined by the following steps: Step S11, for each second detection box (i.e. multiple equivalent entities correspond to multiple second detection boxes), determine a first overlap degree between the first detection box and the second detection box, and the first overlap degree can be the IOU (Intersection over Union) between the two detection boxes.

[0080] Step S12, select a candidate detection box from the first detection box and all second detection boxes.

[0081] For example, for each second detection box, if the first overlap degree corresponding to the second detection box is less than a first threshold, i.e. the overlap degree between the second detection box and the first detection box is low, the first detection box and the second detection box are determined as the candidate detection box, and at this time, the two detection boxes are retained.

[0082] If the first overlap degree corresponding to the second detection box is not less than the first threshold, i.e. the overlap degree between the second detection box and the first detection box is high, the first detection box or the second detection box is determined as the candidate detection box, and at this time, only one detection box is retained. For example, if the confidence of the first detection box is not less than the confidence of the second detection box, the first detection box is determined as the candidate detection box. If the confidence of the first detection box is less than the confidence of the second detection box, the second detection box is determined as the candidate detection box.

[0083] In summary, the candidate detection box can be determined based on the first detection box and all second detection boxes, and the candidate detection box can be at least one. For example, the target detection model can output the detection results of multiple synonyms, i.e. the first detection box and multiple second detection boxes, and the first detection box and multiple second detection boxes can be merged through greedy NMS (Greedy NMS), and a single candidate detection box can be output, so as to eliminate redundant detection and ensure that each entity is output only once, or multiple candidate detection boxes can be output.

[0084] Step S13, for each third detection box (i.e. multiple false positive entities correspond to multiple third detection boxes), determine a second overlap degree between the candidate detection box (which can be at least one candidate detection box) and the third detection box, and the second overlap degree can be the IOU between the two detection boxes.

[0085] Step S14, for each third detection box, perform a weighted operation on the second overlap degree corresponding to the third detection box and the confidence of the third detection box to obtain a false positive score of the candidate detection box and the third detection box.

[0086] For example, the false positive score of the candidate detection box and the third detection box can be determined by using the following formula: . represents the false positive score of the candidate detection box and the third detection box, represents the confidence of the third detection box, that is, the confidence of the third detection box output by the target detection model, Indicates the second overlap degree between the candidate detection frame and the third detection frame. and It is a weighting coefficient. The weighting coefficient can be configured according to actual needs, and the final elimination standard can be obtained through reasonable coefficient matching.

[0087] Step S15: Select a target detection frame corresponding to the target entity from all candidate detection frames.

[0088] Exemplarily, for each candidate detection frame, if the false alarm score of the candidate detection frame and the third detection frame is not greater than a second threshold (i.e., a false alarm threshold, such as 0.3), the candidate detection frame is determined to be a target detection frame; if the false alarm score of the candidate detection frame and the third detection frame is greater than the second threshold, the candidate detection frame is prohibited from being determined as a target detection frame. For example, if the false alarm scores of the candidate detection frame and all third detection frames are not greater than the second threshold, the candidate detection frame is determined to be a target detection frame; if the false alarm score of the candidate detection frame and any third detection frame is greater than the second threshold, it indicates that the candidate detection frame is a false alarm detection frame, and it can be prohibited to determine the candidate detection frame as a target detection frame.

[0089] In summary, we can select a target detection frame from all candidate detection frames, with at least one target detection frame. For example, we can filter all candidate detection frames using the third detection frame corresponding to the false positive entity to filter out false positive targets, significantly reducing the false positive rate and outputting a refined detection frame and confidence score.

[0090] As can be seen from the above, the second and third detection boxes are used to optimize the first detection box, ultimately outputting the target detection box along with its corresponding ID and category label. The ID indicates the target detection box's rank among all detection boxes, and the category label can be the target entity. During the optimization of the first detection box, dynamic synonym expansion and unified semantic space alignment (using text feature similarity as a link) are used to generate consistent detections for different expressions (such as "human" and "人"), reducing detection error. Detection error is further reduced through the suppression of easily confused negative samples and feature decoupling.

[0091] In a possible implementation, if the first detection box corresponding to the target entity, the second detection box corresponding to the equivalent entity, and the third detection box corresponding to the false positive entity can be obtained from the image to be detected, the target detection box is determined based on the overlapping degree between the first detection box, the second detection box, and the third detection box. If only the first detection box corresponding to the target entity and the second detection box corresponding to the equivalent entity can be obtained from the image to be detected, the target detection box is determined based on the overlapping degree between the first detection box and the second detection box, for example, a candidate detection box is determined based on the first detection box and the second detection box, and the candidate detection box is taken as the target detection box. If only the first detection box corresponding to the target entity and the third detection box corresponding to the false positive entity can be obtained from the image to be detected, the target detection box is determined based on the overlapping degree between the first detection box and the third detection box, for example, the first detection box is taken as a candidate detection box, and the candidate detection box and the third detection box are taken as the target detection box. If only the second detection box corresponding to the equivalent entity and the third detection box corresponding to the false positive entity can be obtained from the image to be detected, the target detection box is determined based on the overlapping degree between the second detection box and the third detection box, for example, the second detection box is taken as a candidate detection box, and the candidate detection box and the third detection box are taken as the target detection box. If only the first detection box corresponding to the target entity can be obtained from the image to be detected, the first detection box is taken as the target detection box. If only the second detection box corresponding to the equivalent entity can be obtained from the image to be detected, the second detection box is taken as the target detection box. If only the third detection box corresponding to the false positive entity can be obtained from the image to be detected, the target detection box is not output.

[0092] Step 206: A detection sub-image of the target detection box is intercepted from the image to be detected.

[0093] For example, after the target detection box is obtained, a sub-image of the target detection box can be intercepted (cropped) from the image to be detected based on the coordinates of the target detection box, and this sub-image is referred to as a detection sub-image.

[0094] Step 207: Target attribute information of the target detection box is determined based on the detection sub-image and the target attribute, and the target attribute information can represent the attribute of the target detection box. For example, the target attribute can be taken as the target attribute information of the target detection box, or the target attribute can be optimized, and the optimized target attribute is taken as the target attribute information of the target detection box, which is not limited in this regard.

[0095] Exemplarily, the attribute recognition module can implement the attribute recognition function, and the attribute recognition module can be an attribute recognition model. The attribute recognition model is any model used to implement attribute recognition, such as a cross-modal contrast learning model (such as a CLIP-ViT model or an ALIGN model, etc.), and no limitation is made to the attribute recognition model. The input data of the attribute recognition model can be a detection sub-image (that is, an image region corresponding to a target detection box is intercepted) and a target attribute (the target attribute can be determined based on a prompt word text, such as “wearing a red dress”). The output data of the attribute recognition model can be target attribute information of the target detection box.

[0096] For example, the detection sub-image and the target attribute can be input to the attribute recognition model. The attribute recognition model performs cross-modal feature alignment on the detection sub-image and the target attribute, determines the attribute existence confidence, determines the target attribute information of the target detection box, and outputs the target attribute information. In this way, the fine-grained visual and language association capability can be implemented, and the attribute generalization recognition independent of the entity category can be implemented.

[0097] In a possible implementation, determining the target attribute information of the target detection box based on the detection sub-image and the target attribute (such as the attribute recognition model determining the target attribute information) can include: performing feature extraction on the detection sub-image to obtain first image features (such as image features corresponding to the detection sub-image). Performing feature extraction on the target attribute to obtain first text features (such as word features corresponding to the target attribute). Determining a first cross-modal similarity based on the first image features and the first text features (since it is to determine the similarity of image features and text features, it can be called a cross-modal similarity). How to determine the first cross-modal similarity is not limited in this embodiment.

[0098] A first similarity threshold corresponding to the target attribute is determined. The first similarity threshold can be a pre-configured fixed similarity threshold, or a self-adaptive dynamically adjusted similarity threshold. The self-adaptive dynamically adjusted similarity threshold is described in subsequent embodiments. When the similarity threshold is self-adaptively dynamically adjusted, the first similarity threshold of each attribute is determined independently, that is, the first similarity threshold is determined for each attribute. Therefore, the first similarity thresholds of different attributes are the same or different.

[0099] On this basis, if the first cross-modal similarity is greater than the first similarity threshold, it indicates that the detection sub-image matches the target attribute, and the target attribute is determined as the target attribute information corresponding to the target detection box. If the first cross-modal similarity is not greater than the first similarity threshold, it indicates that the detection sub-image does not match the target attribute, and the target attribute is not determined as the target attribute information corresponding to the target detection box. In this case, the attribute recognition model can generate the target attribute information corresponding to the target detection box.

[0100] Step 208, output the target attribute information and the target detection frame, that is, the target attribute information is the labeling information for the target detection frame, thus completing the target recognition process.

[0101] In this embodiment, the dynamic semantic enhancement and false positive suppression vocabulary (DS-FPSV) is used to solve the semantic sensitivity and visual false positive problem. An entity and attribute decoupling architecture is adopted, which adopts a three-level connection topology structure of a demand analysis module, an entity recognition module and an attribute recognition module, and realizes high-precision separation and recognition of target entities and attributes through the demand analysis module, the entity recognition module and the attribute recognition module. These modules perform end-to-end recognition in the following order: natural language demand input→demand analysis module (decoupling entity / attribute)→entity recognition module (detecting entities and outputting target detection frames)→attribute recognition module (cropping images based on target frames and verifying attributes)→outputting target instances with attribute labels (i.e., target detection frames).

[0102] In a possible implementation, referring to Figure 3 , a schematic diagram of a target detection framework is shown.

[0103] In the target detection process, a prompt word text (i.e., a demand text) can be obtained. The prompt word text is input into a demand analysis model (such as a demand analysis model based on a language model), and the demand analysis model outputs a target entity and a target attribute. A DS-FPSV cooperation process is performed on the target entity to obtain a semantic equivalence set and an easily confused false positive set. The semantic equivalence set can include equivalent entities corresponding to the target entity, and the easily confused false positive set can include false positive entities corresponding to the target entity.

[0104] A to-be-detected image can be obtained, and the to-be-detected image is input into an entity recognition model (such as a target detection model). The target entity, the semantic equivalence set and the easily confused false positive set are input into the entity recognition model, and the entity recognition model outputs a target instance, which includes a target detection frame, an ID and a category label.

[0105] The to-be-detected image is cropped based on the target detection frame to obtain a detection sub-image, and the detection sub-image is input into an attribute recognition model (such as an attribute recognition model based on a cross-modal contrast learning model). The target attribute is input into the attribute recognition model, and the attribute recognition model outputs target attribute information of the target detection frame. In combination with the target attribute information and the target instance, a target instance with target attribute information can be obtained.

[0106] Continuing to refer to Figure 3As shown, the detection sub-image with target attribute information can be used as a sample image, and attribute automatic threshold adaptive adjustment can be realized based on the sample image. The first similarity threshold can be updated through the attribute automatic threshold adaptive adjustment process, and the first similarity threshold updating process is described in the subsequent embodiments. In addition, entity automatic threshold adaptive adjustment can be realized based on the sample image. The second similarity threshold can be updated through the entity automatic threshold adaptive adjustment process, and the second similarity threshold updating process is described in the subsequent embodiments.

[0107] In a possible implementation, referring to Figure 4 As shown, a flowchart of target detection is shown. The prompt word text (i.e., the requirement text description) can be obtained. The prompt word text is input into the requirement analysis model, and the target entity and the target attribute are output by the requirement analysis model. The DS-FPSV collaborative process can be performed for the target entity to obtain a semantic equivalence set and a confused false alarm set. The target entity, the semantic equivalence set, and the confused false alarm set are input into the entity recognition model. The target attribute is input into the attribute recognition model.

[0108] The detection sub-image can be obtained by cropping the detection sub-image based on the target detection frame. The detection sub-image is input into the attribute recognition model, and the requirement detection result is output by the attribute recognition model based on the detection sub-image and the target attribute. The requirement detection result can be a target instance with target attribute information.

[0109] The requirement detection result triggers dynamic detection. In the dynamic detection process, a small amount of scene supervision samples (such as detection sub-images with target attribute information, which can be obtained by cropping the detection sub-image based on the target detection frame) are obtained. Attribute automatic threshold adaptive adjustment is realized based on the small amount of scene supervision samples, and the first similarity threshold is automatically updated. The first similarity threshold is input into the attribute recognition model, and the attribute recognition is realized by the attribute recognition model based on the first similarity threshold. Entity automatic threshold adaptive adjustment is realized based on the small amount of scene supervision samples, and the second similarity threshold is automatically updated. The second similarity threshold is input into the entity recognition model, and the entity recognition is realized by the entity recognition model based on the second similarity threshold.

[0110] In a possible implementation, referring to Figure 5 As shown, a flowchart of automatic threshold adjustment is shown. A small amount of labeled samples (i.e., sample images) of an application scenario can be obtained. Input augmentation preprocessing, cross-modal feature collaborative extraction, and threshold decision boundary optimization are performed based on the small amount of labeled samples to obtain an adaptive threshold. The adaptive threshold is the first similarity threshold and / or the second similarity threshold.

[0111] In a possible implementation, for the determination process of the first similarity threshold, when determining the first similarity threshold corresponding to the target attribute, it is first determined whether there is a positive sample image corresponding to the target attribute, the positive sample image being a detection sub-image with target attribute information (i.e., obtained by cropping a to-be-detected image based on a target detection box), and the target attribute information being the target attribute. If yes, the first similarity threshold corresponding to the target attribute can be determined based on the positive sample image corresponding to the target attribute. If no, the configured default similarity threshold (configured according to actual needs) can be determined as the first similarity threshold corresponding to the target attribute. In addition, the minimum number of positive sample images can also be restricted, for example, 5. When the number of positive sample images corresponding to the target attribute reaches 5, the first similarity threshold corresponding to the target attribute is determined based on the positive sample images corresponding to the target attribute. When the number of positive sample images corresponding to the target attribute does not reach 5, the configured default similarity threshold is determined as the first similarity threshold.

[0112] For example, the first similarity threshold corresponding to the target attribute can be determined by the following steps: Step S21: Obtain m positive sample images corresponding to the target attribute, and obtain n negative sample images corresponding to the target attribute, where m and n can be the same or different, for example, 5 positive sample images and 5 negative sample images. Each positive sample image has an entity object corresponding to the target attribute (i.e., the target attribute information of the entity object is the target attribute), and each negative sample image has an entity object not corresponding to the target attribute (i.e., the target attribute information of the entity object is not the target attribute).

[0113] Step S22: Perform augmentation processing on the positive sample images corresponding to the target attribute to obtain a plurality of augmented positive sample images, for example, augmenting 5 positive sample images into 50 positive sample images.

[0114] For example, in the input augmentation preprocessing stage, a small amount of labeled samples of the target application scenario can be obtained, for example, 5 positive sample images, each positive sample image being a detection sub-image with target attribute information, and a single frame image including multiple target instances corresponding to multiple sample images. For each positive sample image, an augmentation strategy can be used to perform augmentation processing on the positive sample image to obtain a plurality of augmented positive sample images.

[0115] For example, an augmentation strategy library can be pre-configured, which can include but is not limited to illumination disturbance, random occlusion, scale transformation, and adversarial noise injection. On this basis, at least one augmentation strategy can be selected from the augmentation strategy library to perform augmentation processing on the positive sample image. When the augmentation strategy is used to perform augmentation processing on the positive sample image, a plurality of positive sample images can be obtained by changing the parameters of the augmentation strategy (such as the parameters of the illumination disturbance).

[0116] For example, when selecting at least one augmentation strategy from the augmentation strategy library, the augmentation strategy can be randomly selected, or the augmentation strategy can be selected in a probabilistic manner, and the selection manner is not limited.

[0117] After the positive sample images are augmented, an augmented sample set can be generated, which can include multiple positive sample images, such as 50 positive sample images, and the like, so as to construct a scene feature distribution estimation space through the positive sample images. For example, the original sample set , the augmentation strategy library , such as illumination disturbance, random occlusion, scale transformation, and adversarial noise injection. The augmentation probability distribution is weighted according to the operation complexity. In this way, the generation process of the composite augmentation can be represented as: For example, is a randomly sampled augmentation operation, so as to ensure that the amount of augmented samples .

[0118] Step S23, the negative sample images corresponding to the target attribute are augmented to obtain multiple augmented negative sample images, such as augmenting 5 negative sample images to 50 negative sample images.

[0119] For example, at least one augmentation strategy can be selected from the augmentation strategy library, and the negative sample image is augmented by using the augmentation strategy to obtain multiple negative sample images.

[0120] Step S24, the target attribute is extracted to obtain a first text feature (the process is described in the above embodiment), the positive sample cross-modal similarity is determined based on the image feature of the positive sample image and the first text feature, and the negative sample cross-modal similarity is determined based on the image feature of the negative sample image and the first text feature.

[0121] For example, in the cross-modal feature cooperative extraction stage, for each positive sample image, the image feature of the positive sample image can be extracted by feature extraction, the positive sample cross-modal similarity is determined based on the image feature of the positive sample image and the first text feature, and each positive sample image corresponds to a positive sample cross-modal similarity. For each negative sample image, the image feature of the negative sample image can be extracted by feature extraction, the negative sample cross-modal similarity is determined based on the image feature of the negative sample image and the first text feature, and each negative sample image corresponds to a negative sample cross-modal similarity.

[0122] In step S25, a current distribution similarity threshold is determined based on the plurality of positive sample cross-modality similarities and the plurality of negative sample cross-modality similarities, the current distribution similarity threshold being used to distinguish the positive sample cross-modality similarities and the negative sample cross-modality similarities, so that the positive sample cross-modality similarities of all the positive sample images are greater than the current distribution similarity threshold, and the negative sample cross-modality similarities of all the negative sample images are less than the current distribution similarity threshold.

[0123] For example, a minimum positive sample cross-modality similarity is selected from the positive sample cross-modality similarities of all the positive sample images, a maximum negative sample cross-modality similarity is selected from the negative sample cross-modality similarities of all the negative sample images, the current distribution similarity threshold is less than the minimum positive sample cross-modality similarity, and the current distribution similarity threshold is greater than the maximum negative sample cross-modality similarity. For example, the current distribution similarity threshold can be an average of the minimum positive sample cross-modality similarity and the maximum negative sample cross-modality similarity. Of course, this is only an example, and it is only required that the positive sample cross-modality similarities of all the positive sample images are greater than the current distribution similarity threshold, and the negative sample cross-modality similarities of all the negative sample images are less than the current distribution similarity threshold.

[0124] For example, in the threshold decision boundary optimization stage, the feature cluster center and the intra-class dispersion of the positive sample images in the projection space can be calculated, and the feature cluster center and the intra-class dispersion of the negative sample images in the projection space can be calculated, the distribution interval between the positive sample images and the negative sample images is maximized through the feature cluster center and the intra-class dispersion, and the optimal threshold is solved, which is the current distribution similarity threshold.

[0125] For example, the feature cluster center and the intra-class dispersion of the positive sample images can be calculated by using the following formula: , . denotes the feature cluster center, denotes the intra-class dispersion, denotes the positive sample cross-modality similarity, denotes the number of positive sample images.

[0126] For example, the feature cluster center and the intra-class dispersion of the negative sample images can be calculated by using the following formula: , . denotes the feature cluster center, denotes the intra-class dispersion, denotes the negative sample cross-modality similarity, denotes the number of negative sample images.

[0127] Obviously, the feature clustering center and the intra-class dispersion of the positive sample image can reflect the distribution feature of the positive sample cross-modal similarity of the positive sample image, the feature clustering center and the intra-class dispersion of the negative sample image can reflect the distribution feature of the negative sample cross-modal similarity of the negative sample image, and based on the distribution feature of the positive sample cross-modal similarity and the distribution feature of the negative sample cross-modal similarity, an optimal threshold can be found, which can distinguish the distribution feature of the positive sample cross-modal similarity and the distribution feature of the negative sample cross-modal similarity, and this optimal threshold is the current distribution similarity threshold.

[0128] In step S26, the current distribution similarity threshold is determined as the first similarity threshold corresponding to the target attribute. Alternatively, it is determined whether there is a historical distribution similarity threshold corresponding to the target attribute. The historical distribution similarity threshold can be the first similarity threshold of the previous period before the current time. If there is no historical distribution similarity threshold, the current distribution similarity threshold is determined as the first similarity threshold corresponding to the target attribute, and this first similarity threshold is used as the historical distribution similarity threshold of the next period.

[0129] Alternatively, if there is a historical distribution similarity threshold, a divergence value can be determined based on the current distribution similarity threshold and the historical distribution similarity threshold. If the divergence value is greater than a fifth threshold (which can be configured according to actual needs, such as 0.2, etc.), the current distribution similarity threshold is determined as the first similarity threshold corresponding to the target attribute, and this first similarity threshold is used as the historical distribution similarity threshold of the next period. If the divergence value is not greater than the fifth threshold, the historical distribution similarity threshold used in the last period can be determined as the first similarity threshold corresponding to the target attribute, that is, the first similarity threshold is kept unchanged.

[0130] For example, the divergence value can be determined by the following formula: ; wherein, represents the current distribution similarity threshold, represents the s-th historical distribution similarity threshold, and the set S represents all historical distribution similarity thresholds, represents the divergence value.

[0131] In summary, when the divergence value is greater than the fifth threshold value, it indicates that the current scene has changed greatly, and the first similarity threshold value can be updated, the current distribution similarity threshold value is determined as the first similarity threshold value, when the divergence value is not greater than the fifth threshold value, it indicates that the current scene has not changed greatly, and the first similarity threshold value can not be updated, the historical distribution similarity threshold value used in the last period is determined as the first similarity threshold value, that is, the first similarity threshold value remains unchanged. In this way, the first similarity threshold value optimized for the current scene can be output, and the deployment stage continuously detects the confidence distribution drift to trigger threshold recalibration, such as automatically updating the first similarity threshold value iteratively when the divergence value is greater than the fifth threshold value, to obtain a threshold value adapted to the scene.

[0132] In a possible implementation, for the determination process of the second similarity threshold value, when determining the second similarity threshold value corresponding to the target entity (the determination method of the second similarity threshold value corresponding to the equivalent entity and the false positive entity is the same, and will not be described again hereinafter), it is first determined whether there is a positive sample image corresponding to the target entity, which is a detection sub-image (i.e., obtained by cropping the to-be-detected image based on the target detection box) of the target entity, that is, the entity object in the positive sample image corresponds to the target entity. If yes, the second similarity threshold value corresponding to the target entity can be determined based on the positive sample image corresponding to the target entity, and if no, the configured default similarity threshold value (configured according to actual needs) is determined as the second similarity threshold value corresponding to the target entity. In addition, the minimum number of positive sample images can also be restricted, such as 5. For example, when the number of positive sample images corresponding to the target entity reaches 5, the second similarity threshold value is determined based on the positive sample images corresponding to the target entity, and when the number of positive sample images corresponding to the target entity does not reach 5, the configured default similarity threshold value is determined as the second similarity threshold value. For example, the second similarity threshold value corresponding to the target entity can be determined by the following steps: Step S31, m positive sample images corresponding to the target entity are obtained, and n negative sample images corresponding to the target entity are obtained. Wherein, the entity object in each positive sample image corresponds to the target entity, that is, the entity object of the target entity in each positive sample image, and the entity object in each negative sample image does not correspond to the target entity, that is, the entity object that is not the target entity in each negative sample image.

[0133] Step S32, the positive sample images corresponding to the target entity are augmented to obtain a plurality of augmented positive sample images, such as augmenting 5 positive sample images to 50 positive sample images.

[0134] Step S33, the negative sample images corresponding to the target entity are augmented to obtain a plurality of augmented negative sample images, such as augmenting 5 negative sample images to 50 negative sample images.

[0135] Step S34, feature extraction is performed on the target entity to obtain a second text feature (the process is described in the above embodiment), a positive sample cross-modal similarity is determined based on the image feature of the positive sample image and the second text feature, and a negative sample cross-modal similarity is determined based on the image feature of the negative sample image and the second text feature.

[0136] Step S35, a current distribution similarity threshold is determined based on the plurality of positive sample cross-modal similarities and the plurality of negative sample cross-modal similarities, and the current distribution similarity threshold is used to distinguish the positive sample cross-modal similarity and the negative sample cross-modal similarity, so that the positive sample cross-modal similarity of all positive sample images is greater than the current distribution similarity threshold, and the negative sample cross-modal similarity of all negative sample images is less than the current distribution similarity threshold.

[0137] For example, the minimum positive sample cross-modal similarity is selected from the positive sample cross-modal similarities of all positive sample images, the maximum negative sample cross-modal similarity is selected from the negative sample cross-modal similarities of all negative sample images, the current distribution similarity threshold is less than the minimum positive sample cross-modal similarity, and the current distribution similarity threshold is greater than the maximum negative sample cross-modal similarity. For example, the current distribution similarity threshold can be the average of the minimum positive sample cross-modal similarity and the maximum negative sample cross-modal similarity.

[0138] Step S36, the current distribution similarity threshold is determined as the second similarity threshold corresponding to the target entity. Alternatively, it is determined whether there is a historical distribution similarity threshold corresponding to the target entity. If not, the current distribution similarity threshold is determined as the second similarity threshold corresponding to the target entity. Alternatively, if there is a historical distribution similarity threshold, a divergence value is determined based on the current distribution similarity threshold and the historical distribution similarity threshold. If the divergence value is greater than a fifth threshold, the current distribution similarity threshold is determined as the second similarity threshold corresponding to the target entity. If the divergence value is not greater than the fifth threshold, the historical distribution similarity threshold used in the last period is determined as the second similarity threshold corresponding to the target entity.

[0139] From the above technical solutions, in the embodiment, the recognition structure decoupled from entities and attributes maintains the fine-grained attribute joint recognition capability in an open environment, improves the detection accuracy and robustness, and the average precision is improved by more than 20%. The DS-FPSV mechanism effectively reduces the synonym prompt word misrecognition rate by 40% and the false positive entity recognition error rate by 35%. The automatic threshold adjustment only needs about 10 labeled samples, and the threshold near-optimal solution is realized through small sample generalization (KL divergence<0.2), and dynamic adaptive efficiency optimization is realized. It ensures the stability of the multi-modal confidence (cross-modal feature drift sensitivity reduction), which can significantly enhance the reliability and generalization ability in a dynamic open environment. Through the composite amplification strategy, the scene feature manifold is constructed, and the linear separability of positive and negative samples in the cross-modal space is used to approximate the global optimal threshold in this scene.

[0140] By designing a three-module cascade topology (requirement analysis module, entity recognition module, and attribute recognition module), high-precision separation recognition of target entities and attributes in an open environment is realized through a structured process. The open constraint of entity and attribute coupling training is broken through, and the problem of unknown class attribute recognition capability degradation is solved. Through the cooperative mechanism of dynamic semantic enhancement and false positive suppression large vocabulary (DS-FPSV), the semantic consistency modeling capability is improved, and the false positive problem of visually similar objects is significantly reduced. Through automatic threshold adaptive adjustment, cross-modal confidence is adaptively optimized based on a small number of samples, online threshold dynamic updating (KL divergence drift triggers threshold updating) is fused, and the mismatch problem of the static threshold mechanism in the dynamic environment is solved.

[0141] Based on the same application concept as the above method, an embodiment of the present application proposes a target detection device, as shown in Figure 6 The device can include: A determination module 61 is configured to determine a target entity based on the obtained prompt word text, determine equivalent entities corresponding to the target entity, and determine false positive entities corresponding to the target entity; wherein the equivalent entities correspond to the same entity object as the target entity, and the false positive entities are entity objects that produce false positives for the target entity. An acquisition module 62 is configured to acquire a first detection box corresponding to the target entity, a second detection box corresponding to the equivalent entity, and a third detection box corresponding to the false positive entity from a to-be-detected image. The determination module 61 is configured to determine a target detection box based on the overlap degree between the first detection box, the second detection box, and the third detection box. A processing module 63 is configured to cut a detection sub-image of the target detection box from the to-be-detected image, determine target attribute information based on the detection sub-image, and output the target attribute information and the target detection box.

[0142] In an example, the determining module 61 is specifically configured to determine the target detection frame based on the overlapping degree between the first detection frame, the second detection frame and the third detection frame, and the determining includes: determining a candidate detection frame based on the overlapping degree between the first detection frame and the second detection frame, wherein the candidate detection frame includes the first detection frame, or the candidate detection frame includes the second detection frame, or the candidate detection frame includes the first detection frame and the second detection frame; determining the target detection frame based on the overlapping degree between the candidate detection frame and the third detection frame, wherein the target detection frame includes the candidate detection frame filtered based on the overlapping degree.

[0143] In an example, the determining module 61 is specifically configured to determine the target detection frame based on the overlapping degree between the first detection frame, the second detection frame and the third detection frame, and the determining includes: determining a first overlapping degree between the first detection frame and the second detection frame; determining the first detection frame and the second detection frame as a candidate detection frame if the first overlapping degree is less than a first threshold; determining the first detection frame as the candidate detection frame if the first overlapping degree is not less than the first threshold and a confidence of the first detection frame is not less than a confidence of the second detection frame; determining the second detection frame as the candidate detection frame if the first overlapping degree is not less than the first threshold and the confidence of the first detection frame is less than the confidence of the second detection frame; determining a second overlapping degree between the candidate detection frame and the third detection frame, and performing a weighted operation on the second overlapping degree and a confidence of the third detection frame to obtain a false positive score; determining the candidate detection frame as the target detection frame if the false positive score is not greater than a second threshold; and prohibiting the candidate detection frame from being determined as the target detection frame if the false positive score is greater than the second threshold.

[0144] In an example, the processing module 63 is further configured to: cluster the obtained multiple sample texts to obtain multiple clustering clusters, each clustering cluster including at least one sample text belonging to a same entity object; construct a dynamic semantic enhanced vocabulary, the dynamic semantic enhanced vocabulary including multiple first vocabulary items corresponding to the multiple clustering clusters; for each first vocabulary item, the first vocabulary item including an entity of each sample text in a clustering cluster; identify the obtained multiple sample images to obtain a predicted entity of an entity object in each sample image; construct a false positive suppression vocabulary, the false positive suppression vocabulary including multiple second vocabulary items corresponding to multiple entity objects; for each second vocabulary item, the second vocabulary item including a corresponding relationship between a real entity of an entity object and a predicted entity of the entity object.

[0145] Illustratively, the determining module 61 determines the equivalent entity corresponding to the target entity and the false positive entity corresponding to the target entity, specifically for: querying the dynamic semantic enhanced vocabulary for the first vocabulary item corresponding to the target entity, determining the entity in the first vocabulary item as the equivalent entity; the similarity between the at least one entity in the first vocabulary item and the target entity is greater than a third threshold value; querying the false positive suppression vocabulary for the second vocabulary item corresponding to the target entity, determining the predicted entity in the second vocabulary item as the false positive entity; the similarity between the real entity in the second vocabulary item and the target entity is greater than a fourth threshold value.

[0146] Illustratively, the processing module 63 determines the target attribute information based on the detection sub-image, specifically for: performing feature extraction on the detection sub-image to obtain first image features; performing feature extraction on the target attribute to obtain first text features, the target attribute being determined based on the prompt word text; determining a first cross-modal similarity based on the first image features and the first text features; determining whether there is a positive sample image corresponding to the target attribute; if yes, determining a first similarity threshold corresponding to the target attribute based on the positive sample image, and if not, determining a default similarity threshold configured as the first similarity threshold corresponding to the target attribute; if the first cross-modal similarity is greater than the first similarity threshold, determining the target attribute as the target attribute information.

[0147] Illustratively, the processing module 63 determines the first similarity threshold corresponding to the target attribute based on the positive sample image, specifically for: performing augmentation processing on the positive sample image corresponding to the target attribute to obtain a plurality of augmented positive sample images; performing augmentation processing on the negative sample image corresponding to the target attribute to obtain a plurality of augmented negative sample images; wherein each entity object in the positive sample image corresponds to the target attribute, and each entity object in the negative sample image does not correspond to the target attribute; determining a positive sample cross-modal similarity based on the image features of the positive sample image and the first text features, and determining a negative sample cross-modal similarity based on the image features of the negative sample image and the first text features; determining the first similarity threshold based on the positive sample cross-modal similarity and the negative sample cross-modal similarity, the first similarity threshold being used to distinguish the positive sample cross-modal similarity and the negative sample cross-modal similarity, so that the positive sample cross-modal similarity corresponding to all positive sample images is greater than the first similarity threshold, and the negative sample cross-modal similarity corresponding to all negative sample images is less than the first similarity threshold.

[0148] For example, when the processing module 63 determines the first similarity threshold based on the positive cross-modal similarity and the negative cross-modal similarity, the processing module 63 specifically determines a current distribution similarity threshold based on the positive cross-modal similarity and the negative cross-modal similarity; if there is no historical distribution similarity threshold, the processing module 63 determines the current distribution similarity threshold as the first similarity threshold, and the first similarity threshold is used as the historical distribution similarity threshold of the next period; if there is a historical distribution similarity threshold, the processing module 63 determines a divergence value based on the current distribution similarity threshold and the historical distribution similarity threshold, if the divergence value is greater than a fifth threshold, the processing module 63 determines the current distribution similarity threshold as the first similarity threshold, if the divergence value is not greater than the fifth threshold, the processing module 63 determines the historical distribution similarity threshold used in the last period as the first similarity threshold.

[0149] The processing module 63 determines the divergence value by using the following formula: , wherein represents the current distribution similarity threshold, represents the s-th historical distribution similarity threshold, and the set S represents all historical distribution similarity thresholds, represents the divergence value.

[0150] For example, when the acquisition module 62 acquires the first detection frame corresponding to the target entity from the to-be-detected image, the acquisition module 62 specifically performs feature extraction on any candidate frame in the to-be-detected image to obtain a second image feature, performs feature extraction on the target entity to obtain a second text feature, and determines a second cross-modal similarity based on the second image feature and the second text feature; determines whether there is a positive sample image corresponding to the target entity; if yes, the acquisition module 62 determines a second similarity threshold corresponding to the target entity based on the positive sample image, if no, the acquisition module 62 determines a default similarity threshold configured as the second similarity threshold corresponding to the target entity; if the second cross-modal similarity is greater than the second similarity threshold, the acquisition module 62 determines the candidate frame as the first detection frame corresponding to the target entity; if the second cross-modal similarity is not greater than the second similarity threshold, the acquisition module 62 prohibits the candidate frame from being determined as the first detection frame corresponding to the target entity.

[0151] Specifically, the acquisition module 62 determines the second similarity threshold corresponding to the target entity based on the positive sample image, and specifically: performs augmentation processing on the positive sample image corresponding to the target entity to obtain a plurality of positive sample images, performs augmentation processing on the negative sample image corresponding to the target entity to obtain a plurality of negative sample images; determines a positive sample cross-modal similarity based on the image feature of the positive sample image and the second text feature, and determines a negative sample cross-modal similarity based on the image feature of the negative sample image and the second text feature; and determines the second similarity threshold corresponding to the target entity based on the positive sample cross-modal similarity and the negative sample cross-modal similarity.

[0152] Based on the same application concept as the above method, an electronic device is provided in the embodiments of the present application, as shown in Figure 7 The electronic device includes a processor 71 and a machine readable storage medium 72, the machine readable storage medium 72 stores machine executable instructions that can be executed by the processor 71; the processor 71 is configured to execute the machine executable instructions to implement the target detection method disclosed in the above examples of the present application.

[0153] Based on the same application concept as the above method, the embodiments of the present application also provide a machine readable storage medium, the machine readable storage medium stores a plurality of computer instructions, and the computer instructions can implement the target detection method disclosed in the above examples of the present application when executed by a processor.

[0154] The machine readable storage medium can be any electronic, magnetic, optical or other physical storage device, and can contain or store information such as executable instructions, data, etc. For example, the machine readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard drive), solid state disk, any type of storage disk (such as optical disk, dvd, etc.), or similar storage medium, or combination thereof.

[0155] Based on the same application concept as the above method, the embodiments of the present application also provide a computer program product, which can include a computer program. The computer program is executed by a processor to implement the target detection method disclosed in the above examples of the present application.

[0156] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0157] The foregoing is merely illustrative of the principles of the application and various modifications can be made by persons skilled in the art. The present application is not intended to be limited to the embodiments shown, but is to be accorded the full scope that resides in the art thereof. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.

Claims

1. A target detection method, characterized in that: The method comprises: Determine a target entity based on the acquired prompt word text, and determine an equivalent entity and a false positive entity corresponding to the target entity; wherein the equivalent entity and the target entity correspond to the same entity object, and the false positive entity is an entity object that generates a false positive for the target entity; Obtaining a first detection frame corresponding to the target entity, a second detection frame corresponding to the equivalent entity, and a third detection frame corresponding to the false alarm entity from the acquired image to be detected; determining a target detection frame based on a degree of overlap among the first detection frame, the second detection frame, and the third detection frame; A detection sub-image of the target detection frame is captured from the image to be detected, target attribute information is determined based on the detection sub-image, and the target attribute information and the target detection frame are output.

2. The method according to claim 1, characterized in that The determining of the target detection frame based on the degree of overlap among the first detection frame, the second detection frame, and the third detection frame includes: Determining a candidate detection frame based on the degree of overlap between the first detection frame and the second detection frame; wherein the candidate detection frame includes the first detection frame, or the candidate detection frame includes the second detection frame, or the candidate detection frame includes the first detection frame and the second detection frame; A target detection frame is determined based on the degree of overlap between the candidate detection frame and the third detection frame; wherein the target detection frame includes filtering the remaining candidate detection frames based on the degree of overlap.

3. The method according to claim 1 or 2, characterized in that The determining of the target detection frame based on the degree of overlap among the first detection frame, the second detection frame, and the third detection frame includes: determining a first degree of overlap between the first detection frame and the second detection frame; If the first degree of overlap is less than a first threshold, the first detection frame and the second detection frame are determined as candidate detection frames; if the first degree of overlap is not less than the first threshold and the confidence of the first detection frame is not less than the confidence of the second detection frame, the first detection frame is determined as a candidate detection frame; if the first degree of overlap is not less than the first threshold and the confidence of the first detection frame is less than the confidence of the second detection frame, the second detection frame is determined as a candidate detection frame; determining a second degree of overlap between the candidate detection frame and the third detection frame, and performing a weighted operation on the second degree of overlap and the confidence of the third detection frame to obtain a false alarm score; If the false alarm score is not greater than the second threshold, the candidate detection frame is determined as the target detection frame; if the false alarm score is greater than the second threshold, the candidate detection frame is prohibited from being determined as the target detection frame.

4. The method according to claim 1, wherein Before determining the equivalent entities and false positive entities corresponding to the target entity, the method further includes: Clustering the acquired multiple sample texts to obtain multiple clusters, each cluster including at least one sample text belonging to the same entity object; constructing a dynamic semantic enhancement vocabulary, the dynamic semantic enhancement vocabulary including multiple first vocabulary items corresponding to the multiple clusters; for each first vocabulary item, the first vocabulary item includes an entity corresponding to each sample text in the cluster; Identifying the acquired multiple sample images to obtain predicted entities of entity objects in each sample image; constructing a false positive suppression vocabulary, the false positive suppression vocabulary including multiple second vocabulary items corresponding to the multiple entity objects; for each second vocabulary item, the second vocabulary item includes a correspondence between a real entity of the entity object and a predicted entity of the entity object; The determining of equivalent entities and false positive entities corresponding to the target entity includes: Querying a first vocabulary item corresponding to the target entity from the dynamic semantic enhancement vocabulary, and determining an entity in the first vocabulary item as the equivalent entity; wherein a similarity between at least one entity in the first vocabulary item and the target entity is greater than a third threshold; A second vocabulary item corresponding to the target entity is queried from the false alarm suppression vocabulary, and the predicted entity in the second vocabulary item is determined as the false alarm entity; wherein the similarity between the real entity in the second vocabulary item and the target entity is greater than a fourth threshold.

5. The method according to claim 1, wherein The determining target attribute information based on the detected sub-image includes: Extracting features from the detection sub-image to obtain a first image feature; extracting features from a target attribute to obtain a first text feature, wherein the target attribute is determined based on the prompt word text; determining a first cross-modal similarity based on the first image feature and the first text feature; Determine whether there is a positive sample image corresponding to the target attribute; if so, determine a first similarity threshold corresponding to the target attribute based on the positive sample image; if not, determine a configured default similarity threshold as the first similarity threshold corresponding to the target attribute; If the first cross-modal similarity is greater than the first similarity threshold, the target attribute is determined as the target attribute information corresponding to the target detection box.

6. The method according to claim 5, characterized in that The determining, based on the positive sample image, a first similarity threshold corresponding to the target attribute includes: Performing an amplification process on the positive sample images corresponding to the target attribute to obtain a plurality of amplified positive sample images; performing an amplification process on the negative sample images corresponding to the target attribute to obtain a plurality of amplified negative sample images; wherein the entity object in each positive sample image corresponds to the target attribute, and the entity object in each negative sample image does not correspond to the target attribute; Determining a positive sample cross-modal similarity based on the image feature of the positive sample image and the first text feature, and determining a negative sample cross-modal similarity based on the image feature of the negative sample image and the first text feature; The first similarity threshold is determined based on the positive sample cross-modal similarity and the negative sample cross-modal similarity. The first similarity threshold is used to distinguish the positive sample cross-modal similarity and the negative sample cross-modal similarity, so that the positive sample cross-modal similarity corresponding to all positive sample images is greater than the first similarity threshold, and the negative sample cross-modal similarity corresponding to all negative sample images is less than the first similarity threshold.

7. The method according to claim 6, characterized in that The determining the first similarity threshold based on the positive sample cross-modal similarity and the negative sample cross-modal similarity includes: Determine a current distribution similarity threshold based on the positive sample cross-modal similarity and the negative sample cross-modal similarity; If the historical distribution similarity threshold does not exist, the current distribution similarity threshold is determined as the first similarity threshold, and the first similarity threshold is used as the historical distribution similarity threshold for the next period; If a historical distribution similarity threshold exists, determining a divergence value based on the current distribution similarity threshold and the historical distribution similarity threshold; if the divergence value is greater than a fifth threshold, determining the current distribution similarity threshold as the first similarity threshold; if the divergence value is not greater than the fifth threshold, determining the historical distribution similarity threshold used in the previous cycle as the first similarity threshold; The divergence value is determined using the following formula: ; Indicates the current distribution similarity threshold, represents the sth historical distribution similarity threshold, and the set S represents all historical distribution similarity thresholds. represents the divergence value.

8. The method according to claim 1, characterized in that The obtaining of a first detection frame corresponding to the target entity from the acquired image to be detected includes: For any candidate box in the image to be detected, perform feature extraction on the candidate box to obtain a second image feature, perform feature extraction on the target entity to obtain a second text feature, and determine a second cross-modal similarity based on the second image feature and the second text feature; Determine whether there is a positive sample image corresponding to the target entity; if so, determine a second similarity threshold corresponding to the target entity based on the positive sample image; if not, determine a configured default similarity threshold as the second similarity threshold corresponding to the target entity; If the second cross-modal similarity is greater than the second similarity threshold, determining the candidate box as the first detection box corresponding to the target entity; if the second cross-modal similarity is not greater than the second similarity threshold, prohibiting the candidate box from being determined as the first detection box corresponding to the target entity; Among them, determining the second similarity threshold corresponding to the target entity based on the positive sample image includes: amplifying the positive sample image corresponding to the target entity to obtain multiple positive sample images, and amplifying the negative sample image corresponding to the target entity to obtain multiple negative sample images; determining the positive sample cross-modal similarity based on the image features of the positive sample image and the second text features, and determining the negative sample cross-modal similarity based on the image features of the negative sample image and the second text features; and determining the second similarity threshold based on the positive sample cross-modal similarity and the negative sample cross-modal similarity.

9. A target detection device, characterized in that: The device comprises: a determination module, configured to determine a target entity based on the acquired prompt word text, and determine an equivalent entity and a false positive entity corresponding to the target entity; wherein the equivalent entity and the target entity correspond to the same entity object, and the false positive entity is an entity object that generates a false positive for the target entity; An acquisition module is used to acquire a first detection frame corresponding to the target entity, a second detection frame corresponding to the equivalent entity, and a third detection frame corresponding to the false alarm entity from the image to be detected; a determination module, configured to determine a target detection frame based on a degree of overlap among the first detection frame, the second detection frame, and the third detection frame; The processing module is used to intercept the detection sub-image of the target detection frame from the image to be detected, determine the target attribute information based on the detection sub-image, and output the target attribute information and the target detection frame.

10. An electronic device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions capable of being executed by the processor; The processor is configured to execute machine-executable instructions to implement the method according to any one of claims 1 to 8.

Images