Target identification method, electronic equipment and computer product

Abstract

The invention provides a target recognition method, electronic equipment and a computer product, and is applied to the technical field of computers, and the method comprises the steps: obtaining an original image and language description information for a to-be-recognized target in the original image; target class information is extracted from the language description information through a first model, and all first targets corresponding to the target class information are recognized in the original image through a second model; generating a multi-modal cue word according to the original image, the language description information and each first target, screening out a second target meeting the multi-modal cue word from each first target through a third model, and taking the screened second target as a to-be-recognized target; the ability of understanding complex semantics is enhanced, and the accuracy of target recognition is improved; and the third model can fuse image and text bimodal information to judge whether each target accords with the description information or not, and the targets which do not accord with the description information are eliminated, so that the accuracy of target recognition is further improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and particularly to an object recognition method, an electronic device, and a computer product. Background Art

[0002] Object recognition is a key application in computer vision, aiming to identify and count specific objects in images or videos, such as people, animals, or vehicles. This technology has broad application prospects in computer vision, especially for object recognition and counting in the fields of intelligent industry, agriculture, urban areas, and transportation, etc., which involve complex language descriptions. For example, in the field of traffic monitoring, vehicle flow can be monitored to make better traffic management decisions; in the retail industry, by counting the customer flow in a store, enterprises can understand customer shopping behaviors and improve sales efficiency; in the field of security monitoring, the flow of people in public places can be monitored to enhance security and safety measures.

[0003] In related technologies, object recognition solutions usually only support inputting object description words and / or image examples. For complex language descriptions of objects to be recognized, existing solutions cannot understand and give correct responses. Summary of the Invention

[0004] The present disclosure provides an object recognition method, an electronic device, and a computer product.

[0005] In a first aspect, an embodiment of the present disclosure provides an object recognition method, including:

[0006] Obtaining an original image and language description information for an object to be recognized in the original image;

[0007] Inputting the language description information into a first model, and extracting object class information from the language description information through the first model, where the object class information is information about the class to which the object to be recognized belongs;

[0008] Inputting the original image and the object class information into a second model, and identifying all first objects corresponding to the object class information in the original image through the second model;

[0009] Generating a multimodal prompt word according to the original image, the language description information, and each of the first objects;

[0010] Inputting the multimodal prompt word into a third model, and screening out second objects that meet the multimodal prompt word from each of the first objects through the third model, and determining the second objects as the objects to be recognized.

[0011] In some embodiments, inputting the original image and the target class information into a second model, and identifying all first targets corresponding to the target class information in the original image through the second model includes:

[0012] Inputting the original image and the target class information into at least two second models respectively, identifying targets corresponding to the target class information in the original image through each of the second models, and aggregating the targets identified by each of the second models to obtain a first target set; wherein each of the targets includes a target position and a confidence level;

[0013] Merging each of the targets according to the target positions and confidence levels of the targets in the first target set to obtain a second target set, and determining the targets in the second target set as all first targets corresponding to the target class information.

[0014] In some embodiments, the target position is represented by a bounding box or a fiducial point, and the fiducial point is the center point of the bounding box.

[0015] In some embodiments, the target position is represented by a bounding box, and the merging each of the targets according to the target positions and confidence levels of the targets in the first target set to obtain a second target set includes:

[0016] Intersection over Union (IoU) calculation step: Determining a specified target corresponding to the maximum confidence level from the first target set and first candidate targets other than the specified target, and calculating the IoU between the bounding box of the specified target and the bounding boxes of each of the first candidate targets respectively;

[0017] Target screening step: Determining redundant first candidate targets with the IoU greater than a preset threshold, and deleting the redundant first candidate targets from the first target set to obtain an updated first target set;

[0018] Repeatedly executing the IoU calculation step and the target screening step until all the targets in the first target set are used as the specified target and the IoU calculation step and the target screening step are completed, and determining the currently updated first target set as the second target set.

[0019] In some embodiments, generating a multi-modal prompt according to the original image, the language description information, and each of the first targets includes:

[0020] For each of the first targets, generating a multi-modal prompt according to the original image, the language description information, and the target position of the first target; or,

[0021] Generate multimodal prompt words according to the language description information and the target positions of the first targets.

[0022] In some embodiments, there are at least two third models. The step of inputting the multimodal prompt words into the third models and screening out second targets that meet the multimodal prompt words from the first targets through the third models includes:

[0023] Input the multimodal prompt words into each of the third models respectively;

[0024] For each third model, screen out second candidate targets from the first targets through the third model. The second candidate targets are targets that meet the multimodal prompt words corresponding to the third model;

[0025] Select the second target from the second candidate targets corresponding to each of the third models.

[0026] In some embodiments, after inputting the multimodal prompt words into the third models and screening out second targets that meet the multimodal prompt words from the first targets through the third models, the method further includes:

[0027] Count the number of targets to be recognized.

[0028] In some embodiments, the first model, the second model, and the third model are deployed on a terminal device; or,

[0029] The first model and the second model are deployed on a terminal device, and the third model is deployed in the cloud; or,

[0030] The first model is deployed on a terminal device, and the second model and the third model are deployed in the cloud.

[0031] In a second aspect, an embodiment of the present disclosure further provides an electronic device, including a memory and a processor; the memory stores a computer program executable by the processor, and when the computer program is executed by the processor, the target recognition method described above is implemented.

[0032] In a third aspect, an embodiment of the present disclosure further provides a computer-readable medium, on which a computer program is stored, and when the program is executed, the target recognition method described above is implemented.

[0033] In a fourth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the target recognition method described above is implemented.

[0034] The target recognition method in the embodiments of the present disclosure includes: obtaining an original image and language description information for the target to be recognized in the original image; extracting target class information from the language description information through the first model, and identifying all first targets corresponding to the target class information in the original image through the second model; generating a multi-modal prompt word based on the original image, the language description information, and each first target, and screening out second targets that meet the multi-modal prompt word from each first target through the third model, and taking the screened second targets as the targets to be recognized; in the embodiments of the present disclosure, the first model performs language processing on the language description information to extract the target class in the language description information, enhancing the ability to understand complex semantics and improving the accuracy of target recognition; constructing a multi-modal prompt word and inputting it to the third model, the third model can fuse the dual-modal information of the image and text, determine whether each target conforms to the language description information, and eliminate the targets that do not conform to the description information, further improving the accuracy of target recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In the drawings of the embodiments of the present disclosure:

[0036] Figure 1 is a schematic diagram of the target recognition process provided by the embodiments of the present disclosure;

[0037] Figure 2 is a schematic diagram of the original image and the language description information of the target to be recognized provided by the embodiments of the present disclosure;

[0038] Figure 3 is a schematic diagram of the effect comparison of target recognition based on different language description information provided by the embodiments of the present disclosure;

[0039] Figure 4 is a schematic diagram of extracting target class information from language description information through the first model provided by the embodiments of the present disclosure;

[0040] Figure 5 is a schematic diagram of indicating the target position with marked points in the original image provided by the embodiments of the present disclosure;

[0041] Figure 6 is a schematic diagram of calculating the intersection over union provided by the embodiments of the present disclosure;

[0042] Figure 7 is a schematic diagram of the effect of target recognition only through the second model in a specific example of the present disclosure;

[0043] Figure 8 is a schematic diagram for judging the target recognition result of Figure 7 through the third model;

[0044] Figure 9 is a schematic diagram of the module composition of the electronic device provided by the embodiments of the present disclosure. Detailed implementation manners

[0045] To enable those skilled in the art to better understand the technical solutions of the present disclosure, the embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0046] The present disclosure will be described more fully hereinafter with reference to the accompanying drawings. However, the illustrated embodiments may be embodied in different forms, and the present disclosure should not be construed as limited to the embodiments set forth below. On the contrary, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0047] The accompanying drawings of the embodiments of the present disclosure are used to provide a further understanding of the embodiments of the present disclosure, and constitute a part of the specification. Together with the detailed embodiments, they are used to explain the present disclosure, and do not constitute a limitation to the present disclosure. By describing the detailed embodiments with reference to the accompanying drawings, the above and other features and advantages will become more apparent to those skilled in the art.

[0048] The present disclosure may be described with reference to the plan view and / or sectional view by means of the ideal schematic diagrams of the present disclosure. Therefore, the example illustrations can be modified according to the manufacturing technology and / or tolerances.

[0049] Without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0050] The terms used in the present disclosure are only for describing specific embodiments and are not intended to limit the present disclosure. As used in the present disclosure, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used in the present disclosure, the singular forms "a" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise. As used in the present disclosure, the terms "comprising", "made of", specify the presence of the stated features, wholes, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their groups.

[0051] Unless otherwise defined, all terms (including technical and scientific terms) used in the present disclosure have the same meaning as commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless the present disclosure clearly so defines.

[0052] The present disclosure is not limited to the embodiments shown in the accompanying drawings, but includes modifications to the configurations formed based on the manufacturing process. Therefore, the regions illustrated in the accompanying drawings have schematic properties, and the shapes of the regions shown in the figures illustrate the specific shapes of the regions of the elements, but are not intended to be restrictive.

[0053] In related technologies, the target recognition scheme usually only supports inputting target description words and / or image examples. If it is a complex language description of the target to be recognized, the related technologies cannot understand and give a correct response. That is to say, the target to be recognized can only be "a certain type of object", rather than "an object with defined attributes / relationships". The related technologies lack the understanding of the context of complex language descriptions and cannot use language to more precisely define the requirements of the target to be recognized, resulting in poor flexibility in practical applications. Therefore, there is a significant need for target recognition for complex language descriptions, that is, a target recognition scheme with strong generalization ability is required to achieve accurate target recognition tasks.

[0054] To solve the above problems, embodiments of the present disclosure propose a target recognition method. Figure 1 The following is a schematic diagram of the target recognition process provided by embodiments of the present disclosure, as Figure 1 shown, the target recognition method includes the following steps:

[0055] Step S11, obtain the original image and the language description information of the target to be recognized in the original image.

[0056] The original image and the language description information of the target to be recognized in the original image are input by the user. The original image usually comes from a photo taken by a camera or a key frame extracted from a video.

[0057] In some embodiments, the language description information may include one of the following: simple sentence, compound sentence, keyword, noun phrase including at least two attributives. That is to say, the language description information of the target to be recognized can be a simple sentence, for example, count the number of all red safety helmets worn on the head; it can also be a compound sentence, for example, count the number of all safety helmets worn on the head and ensure that only red safety helmets will be included in the count; it can also be a keyword, for example, safety helmet; it can also be a noun phrase including at least two attributives, for example, red safety helmet worn on the head.

[0058] Figure 2 The following is a schematic diagram of the original image and the language description information of the target to be recognized provided by embodiments of the present disclosure, as Figure 2 shown, the original image is a working image of workers and technicians wearing safety helmets at a construction site, and the language description information of the target to be recognized in the original image is "red safety helmet worn on the head".

[0059] Step S12, input the language description information into the first model, and extract target class information from the language description information through the first model. The target class information is the information of the category to which the target to be recognized belongs.

[0060] In this step, for the language description information input by the user, the first model is used to automatically extract the target class information from it. The target class information refers to the word in the language description information that represents the category to which the target belongs, and it clearly indicates "which type of object" the user expects to count. Exemplarily, for Figure 2 the original image shown, the language description information of the target to be recognized can be one of the following:

[0061] (1) The red safety helmet worn on the head;

[0062] (2) Count the number of all red safety helmets worn on the head;

[0063] (3) Count the number of all safety helmets worn on the head and ensure that only red safety helmets will be included in the count;

[0064] (4) Safety helmet;

[0065] The target class information corresponding to the above language description information is all "safety helmet".

[0066] Step S13: Input the original image and the target class information into the second model, and the second model identifies all the first targets corresponding to the target class information in the original image.

[0067] In this step, the second model identifies the corresponding first targets in the original image according to the target class information.

[0068] Step S14: Generate multimodal prompt words according to the original image, the language description information, and each first target.

[0069] Step S15: Input the multimodal prompt words into the third model. Through the third model, the second targets that meet the multimodal prompt words are screened out from each first target, and the second targets are determined as the targets to be recognized.

[0070] In this step, the third model receives the multimodal prompt words, automatically analyzes whether the current first target meets the language description information input by the user, and determines the second targets that meet the language description information as the targets to be recognized.

[0071] It should be noted that in the case where the language description information is a single keyword, such as "safety helmet", target screening can also be achieved in this step. The third model is used to screen out the targets that do not meet the description information, such as other targets that are not safety helmets recognized.

[0072] Figure 3Schematic diagram for comparing the effects of object recognition based on description information in different languages. The second model selects the GroundingDINO1.6Pro model of the DINO (DETR with Improved deNoising AnchOr Boxes) series. Object recognition is performed in the original image based on the description information "safety helmet" and "yellow safety helmet" respectively. The recognition effects are as follows Figure 3 As shown, when the description information is "safety helmet", the second model recognizes 4 first objects (including 3 yellow safety helmets and 1 red safety helmet), which are marked by red bounding boxes; when the description information is "yellow safety helmet", the second model also recognizes 4 first objects (including 3 yellow safety helmets and 1 red safety helmet), which are marked by red bounding boxes. Obviously, this recognition result is not accurate. It is not difficult to find that the second model can accurately recognize the objects, but it cannot understand the restrictive semantics of the objects in the description information, resulting in inaccurate object recognition. Therefore, in the embodiments of the present disclosure, the third model is further used to screen the recognized first objects to obtain second objects that meet the additional restrictive semantic requirements in the description information as the objects to be recognized. In this way, the accuracy of object recognition can be improved. Generally, the additional restrictive semantic requirements in the description information can be the state of the object, such as "worn on the head", or the color of the object, such as "yellow", etc.

[0073] The object recognition method in the embodiments of the present disclosure includes: obtaining an original image and description information of the object to be recognized in the original image; extracting object class information from the description information through the first model, and recognizing all first objects corresponding to the object class information in the original image through the second model; generating a multimodal prompt word according to the original image, the description information and each first object, and screening out second objects that meet the multimodal prompt word from each first object through the third model, and using the screened second objects as the objects to be recognized; in the embodiments of the present disclosure, the first model performs language processing on the description information to extract the object class in the description information, enhancing the ability to understand complex semantics and improving the accuracy of object recognition; constructing a multimodal prompt word and inputting it to the third model, and the third model can fuse the dual-modal information of the image and the text to judge whether each object conforms to the description information in the language, and eliminate the objects that do not conform to the description information, further improving the accuracy of object recognition.

[0074] In some embodiments, the first model may be a natural language processing model or a large language model.

[0075] Natural language processing models include, but are not limited to, the BERT (Bidirectional Encoder Representations from Transformers) model. Taking the BERT model as an example below, the process of extracting target class information from language description information through the BERT model is described. Suppose the language description information is "a red safety helmet on the head". Input this language description information into the trained BERT model, and the BERT model outputs text in the BIO format, as shown in Table 1:

[0076] Table 1

[0077]

[0078] Among them, B indicates the start of the target class information, I indicates the rest of the target class information, and O indicates the non-target class information part. According to this BIO-format text, the target class information corresponding to the language description information "a red safety helmet on the head" is "safety helmet".

[0079] Large language models include, but are not limited to, the Qwen model and the DeepSeek model. To extract target class information from language description information using a large language model, only need to construct prompt words to guide the model to output the target class in the language description.

[0080] Figure 4 The following is a schematic diagram of extracting target class information from language description information by the first model provided by the embodiments of the present disclosure. As Figure 4 shown, exemplarily, the user inputs the following content into the large language model as the first model: "Target class information refers to the word in the language description information input by the user that refers to the category to which the target belongs. For example, the target class information corresponding to the language description information 'a red safety helmet worn on the head' is'safety helmet'. Please output the target class information of the following language description information: Count the workers wearing safety helmets standing on the high platform". The large language model outputs the following content: "The target class information is: workers".

[0081] In some embodiments, the second model can be a target detection model or a multimodal large model. Exemplarily, the second model includes, but is not limited to: the YOLO model, the DETR model, the DINO model, etc. These models have the ability to perform target recognition based on descriptive words, so they can accurately identify all the targets corresponding to the target class information in the original image.

[0082] In some embodiments, the third model can be a multimodal large model. Exemplarily, the third model includes, but is not limited to: the Qwen-VL model, the InternVL model, the DeepSeek-VL model, etc.

[0083] In some embodiments, the second model is an object detection model and the third model is a multimodal large model, or both the second model and the third model are multimodal large models. It should be noted that in the case where both the second model and the third model are multimodal large models, they can be the same or different multimodal large models.

[0084] In some embodiments, after inputting the multimodal prompt into the third model and screening out the second objects that meet the multimodal prompt from each first object through the third model (i.e., step S15), the object recognition method may further include the following steps:

[0085] Step S16, count the number of objects to be recognized.

[0086] Automatically count the objects that meet the language description information and output the counting result.

[0087] When all the first objects have been analyzed and screened, the finally obtained second objects are the objects to be recognized, that is, the objects that meet the language description information. In this step, only the number of the second objects needs to be counted and output. It should be noted that in some embodiments, the visualization result of the second objects can also be further output, that is, an image including the object positions of the second objects is output, and the object positions can be bounding boxes or annotation points.

[0088] Take Figure 2 as an example. Taking the original image shown in it and the language description information of the objects to be recognized as input, the final output result is 5, indicating that there are 5 "red safety helmets worn on the head" in the original image. Although there are a total of 7 safety helmets in the original image, since the red safety helmet held in the hand and the yellow safety helmet in the background do not meet the language description information "red safety helmets worn on the head", they are not counted in the final count.

[0089] In order to improve the recall rate of the model output as much as possible so that all objects can be correctly detected and recognized, multiple different second models can be used for object recognition, and then the object recognition results are summarized to achieve the effect of comprehensive search and further improve the accuracy of object recognition. Therefore, in some embodiments, the step of inputting the original image and the object class information into the second model and recognizing all the first objects corresponding to the object class information in the original image through the second model (i.e., step S13) includes the following steps:

[0090] Step S131, input the original image and the object class information into at least two second models respectively, recognize the objects corresponding to the object class information in the original image through each second model respectively, and summarize the objects recognized by each second model to obtain a first object set; where each object includes an object position and a confidence level.

[0091] Step S132: Merge each target in the first target set according to the target position and confidence of each target in the first target set to obtain a second target set, and determine the targets in the second target set as all the first targets corresponding to the target class information.

[0092] By "summarizing" the target recognition results of each second model, the missed detection situation caused by a single model can be prevented. Then, by "merging" the "summarized" target recognition results, redundant targets in the target recognition results of multiple second models can be removed, thereby obtaining a comprehensive and accurate target recognition result.

[0093] In some embodiments, the target position can be represented by a bounding box or a landmark point, where the landmark point is the center point of the bounding box.

[0094] The target position can have the following two forms: text form and visual form. The target position in text form can be the coordinates of the bounding box or the landmark point. The coordinates of the bounding box are represented as (x1, y1, x2, y2), where (x1, y1) represents the coordinates of the upper left corner of the bounding box, and (x2, y2) represents the coordinates of the lower right corner of the bounding box; the coordinates of the landmark point are represented as (x, y), and the landmark point is generally located at the center position of the target. The target position in visual form can be marked on the original image in the form of a bounding box, such as Figure 3 the safety helmet shown by the bounding box as shown; the target position in visual form can also be marked on the original image in the form of a landmark point, such as Figure 5 the safety helmet shown by the landmark point as shown.

[0095] In the embodiments of the present disclosure, the non-maximum suppression (NMS) algorithm is used to "merge" the "summarized" target recognition results. Correspondingly, in some embodiments, when the target position is represented by a bounding box, the step of merging each target according to the target position and confidence of each target in the first target set to obtain a second target set (i.e., step S132) includes the following steps:

[0096] Step S1321: Determine the specified target corresponding to the maximum confidence in the first target set and the first candidate targets other than the specified target, and calculate the intersection over union (IoU) between the bounding box of the specified target and the bounding boxes of the first candidate targets respectively according to the bounding box of the specified target and the bounding boxes of the first candidate targets.

[0097] This step is the Intersection over Union (IoU) calculation step. In this step, the targets in the first target set can be sorted from highest to lowest confidence to obtain a target sequence. The target with the highest confidence in the current target sequence is used as the specified target, and the other targets in the target sequence except the specified target are used as the first candidate targets. According to the bounding box of the specified target and the bounding boxes of each first candidate target, the IoU between the bounding box of the specified target and the bounding boxes of each first candidate target is calculated respectively.

[0098] Figure 6 is a schematic diagram for calculating the Intersection over Union (IoU) provided by an embodiment of the present disclosure. As Figure 6 shown, the Intersection over Union (IoU) between the bounding box of the specified target and the bounding box of the first candidate target = the intersection (Area of Overlap) of the bounding box of the specified target and the bounding box of the first candidate target / the union (Area of Union) of the bounding box of the specified target and the bounding box of the first candidate target.

[0099] Step S1322: Determine the redundant first candidate targets with an IoU greater than a preset threshold, and delete the redundant first candidate targets from the first target set to obtain an updated first target set.

[0100] This step is the target screening step. Based on the preset threshold, each calculated IoU is screened to select the IoUs greater than the preset threshold. The first candidate targets corresponding to these screened IoUs are the redundant first candidate targets. After deleting these first candidate targets from the first target set, an updated first target set is obtained, and this round of merging ends.

[0101] Step S1323: Repeat Step S1321 and Step S1322 until all the targets in the first target set have been used as the specified target and Steps S1321 and S1322 have been completed, and determine the current updated first target set as the second target set.

[0102] Iteratively perform multiple rounds of merging operations, that is, repeat Step S1321 and Step S1322. Each round of merging operation is performed on the updated first target set obtained in the previous round. Select the specified target corresponding to this round of merging operation from the current first target set, calculate the IoU, and screen the redundant first candidate targets based on the IoU. When all the targets in the first target set have been used as the specified target to complete the merging operation, stop the iteration, and determine the updated first target set corresponding to the stop iteration as the second target set.

[0103] In some embodiments, the first target includes a target position. Generating multi-modal prompt words according to the original image, the language description information, and each first target (i.e., step S14) includes the following steps: For each first target, generating multi-modal prompt words according to the original image, the language description information, and the target position of each first target; or, generating multi-modal prompt words according to the language description information and the target position of each first target. That is to say, multi-modal prompt words can be generated separately for each first target, or can be generated at one time for all first targets. However, for the scheme of generating multi-modal prompt words for all first targets at one time, which allows the third model to screen all first targets at one time, although it can reduce the computational amount of multi-modal large model analysis, its accuracy is not as good as the scheme of screening for each first target separately.

[0104] It should be noted that the multi-modal prompt words can have one form of expression or multiple forms of expression. Exemplarily, according to different forms of expression of the target position, multi-modal prompt words of the following various forms of expression can be generated:

[0105] Given that the coordinates of the annotation box of a target in the {original image} are (x1, y1, x2, y2), please determine whether the target meets the following language description: {language description information};

[0106] Given that the coordinates of the annotation point of a target in the {original image} are (x, y), please determine whether the target meets the following language description: {language description information};

[0107] Given that a target in the {original image annotated by the annotation box} is marked in the form of an annotation box, please determine whether the target meets the following language description: {language description information};

[0108] Given that a target in the {original image annotated by the annotation point} is marked in the form of an annotation point, please determine whether the target meets the following language description: {language description information}.

[0109] Since the target positions in text form and visual form can be transformed into each other, and the center point of the identification box can also be taken to transform it into an identification point, the embodiments of the present disclosure can perform reasoning using multi-modal prompt words in one or more forms of expression. The selected multi-modal large model (i.e., the third model) can also be one or more. For the scenario of multi-modal prompt words in multiple forms of expression and multi-model reasoning, a voting method can be used to obtain the final target recognition result.

[0110] In some embodiments, there are at least two third models. Inputting the multi-modal prompt words into the third model and screening out the second targets that meet the multi-modal prompt words from each first target through the third model (i.e., step S15) includes the following steps:

[0111] Step S151: Input the multi-modal prompt words into each third model respectively.

[0112] It should be noted that when there are multiple third models, multiple multi-modal prompt words can be generated, and each multi-modal prompt word is input into each corresponding third model. Each third model corresponds to one multi-modal prompt word, that is, the multi-modal prompt words input into each third model are different; or, one multi-modal prompt word can also be generated and input into each third model respectively, that is, the multi-modal prompt words input into each third model are the same.

[0113] Step S152: For each third model, through this third model, screen out the second candidate targets from each first target. The second candidate targets are the targets that conform to the multi-modal prompt word corresponding to the third model.

[0114] Each third model judges whether each first target conforms to the language description information based on the corresponding multi-modal prompt word, and takes the first target that conforms to the language description information as the second candidate target.

[0115] Step S153: Elect the second target from the second candidate targets corresponding to each third model.

[0116] In some embodiments, the first model, the second model, and the third model can all be deployed on the terminal device; or, the first model and the second model are deployed on the terminal device, and the third model is deployed in the cloud; or, the first model is deployed on the terminal device, and the second model and the third model are deployed in the cloud.

[0117] When the resources on the terminal side are sufficient, the first model, the second model, and the third model can all be deployed on the terminal side. If the resources on the terminal side are limited, in order to achieve the best target recognition and counting effect, a terminal-cloud collaboration deployment strategy can also be used to deploy the multi-modal large model (the second model and the third model, or the third model) on the cloud side and access it by calling the API (Application Programming Interface).

[0118] To clearly illustrate the solution of the embodiments of the present disclosure, the following is combined with Figure 7 and Figure 8 , and a specific example is used to compare and illustrate the recognition effects before and after target screening. Figure 7 It is a schematic diagram of the effect of target recognition only through the second model in a specific example of the present disclosure; as Figure 7As shown, directly using the output of the multi-modal large model to obtain the target counting result of the target to be recognized in the original image, without screening and judging the recognized targets. The multi-modal large model recognizes that the number of motor vehicles occupying the emergency lane of the highway in the original image is 4, but obviously this statistical value is not accurate, and the number of motor vehicles occupying the emergency lane of the highway is significantly greater than 4. It can be seen from this that without target screening and directly performing target counting, it is difficult to ensure the accuracy of target recognition and counting.

[0119] Figure 8 For judging the target recognition result through the third model Figure 7 The schematic diagram is as follows. As Figure 8 shown, the constructed prompt is as follows: "Given that a target in the original image marked by a bounding box is marked in the form of a bounding box, please judge whether the target meets the following language description: count the number of motor vehicles occupying the emergency lane of the highway." Input this prompt into the multi-modal large model, and the multi-modal large model gives the following judgment result:

[0120] "Based on the image and the position marked by the bounding box you provided, it can be judged that: this target (the vehicle marked by the red bounding box) does occupy the emergency lane of the highway, so it meets the requirements of the language description 'count the number of motor vehicles occupying the emergency lane of the highway'. This vehicle is on the rightmost emergency lane and is significantly deviated from the normal lane line, which is a violation."

[0121] It can be seen from this that through further judgment and screening of the target recognition result, the advantages of the multi-modal large model in visual understanding can be fully utilized, and the target recognition result is more accurate.

[0122] The embodiments of the present disclosure can be applied not only in the field of target recognition but also in the field of target counting, and can be specifically applied to the following scenarios:

[0123] 1. Smart industrial scenarios: Safety helmet wearing monitoring (such as "red safety helmet worn on the head"), factory personnel action recognition (such as "person operating the equipment"), production line item counting and classification (such as "number of labeled packaging boxes").

[0124] 2. Smart agricultural scenarios: Automatic counting of crops (such as "number of ripe red tomatoes"), livestock and poultry number recognition (such as "cows standing in the drinking area"), estimation of the number of fruits hanging on fruit trees (such as "unpicked apples on the branches").

[0125] 3. Urban traffic scenarios: Road vehicle type counting (such as "trucks stopped at a red light"), non-motor vehicle behavior recognition (such as "electric vehicle going in the wrong direction"), pedestrian safety behavior analysis (such as "construction workers not wearing reflective vests").

[0126] 4. Smart security scenarios: Suspicious behavior monitoring (e.g., "people entering the elevator without wearing a mask"), regional intrusion target counting (e.g., "the number of people crossing the warning line"), and nighttime intrusion target recognition (combining infrared images).

[0127] 5. Medical and health monitoring scenarios: Supervision of medical staff operation specifications (e.g., "the number of doctors performing surgery"), and recognition of patient status in medical scenarios (e.g., "the elderly sitting in a wheelchair").

[0128] 6. Retail and business analysis scenarios: Customer behavior statistics (e.g., "the number of customers queuing to pay"), analysis of the status of product display (e.g., "the number of mineral water bottles remaining on the shelf"), and recognition of advertising exposure targets (e.g., "people standing in front of the advertising screen for more than 3 seconds").

[0129] 7. Education and teaching management scenarios: Monitoring the status of students in the classroom (e.g., "students raising their hands"), analyzing the attention of students in distance education (e.g., "students looking at the screen"), and monitoring the behavior in the examination room (e.g., "examinees standing up and leaving their seats").

[0130] 8. Scenarios combined with augmented reality (AR) / mixed reality (MR): Real-time target counting and projection of prompt information (e.g., marking "5 people not wearing work uniforms in the dangerous area" on AR glasses), and intelligent marking in the virtual-real combined scenario (e.g., recognizing "the student avatar raising their hand" in educational AR).

[0131] 9. Scenarios combined with robots / automatic control systems: Automatic robot autonomous task planning (e.g., "carrying all red crates"), and active interaction of service robots (e.g., "recognizing people looking at the robot").

[0132] 10. Scenarios combined with in-vehicle systems: Screening of special targets in intelligent driving (e.g., "pedestrians wearing red clothes while cycling"), and automatic parking recognition (e.g., "the number of people standing in the empty parking spaces").

[0133] 11. Scenarios combined with wearable devices: Sports analysis (e.g., "the number of people running"), and medical assistance (e.g., "patients not wearing hospital gowns").

[0134] The embodiments of the present disclosure propose a two-stage target recognition scheme. In the first stage, target class extraction and target recognition are performed, and in the second stage, a multi-modal large model is used for complex semantic understanding and target screening. Based on the original image input by the user and the language description information containing complex semantics, target class extraction, semantic understanding, and target screening are automatically completed, and finally the counting result of the target to be recognized is output, solving the problem of automatic recognition of targets in images under complex language descriptions, and being applicable to practical scenarios in multiple fields such as intelligent industry, agriculture, and urban transportation.

[0135] In the embodiments of the present disclosure, through natural language processing technology, the target classes in the user input language are automatically identified as the core recognition objects for subsequent image analysis, significantly enhancing the ability to understand complex semantics; the target class extraction can be performed based on traditional natural language processing models (such as BERT), or prompts can be constructed through large language models (such as Qwen, DeepSeek) for extraction. Since the corpus of traditional object detection models does not contain complex semantic constraints, if the target classes are not recognized and the user's language description information is directly analyzed by the object detection model, chaotic and incorrect object recognition results will be output.

[0136] In the embodiments of the present disclosure, multiple object detection models are used to collaboratively identify all the objects in the element image, and the non-maximum suppression (NMS) algorithm is used to merge the object recognition results to remove redundant candidate objects, maximizing the accuracy and recall rate of object recognition. Further, in the embodiments of the present disclosure, each recognized object is combined with the language description, and a multi-modal prompt is constructed and input to the multi-modal large model for judgment. The multi-modal large model can fuse the dual-modal information of images and texts to accurately judge whether each object meets the additional conditions in the user input language description information, such as color, position, etc. Finally, the number of all objects that meet the conditions is counted, and the counting result and the visual information of the objects are output.

[0137] The embodiments of the present disclosure have a flexible deployment architecture, which can run completely on the terminal side or adopt a terminal-cloud collaboration mode to support the efficient operation of resource-constrained devices, significantly improving the image object recognition and counting accuracy and intelligence level in complex semantic scenarios.

[0138] The embodiments of the present disclosure support the target counting task of complex language descriptions. Most traditional object recognition is based on category labels for object recognition and counting, and usually can only process simple instructions such as "all people in the statistical chart", and it is difficult to understand and execute language descriptions with multiple semantic restrictions, such as "red safety helmets worn on the head". Through the introduction of natural language processing models and multi-modal large models, the embodiments of the present disclosure achieve the automatic understanding and object screening of such complex descriptions, thus greatly expanding the applicable range and intelligence level of object recognition. In the first stage of the embodiments of the present disclosure, target class extraction and object recognition are performed, and in the second stage, semantic screening is performed on the recognized objects. The above two stages cooperate with each other, effectively taking into account the efficiency and accuracy of object recognition and counting.

[0139] The embodiments of the present disclosure propose a process chain of "extracting target classes → identifying the objects of the target classes → multi-modal screening of objects", deeply integrating natural language understanding and visual detection. Especially this intermediate semantic bridge of extracting target classes greatly enhances the collaborative ability between the visual model and the language model, which is the core progress different from the traditional direct coupling method of vision and language.

[0140] The multi-modal prompt assembly method in the embodiments of the present disclosure not only enhances the large model's ability to understand the semantic relationship between images and texts, but also, by supporting various representation forms (such as text form, visual form) at the target position and various model decision-making mechanisms, has greatly improved performance compared to single-modal or single-model.

[0141] In the case where there are multiple targets in an image but only some targets meet the restrictive conditions of complex descriptions, traditional object recognition schemes will "count all" or misjudge, while the embodiments of the present disclosure can accurately identify the targets that "match the language description", such as distinguishing between "a red safety helmet worn on the head" and "a yellow safety helmet held in the hand", effectively solving the shortcomings of traditional schemes in semantic resolution and logical judgment capabilities.

[0142] The embodiments of the present disclosure also provide an electronic device, such as Figure 9 shown, including a memory and a processor; the memory stores a computer program executable by the processor, and when the computer program is executed by the processor, any one of the object recognition methods in the embodiments of the present disclosure is implemented.

[0143] Among them, the processor is a device with data processing capabilities, including but not limited to a central processing unit (CPU), etc.; the memory is a device with data storage capabilities, including but not limited to a random access memory (RAM, more specifically such as SDRAM, DDR, etc.), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory (FLASH); the I / O interface (read / write interface) is connected between the processor and the memory and can realize the information interaction between the memory and the processor, including but not limited to a data bus (Bus), etc.

[0144] The embodiments of the present disclosure also provide a computer-readable medium, on which a computer program is stored, and when the program is executed, the object recognition method as described above is implemented.

[0145] The embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the object recognition method as described above is implemented.

[0146] Those of ordinary skill in the art can understand that all or some of the steps, systems, and functional modules / units in the devices disclosed above can be implemented as software, firmware, hardware, and their appropriate combinations.

[0147] In the hardware implementation, the division between the functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be executed by several physical components in cooperation.

[0148] Some or all physical components may be implemented as software executed by a processor, such as a central processing unit (CPU), a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH) or other disk storage; read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical disk storage; magnetic cassettes, tapes, disk storage or other magnetic storage; any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0149] The present disclosure has disclosed example embodiments, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for limiting purposes. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly stated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, those skilled in the art will appreciate that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A target recognition method, characterized in that: The method comprises: Acquire an original image and language description information of an object to be identified in the original image; Inputting the language description information into a first model, and extracting target class information from the language description information through the first model, wherein the target class information is information about the category to which the target to be identified belongs; Inputting the original image and the target class information into a second model, and identifying all first targets corresponding to the target class information in the original image through the second model; Generate a multimodal prompt word according to the original image, the language description information and each of the first targets; The multimodal prompt word is input into a third model, and through the third model, a second target satisfying the multimodal prompt word is screened out from each of the first targets, and the second target is determined as the target to be identified.

2. The method according to claim 1, characterized in that The step of inputting the original image and the target class information into a second model, and identifying all first targets corresponding to the target class information in the original image by using the second model, comprises: Inputting the original image and the target class information into at least two second models respectively, identifying the target corresponding to the target class information in the original image by each of the second models respectively, and summarizing the targets identified by each of the second models to obtain a first target set; wherein each of the targets includes a target position and a confidence level; The targets in the first target set are merged according to their target positions and confidence levels to obtain a second target set, and the targets in the second target set are determined to be all the first targets corresponding to the target class information.

3. The method according to claim 2, characterized in that The target position is represented by a marked box or a marked point, and the marked point is the center point of the marked box.

4. The method according to claim 3, characterized in that The target position is represented by a marked box, and the targets in the first target set are merged according to the target position and confidence of each target to obtain a second target set, including: The intersection-and-union ratio calculation step includes: determining a designated target corresponding to a maximum confidence value and a first candidate target other than the designated target from the first target set, and calculating the intersection-and-union ratio between the designated target's labeled box and the labeled boxes of the first candidate targets respectively according to the labeled box of the designated target and the labeled boxes of the first candidate targets; Target screening step: determining redundant first candidate targets whose intersection-over-union ratio is greater than a preset threshold, deleting the redundant first candidate targets from the first target set, and obtaining an updated first target set; Repeat the intersection-and-union ratio calculation step and the target screening step until all targets in the first target set are used as the designated targets and the intersection-and-union ratio calculation step and the target screening step are completed, and the currently updated first target set is determined as the second target set.

5. The method according to claim 1, characterized in that The generating of a multimodal prompt word according to the original image, the language description information and each of the first targets includes: For each of the first targets, generating a multimodal prompt word according to the original image, the language description information and the target position of the first target; or, A multimodal prompt word is generated according to the language description information and the target position of each of the first targets.

6. The method according to claim 1, characterized in that The third model is at least two, and the multimodal prompt word is input into the third model, and the second goal that satisfies the multimodal prompt word is screened out from each of the first goals through the third model, including: Inputting the multimodal prompt words into each of the third models respectively; For each of the third models, second candidate targets are screened out from the first targets through the third model, where the second candidate targets are targets that meet the multimodal prompt words corresponding to the third model; The second target is selected from the second candidate targets corresponding to the third models.

7. The method according to claim 1, characterized in that After inputting the multimodal prompt word into a third model and selecting a second goal satisfying the multimodal prompt word from each of the first goals through the third model, the method further includes: Count the number of the targets to be identified.

8. The method according to any one of claims 1 to 7, characterized in that: The first model, the second model and the third model are deployed on a terminal device; or, The first model and the second model are deployed on the terminal device, and the third model is deployed on the cloud; or, The first model is deployed on a terminal device, and the second model and the third model are deployed on the cloud.

9. An electronic device comprising a memory and a processor; the memory stores a computer program that can be executed by the processor, and when the computer program is executed by the processor, the target recognition method according to any one of claims 1 to 8 is implemented.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the target recognition method according to any one of claims 1 to 8 is implemented.

Images