Target object identification method and device based on large model

Abstract

The invention relates to the technical field of image processing, in particular to a target object recognition method and device based on a large model. Identifying the positions of candidate targets in the to-be-identified image based on the target identification large model and the target identification small model, then determining a target sub-image including each candidate target in the to-be-identified image, inputting the target sub-image into the target identification large model, determining a sub-image set including the sub-images of each target object, and identifying the positions of the candidate targets in the to-be-identified image based on the sub-image set. The resolution of the image input into the target identification large model is reduced, other irrelevant areas are removed, the target identification large model more focuses on the image area including the candidate target, and the sub-image corresponding to each target object and the first prompt text are input into the target identification large model, so that the target identification accuracy is improved. The description text of the target object included in each sub-image is obtained, so that the category to which each target object belongs is determined according to the description text, and the target object recognition accuracy of the target recognition large model is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method and device for identifying a target object based on a large model. Background Art

[0002] In some business scenarios, it is necessary to identify objects in an image and determine the category to which each object belongs. For example, in the railway access scenario, railway workers need to carry tools to enter the railway for work. At this time, it is necessary to identify the number of tools carried by the workers and the category to which each tool belongs. Due to the complexity of the real working environment, the types of tools carried by the workers are not limited, and the tools carried by different workers may be different. In order to clearly understand the detailed information of the tools carried by each worker for easy management, we hope to identify the category to which each tool carried by the worker belongs.

[0003] At present, the small target detection model cannot recognize and detect objects that have not been learned. Since the categories of the objects to be recognized are not fixed, the small target detection model alone cannot accurately detect each object, and it is even more impossible to accurately determine the category to which each object belongs. Although the current large model can recognize a wider range of objects than the small target detection model, the large model's recognition accuracy for target objects in images is not high.

[0004] Therefore, how to accurately identify objects in images becomes a problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a target object recognition method and device based on a large model, so as to solve the problem of low accuracy in object recognition in images in the prior art.

[0006] The present application provides a target object recognition method based on a large model, the method comprising:

[0007] Recognize the positions of the candidate targets in the image to be identified based on the target recognition large model and the target recognition small model respectively, and determine the target sub-image in the image to be identified, wherein each candidate target is included in the target sub-image;

[0008] The target sub-image is input into the target recognition large model to obtain a sub-image set, wherein the sub-image set includes a sub-image corresponding to each target object; each sub-image and a first prompt text are input into the target recognition large model to obtain a description text of the target object included in each sub-image; and the category to which each target object belongs is determined based on the description text of each target object.

[0009] The above technical scheme, when performing target object recognition, identifies the positions of candidate targets in the image to be recognized based on the target recognition large model and the target recognition small model respectively, and then determines the target sub-image including each candidate target in the image to be recognized according to the position of the candidate target, and inputs the target sub-image into the target recognition large model, determines the sub-image set including the sub-image of each target object, reduces the resolution of the image input into the target recognition large model, removes other irrelevant areas, makes the target recognition large model more focused on the image area including the candidate targets, makes the target recognition large model recognize all target objects as much as possible, and then inputs the sub-image corresponding to each target object and the first prompt text into the target recognition large model, obtains the description text of the target object included in each sub-image, thereby determining the category to which each target object belongs according to the description text, and improves the accuracy of the target recognition large model in recognizing target objects.

[0010] Furthermore, the determining of the target sub-image in the image to be identified based on the positions of the candidate targets identified by the target recognition large model and the target recognition small model in the image to be identified respectively includes:

[0011] Inputting the image to be recognized into the target recognition small model to recognize the position of each first candidate target;

[0012] Inputting the image to be recognized and the second prompt text into the target recognition large model to recognize the position of each second candidate target;

[0013] According to the position of each first candidate target and the position of each second candidate target, a target sub-image in the image to be identified is determined.

[0014] The above technical solution uses the target recognition large model and the target recognition small model to recognize the targets in the image to be recognized, and then determines a target sub-image including all targets based on the recognition results of each model, which effectively avoids the problem that a single model cannot recognize all objects included in the image at once during the target recognition process. In other words, superimposing the recognition results of the target recognition large model and the target recognition small model can ensure that all objects in the image to be recognized are recognized as much as possible, thereby improving the accuracy of target object recognition.

[0015] Furthermore, the image to be recognized is input into the target recognition model to recognize the position of each first candidate target, including:

[0016] Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target;

[0017] Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes:

[0018] If the confidence corresponding to any first candidate target is greater than a preset threshold, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if not, the sub-image corresponding to the first candidate target is added to the sub-image set.

[0019] In the above technical solution, if the confidence corresponding to any first candidate target identified by the target recognition small model is greater than a preset threshold, it is determined whether the first candidate target has been recognized by the target recognition large model. If not, the sub-image of the first candidate target is added to the sub-image set to ensure that all target objects in the image to be identified are identified without omission, thereby improving the accuracy of target object recognition.

[0020] Furthermore, the image to be recognized is input into the target recognition model to recognize the position of each first candidate target, including:

[0021] Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target;

[0022] Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes:

[0023] If the confidence level corresponding to any first candidate target is not greater than the preset threshold, determining whether the first candidate target exists in each of the second candidate targets according to the degree of overlap between the position of the first candidate target and the position of each of the second candidate targets;

[0024] If yes, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if no, the sub-image corresponding to the first candidate target is added to the sub-image set.

[0025] In the above technical scheme, if the confidence corresponding to any first candidate target identified by the small target recognition model is not greater than the preset threshold, it is determined whether the large target recognition model has recognized the first candidate target when processing the image to be recognized. If so, it is possible to continue to determine whether the large target recognition model has recognized the first candidate target when processing the target sub-image. If not, the sub-image corresponding to the first candidate target can be added to the sub-image set to ensure that all target objects in the image to be recognized are recognized without omission, thereby improving the accuracy of target object recognition.

[0026] Furthermore, the step of inputting the target sub-image into the target recognition model to determine the sub-image corresponding to each target object includes:

[0027] Inputting the target sub-image and the third prompt text into the target recognition large model to recognize the position of each target object;

[0028] According to the position of each target object, a sub-image corresponding to each target object is determined.

[0029] The above technical solution does not allow the target recognition model to directly output the sub-image corresponding to each target object, but first uses the target recognition model to identify the position of each target object, and then determines the sub-image corresponding to each target object based on the position of each target object. Compared with directly outputting the final sub-image corresponding to each target object, only outputting the position of each target object reduces the processing steps of the target recognition model and improves the accuracy of the target recognition model in identifying each target object to a certain extent.

[0030] Furthermore, after determining the target sub-image in the image to be recognized and before inputting the target sub-image into the target recognition large model, the method further includes:

[0031] Expanding the coverage area of ​​the target sub-image in the image to be identified by a set multiple to obtain a complete sub-image, so as to ensure that the complete sub-image includes complete images of all candidate targets;

[0032] The target sub-image is updated using the complete sub-image.

[0033] The above technical solution, after determining the target sub-image, expands the coverage area of ​​the target sub-image in the image to be identified by a set multiple to obtain a complete sub-image of the complete image including all targets, and then uses the complete sub-image to update the target sub-image to ensure that the target object is identified based on the complete image including all candidate targets in the subsequent target object recognition process, thereby improving the accuracy of target object recognition.

[0034] Further, the target recognition large model includes an input layer, a high-resolution visual encoder, a low-resolution visual encoder, a visual adapter, a text encoder and a large language model;

[0035] The process of the target recognition model recognizing any received image includes:

[0036] The input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, and inputs the image into the target visual encoder for feature extraction to obtain a first feature matrix;

[0037] Inputting the first feature matrix into the visual adapter for feature mapping to obtain a second feature matrix;

[0038] Inputting the prompt text received when receiving the image into the text encoder to obtain a text feature matrix;

[0039] The second feature matrix and the text feature matrix are input into the large language model to obtain description information for describing the target included in the image.

[0040] Further, the input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, including:

[0041] The input layer determines whether the resolution of the image is greater than a preset resolution threshold;

[0042] If yes, determining the high-resolution visual encoder as the target visual encoder;

[0043] If not, the low-resolution visual encoder is determined as the target visual encoder.

[0044] The above technical solution, when the large model is performing inference, adaptively selects the corresponding visual encoder according to the resolution of the received image, so that different visual encoders focus on processing images of different resolutions, thereby improving the accuracy of feature extraction by the visual encoder, and thereby improving the ability of the large target recognition model to recognize the received image.

[0045] Furthermore, the target recognition large model is obtained by fine-tuning training based on the instruction data set;

[0046] The construction process of the instruction data set includes:

[0047] Get a sample image;

[0048] Recognize the sample image based on the pre-trained large model to obtain a global description;

[0049] Based on the small target recognition model, identifying the position information of the target included in the sample image;

[0050] Inputting the sub-image corresponding to the position information in the sample image into a universal recognition model to obtain a description of the category to which the target object corresponding to the sub-image belongs;

[0051] Input the global description, the location information and the corresponding category description into the pre-trained large model to obtain answer information corresponding to each preset question;

[0052] Each preset question and the corresponding answer information are obtained and stored in the instruction data set.

[0053] The above technical solution, a method for constructing a multi-level and multi-scale instruction dataset, improves the recognition effect of the target recognition large model on this type of target object by fine-tuning training based on the instruction dataset constructed in this way.

[0054] The present application provides a target object recognition device based on a large model, the device comprising:

[0055] A determination module is used to identify the positions of candidate targets in the image to be identified based on the target recognition large model and the target recognition small model, respectively, to determine a target sub-image in the image to be identified, wherein each candidate target is included in the target sub-image; the target sub-image is input into the target recognition large model to obtain a sub-image set, wherein the sub-image set includes a sub-image corresponding to each target object;

[0056] The recognition module is used to input each sub-image and the first prompt text into the target recognition model to obtain the description text of the target object included in each sub-image; and determine the category to which each target object belongs based on the description text of each target object.

[0057] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned large model-based target object recognition methods when executing a computer program stored in a memory.

[0058] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned large model-based target object recognition methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 A schematic diagram of a process of identifying a target object based on a large model provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a sub-image provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a target object recognition result provided in an embodiment of the present application;

[0063] Figure 4 A schematic diagram of a target sub-image in an image to be identified provided in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of a target sub-image provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the structure of a large target recognition model provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of a target object recognition process based on a large model provided in an embodiment of the present application;

[0067] Figure 8 A schematic diagram of the structure of a target object recognition device based on a large model provided in an embodiment of the present application;

[0068] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0070] An embodiment of the present application provides a target object recognition method and device based on a large model, in which the position of the candidate target is identified in the image to be recognized based on the target recognition large model and the target recognition small model, and the target sub-image in the image to be recognized is determined, and each candidate target is included in the target sub-image; the target sub-image is input into the target recognition large model to obtain a sub-image set, and the sub-image set includes a sub-image corresponding to each target object; each sub-image and a first prompt text are input into the target recognition large model to obtain a description text of the target object included in each sub-image; and the category to which each target object belongs is determined according to the description text of each target object.

[0071] The target recognition method based on a large model provided in the embodiment of the present application can be used for the number and category name recognition of target objects of uncertain classes in open scenes. The method is suitable for detection and recognition in scenes where only the large category of the target object is known, but its specific category name is unclear. Through the method of combining large and small models, the position of each target object can be accurately detected, thereby obtaining an accurate quantity result. With the help of a small model, the position of the target object can be accurately obtained. For targets that cannot be recognized by the small model, its position can be obtained through the large model. In addition, through the large model, a detailed description of the category name of each target object can also be obtained. By detecting the above information, the target quantity information can be obtained. Based on this method, the problem that the small model cannot recognize unseen classes is solved, and the accuracy of target quantity and name recognition is further improved. It has the advantages of high recognition accuracy, not being limited to specified targets, and wide application. It is described below in conjunction with each embodiment.

[0072] Figure 1 A flowchart of a target object recognition process based on a large model is provided in an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0073] S101: Identify positions of candidate targets in an image to be identified based on a large target recognition model and a small target recognition model, respectively, and determine a target sub-image in the image to be identified, wherein each candidate target is included in the target sub-image.

[0074] The large model-based target object recognition method provided in the present application is applied to an electronic device, which may be a server, a PC, a mobile terminal, etc.

[0075] In some business scenarios, due to the complexity of the real working environment, the types of tools carried by the staff are not limited, and the tools carried by different staff may be different. Non-professionals may not be able to accurately tell the category to which some tools belong. In order to accurately identify the target object, in an embodiment of the present application, the target object in the image to be identified can be identified based on a pre-trained model. Among them, the target object may be a tool carried by the staff, such as a work light, a shovel, an electric drill, etc. Of course, the target object is not limited to tools, and the target object can also be a vehicle, an animal, a plant, etc. Among them, the image to be identified is an image including one or more target objects, and the image to be identified may be stored in a preset storage location, or may be sent by a user or other electronic device.

[0076] Since the small target recognition model may not accurately identify the target object because it has not learned a certain object during training. In order to accurately and comprehensively identify all target objects included in the image to be identified, in an embodiment of the present application, the target recognition large model and the target recognition small model can be used to identify the image to be identified respectively, and the position of each candidate target identified by the target recognition large model and the position of each candidate target identified by the target recognition small model are obtained. Among them, the position can be understood as the coordinate information corresponding to the candidate target, for example, the coordinate information of the detection frame corresponding to the candidate target. Specifically, if the detection frame is rectangular, the coordinate information of the detection frame can be the coordinate information of the upper left corner and the lower right corner of the detection frame. Among them, the small target recognition model can be a pre-trained detection model that can detect target objects of known categories. The large target recognition model can be any multimodal large model with image processing capabilities. The large target recognition model can be a multimodal large model fine-tuned based on the category of the object to be identified. The large target recognition model has reference understanding ability, Grounding ability, detailed description ability, and complex reasoning ability.

[0077] After obtaining the positions of the candidate targets identified by the target recognition large model and the target recognition small model respectively, the minimum bounding rectangle that includes all the candidate targets can be determined according to the position of each candidate target, and the image area corresponding to the minimum bounding rectangle can be called a target sub-image. In other words, the target sub-image includes all the candidate targets currently identified.

[0078] S102: Input the target sub-image into the target recognition large model to obtain a sub-image set, wherein the sub-image set includes a sub-image corresponding to each target object.

[0079] After obtaining the target sub-image including each candidate target, the target sub-image can be input into the target recognition large model, which processes the target sub-image and identifies the sub-image corresponding to each candidate target included in the target sub-image. In the embodiment of the present application, the resolution of the input image is reduced, other irrelevant areas are removed, and only the focus part is input into the target recognition large model, so that the target recognition large model can identify each target more accurately. In the embodiment of the present application, each candidate target identified by the target recognition large model in the target sub-image can be called a target object. The set of sub-images corresponding to all target objects can be called a sub-image set.

[0080] In an embodiment of the present application, when using the target recognition large model to identify the sub-image corresponding to each target object in the target sub-image, the prompt text "Please identify the target object included in the image, and output a sub-image including only the target object. The target object can be a tool, such as a tool light, ruler, axe, shovel, etc." can be input into the target recognition large model together with the target sub-image, so that the target recognition large model processes the received target sub-image according to the requirements of the prompt text.

[0081] S103: Input each sub-image and the first prompt text into the target recognition model to obtain a description text of the target object included in each sub-image; and determine the category to which each target object belongs based on the description text of each target object.

[0082] In order to determine the category to which each target object belongs, in an embodiment of the present application, the sub-image corresponding to each target object included in the sub-image set and the first prompt text can be input into the target recognition model to obtain the description text of the target object included in each sub-image.

[0083] Specifically, Figure 2 A schematic diagram of a sub-image provided in an embodiment of the present application. In an embodiment of the present application, Figure 2 And the first prompt text is input into the target recognition model. Exemplarily, the first prompt text may be “What tool is in the picture?” After receiving the sub-image and the first prompt text, the description text output by the target recognition model may be “It is a portable work light”.

[0084] After obtaining the description text of each target object, the category to which each target object belongs can be determined according to the description text of each target object. For example, Figure 2 The target object in the figure belongs to the category of portable work light.

[0085] In a possible implementation manner, the identified target objects may be counted to determine the number of the target objects.

[0086] The target object recognition method based on the large model provided in the embodiment of the present application can obtain the position of the detection frame of each target object and the corresponding category name, thereby obtaining the data volume, position and category name of the target object in the image. For example, the final recognition result obtained can be:

[0087] “Number of tools: 6;

[0088] Tool location and category name:

[0089] “ <ref> Dragon Saw< / ref> <box> (333,159),(947,1011)< / box> ,

[0090] <ref> Maintenance hammer< / ref> <box> (756,62),(1045,211)< / box> ,

[0091] <ref> sprayer< / ref> <box> (671,127),(951,459)< / box> ,

[0092] <ref> Pile Driver< / ref> <box> (1079,235),(1434,524)< / box> ,

[0093] <ref> Small shovel< / ref> <box> (958,525),(1047,612)< / box> ,

[0094] <ref> Portable tool light spray< / ref> <box> (1027,571),(1443,947)< / box> . "

[0095] in," <ref> …< / ref> " indicates the category to which the corresponding target object belongs. <box> …< / box> The content between " represents the location information of the corresponding target object detection frame. <box> (333,159),(947,1011)< / box> Take as an example, where (333,159) is the coordinate of the upper left corner of the corresponding detection box, and (947,1011) is the coordinate of the lower right corner of the corresponding detection box.

[0096] In order to facilitate viewing the recognition results of the target object, the following can be obtained based on the position information of the recognized target object and the category to which it belongs: Figure 3 The recognition results shown are Figure 3 A schematic diagram of a target object recognition result provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, each target object is marked out in the original image (ie, the image to be identified) using a rectangle, and the identified category name is added to each target object.

[0097] Because in the embodiment of the present application, when performing target object recognition, the positions of candidate targets are identified in the image to be recognized based on the target recognition large model and the target recognition small model respectively, and then the target sub-image including each candidate target in the image to be recognized is determined according to the position of the candidate target, and the target sub-image is input into the target recognition large model, and the sub-image set including the sub-image corresponding to each target object is determined, thereby reducing the resolution of the image input into the target recognition large model, removing other irrelevant areas, making the target recognition large model more focused on the image area including the candidate targets, and making the target recognition large model recognize all target objects as much as possible, and then the sub-image corresponding to each target object and the first prompt text are input into the target recognition large model, and the description text of the target object included in each sub-image is obtained, thereby determining the category to which each target object belongs according to the description text, and improving the accuracy of the target recognition large model in recognizing target objects.

[0098] In order to further improve the accuracy of target object recognition, based on the above embodiments, in the embodiment of the present application, the positions of the candidate targets respectively identified by the target recognition large model and the target recognition small model in the image to be recognized, and determining the target sub-image in the image to be recognized, include:

[0099] Inputting the image to be recognized into the target recognition small model to recognize the position of each first candidate target;

[0100] Inputting the image to be recognized and the second prompt text into the target recognition large model to recognize the position of each second candidate target;

[0101] According to the position of each first candidate target and the position of each second candidate target, a target sub-image in the image to be identified is determined.

[0102] In an embodiment of the present application, when determining the target sub-image in the image to be identified based on the positions of the candidate targets identified in the image to be identified by the target recognition large model and the target recognition small model respectively, the image to be identified can be input into the target recognition small model to identify the position of each first candidate target, for example, to identify the coordinate information corresponding to the detection box of each first candidate target.

[0103] When using the target recognition small model to identify the first candidate target, the image to be identified and the second prompt text can be input into the target recognition large model to identify the position of each second candidate target, for example, to identify the coordinate information corresponding to the detection box of each second candidate target. Exemplarily, the second prompt text can be "Please output the position of each tool in the figure", and the second prompt text can be determined based on a prompt template, and the prompt template can be "Please output the position of each {} in the figure", where the content in "{}" can be filled in according to the actual situation. For example, when identifying tools, "tools" can be filled in the "{}"; when identifying vehicles, "vehicles" can be filled in the "{}"; when identifying pedestrians, "people" can be filled in the "{}". Of course, for ease of understanding, the target recognition large model can not only output the position of each second candidate target, but also output other information such as the name of each second candidate target.

[0104] After the positions of the first candidate target and the second candidate target are obtained, the target sub-image in the image to be recognized can be determined according to the position of each first candidate target and the second candidate target.

[0105] Specifically, Figure 4 A schematic diagram of a target sub-image in an image to be identified provided in an embodiment of the present application, Figure 4The objects marked by the four white solid rectangles in the figure are the first candidate targets identified by the small target recognition model, and the objects marked by the three black solid rectangles are the second candidate targets identified by the large target recognition model. Figure 4 It can be seen that there may be duplication between the candidate targets identified by the target recognition large model and the target recognition small model. After obtaining the position of each first candidate target and the position of each second candidate target, the rectangular boxes corresponding to the positions of the candidate targets (including the first candidate target and the second candidate target) identified by the target recognition large model and the target recognition small model can be superimposed on the image to be identified, and the image area corresponding to the minimum circumscribed rectangular box of all identified candidate targets is determined as the target sub-image. Figure 4 The image area corresponding to the largest rectangular box of the white dashed line is the target sub-image. When the target sub-image is processed by the target recognition model, the target sub-image can be cut out from the image to be recognized. Figure 5 The target sub-image shown is input into the target recognition model to identify the target object. Figure 5 A schematic diagram of a target sub-image provided in an embodiment of the present application.

[0106] In the embodiment of the present application, the target in the image to be identified is identified using a large target identification model and a small target identification model, and then a target sub-image including all targets is determined based on the identification results of each model, which effectively avoids the problem that a single model cannot identify all objects included in the image at once during the target identification process. In other words, by superimposing the identification results of the large target identification model and the small target identification model, it is possible to ensure that all objects in the image to be identified are identified as much as possible, thereby improving the accuracy of target object identification.

[0107] In order to further improve the accuracy of target object recognition, based on the above embodiments, in the embodiment of the present application, the image to be recognized is input into the target recognition small model to recognize the position of each first candidate target, including:

[0108] Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target;

[0109] Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes:

[0110] If the confidence corresponding to any first candidate target is greater than a preset threshold, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if not, the sub-image corresponding to the first candidate target is added to the sub-image set.

[0111] Although inputting the target sub-image with reduced resolution into the target recognition large model can improve the accuracy of target recognition of the target recognition large model, since the target recognition large model may be uncontrollable, in order to further improve the accuracy of target object recognition, in an embodiment of the present application, the target recognition small model can also output the confidence when outputting the position of each first candidate target. The confidence can be understood as the possibility that the first candidate target corresponding to the position is indeed the target. For example, the confidence that the category to which the identified first candidate target belongs is "category A". The greater the confidence, the greater the possibility of correctly identifying the corresponding first candidate target. In other words, the target recognition small model can output the position of the first candidate target, the category to which the first candidate target belongs, and the confidence. The target recognition small model can identify the large category to which the target object belongs, for example, a shovel. It can also identify the small category to which the target object belongs, for example, a long-handled shovel, a short-handled shovel, etc.

[0112] In order to ensure that all target objects in the image to be identified are identified during the target object recognition process, before each sub-image and the first prompt text are input into the target recognition model, the relationship between the confidence of each first candidate target and the preset threshold can also be determined.

[0113] If the confidence level corresponding to any first candidate target is greater than a preset threshold, it means that the first candidate target is indeed very likely to be the target object, and the first candidate target can be directly determined as the target object. However, for the first candidate target object, since the first candidate target object is included in the target sub-image, the target recognition large model may have identified the first candidate target object when identifying the target sub-image. Therefore, in an embodiment of the present application, if the confidence level corresponding to any first candidate target is greater than a preset threshold, it can be determined whether the sub-image corresponding to the first candidate target exists in the sub-image set for storing the sub-image of the target object. If it exists, it means that the target recognition large model has identified the first candidate target when identifying the target sub-image. If it does not exist, it means that the target recognition large model did not identify the first candidate target when identifying the target sub-image, and the target recognition small model believes that the confidence level of the first candidate target is very high, then the first candidate target can be determined as the target object, and the sub-image corresponding to the first candidate target is added to the sub-image set.

[0114] In an embodiment of the present application, if the confidence corresponding to any first candidate target identified by the target recognition small model is greater than a preset threshold, it is determined whether the first candidate target has been recognized by the target recognition large model. If not, the sub-image of the first candidate target is added to the sub-image set to ensure that all target objects in the image to be identified are identified without omission, thereby improving the accuracy of target object recognition.

[0115] In order to further improve the accuracy of target object recognition, based on the above embodiments, in the embodiment of the present application, the image to be recognized is input into the target recognition small model to recognize the position of each first candidate target, including:

[0116] Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target;

[0117] Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes:

[0118] If the confidence level corresponding to any first candidate target is not greater than the preset threshold, determining whether the first candidate target exists in each of the second candidate targets according to the degree of overlap between the position of the first candidate target and the position of each of the second candidate targets;

[0119] If yes, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if no, the sub-image corresponding to the first candidate target is added to the sub-image set.

[0120] If the confidence level corresponding to any first candidate target is not greater than the preset threshold, it means that the small target recognition model believes that the first candidate target is unlikely to be the target object. However, the small target recognition model may have a low confidence level in identifying the first candidate target because it did not learn the relevant knowledge of the first candidate target during the training phase. Therefore, in the embodiment of the present application, it can be further determined whether the first candidate target has been identified by the large target recognition model when identifying the image to be identified.

[0121] Specifically, it can be determined whether the first candidate target exists in each second candidate target based on the degree of overlap between the position of the first candidate target and the position of each second candidate target. Exemplarily, it can be determined that the rectangular frame corresponding to each candidate target is based on the position of each candidate target. When determining whether the first candidate target exists in the second candidate target, the degree of overlap between the matrix frame of the first candidate target and the rectangular frame corresponding to each second candidate target can be determined respectively. When the degree of overlap is greater than the overlap threshold, it can be considered that the corresponding second candidate target is the same target as the first candidate target. Exemplary overlap thresholds can be 98%, 90%, etc. In other words, the first candidate target exists in the second candidate target identified by the target recognition large model.

[0122] When it is determined that the first candidate target exists in the second candidate target identified by the target recognition large model, it means that the target recognition small model believes that the possibility of the first candidate target being the target object is relatively small, but the target recognition large model believes that the first candidate target may be the target object. Therefore, in an embodiment of the present application, it can be determined whether the sub-image corresponding to the first candidate target exists in the sub-image set. If it exists, it means that the target recognition large model has identified the first candidate target when identifying the target sub-image. If it does not exist, it means that the target recognition large model did not identify the first candidate target when identifying the target sub-image, and the target recognition large model identified the first candidate target when identifying the image to be identified, then the first candidate target can be determined as the target object, and the sub-image corresponding to the first candidate target is added to the sub-image set.

[0123] In an embodiment of the present application, if the confidence corresponding to any first candidate target identified by the small target recognition model is not greater than a preset threshold, it is determined whether the large target recognition model has recognized the first candidate target when processing the image to be recognized. If so, it is possible to continue to determine whether the large target recognition model has recognized the first candidate target when processing the target sub-image. If not, the sub-image corresponding to the first candidate target can be added to the sub-image set to ensure that all target objects in the image to be recognized are recognized without omission, thereby improving the accuracy of target object recognition.

[0124] In order to further improve the accuracy of target object recognition, based on the above embodiments, in the embodiment of the present application, the target sub-image is input into the target recognition large model to determine the sub-image corresponding to each target object, including:

[0125] Inputting the target sub-image and the third prompt text into the target recognition large model to recognize the position of each target object;

[0126] According to the position of each target object, a sub-image corresponding to each target object is determined.

[0127] In order to further improve the accuracy of target object recognition, when using the target recognition large model to determine the sub-image corresponding to each target object, the target recognition large model can be made to do as little processing as possible to improve the accuracy of determining the sub-image. In an embodiment of the present application, the target sub-image and the third prompt text can be input into the target recognition large model to identify the position of each target object. Among them, the third prompt text is a text used to prompt the target recognition large model to analyze the received image, identify each target object included in the image, and output the coordinate information of each target object.

[0128] After the position of each target object is obtained, the sub-image corresponding to each target object can be determined in the target sub-image according to the position of each target object.

[0129] In the embodiment of the present application, the target recognition model is not allowed to directly output the sub-image corresponding to each target object, but the target recognition model is first used to identify the position of each target object, and then the sub-image corresponding to each target object is determined according to the position of each target object. Compared with directly outputting the final sub-image corresponding to each target object, only outputting the position of each target object reduces the processing steps of the target recognition model, and to a certain extent improves the accuracy of the target recognition model in identifying each target object.

[0130] In order to further improve the accuracy of target object recognition, based on the above embodiments, in the embodiment of the present application, after determining the target sub-image in the image to be recognized and before inputting the target sub-image into the target recognition large model, the method further includes:

[0131] Expanding the coverage area of ​​the target sub-image in the image to be identified by a set multiple to obtain a complete sub-image, so as to ensure that the complete sub-image includes complete images of all candidate targets;

[0132] The target sub-image is updated using the complete sub-image.

[0133] Because the position of each candidate target identified by the small target recognition model and the large target recognition model may not completely identify the candidate target. In other words, the candidate target included in the image area corresponding to the position of a certain candidate target in the image to be recognized is not a complete object. Then, the object included in the target sub-image determined based on the position corresponding to the incomplete object is also incomplete. Therefore, in an embodiment of the present application, after determining the target sub-image, before inputting the target sub-image into the large target recognition model, the coverage area of ​​the target sub-image in the image to be recognized can also be expanded by a set multiple to obtain a complete sub-image, which includes the complete image of all candidate targets. For example, the coverage area of ​​the target sub-image in the image to be recognized is expanded outward by α times, where α≥1. Specifically, it can be ensured that the corresponding center point of the target sub-image in the image to be recognized remains unchanged, and the length and width of the target sub-image are changed to α times of the original.

[0134] After the complete sub-image is obtained, the target sub-image may be updated using the complete sub-image.

[0135] In an embodiment of the present application, after the target sub-image is determined, the coverage area of ​​the target sub-image in the image to be identified is expanded by a set multiple to obtain a complete sub-image of the complete image including all targets, and then the target sub-image is updated using the complete sub-image to ensure that the target object is identified based on the complete image including all candidate targets in the subsequent target object recognition process, thereby improving the accuracy of target object recognition.

[0136] In order to further improve the accuracy of target object recognition, based on the above embodiments, in the embodiment of the present application, the target recognition large model includes an input layer, a high-resolution visual encoder, a low-resolution visual encoder, a visual adapter, a text encoder and a large language model;

[0137] The process of the target recognition model recognizing any received image includes:

[0138] The input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, and inputs the image into the target visual encoder for feature extraction to obtain a first feature matrix;

[0139] Inputting the first feature matrix into the visual adapter for feature mapping to obtain a second feature matrix;

[0140] Inputting the prompt text received when receiving the image into the text encoder to obtain a text feature matrix;

[0141] The second feature matrix and the text feature matrix are input into the large language model to obtain description information for describing the target included in the image.

[0142] Since the characteristics of feature distribution of images with different resolutions are different, in order to further improve the accuracy of target object recognition, in an embodiment of the present application, the image encoder of the target recognition large model can adopt a hybrid visual encoder, that is, a mixture of high-resolution and low-resolution visual encoders. Specifically, the target recognition large model may include an input layer, a high-resolution visual encoder, a low-resolution visual encoder, a visual adapter, a text encoder, and a large language model. In an embodiment of the present application, a high-resolution visual encoder SAM-B can be used to process global features, and a 1024*1024 high-resolution image can be used as a high-resolution visual encoder input, and another branch uses a low-resolution visual encoder SigLIP-L to process local features, and a 384*384 low-resolution image is used as a low-resolution visual encoder input, and the two branches are used in a mixed manner.

[0143] In an embodiment of the present application, when the target recognition large model recognizes any received image, the input layer can determine the target visual encoder among the high-resolution visual encoder and the low-resolution visual encoder based on the resolution of the received image and a preset resolution threshold.

[0144] Specifically, the input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, including:

[0145] The input layer determines whether the resolution of the image is greater than a preset resolution threshold;

[0146] If yes, determining the high-resolution visual encoder as the target visual encoder;

[0147] If not, the low-resolution visual encoder is determined as the target visual encoder.

[0148] When determining the target visual encoder, the input layer can determine whether the resolution of the received image is greater than a preset resolution threshold. Those skilled in the art can configure the preset resolution threshold as needed, for example, the preset resolution threshold is 768*768.

[0149] If it is determined that the resolution of the image is greater than the preset resolution threshold, the high-resolution visual encoder is determined as the target visual encoder. For example, if the length of the resolution of the image is greater than 768 and the width is greater than 768, the high-resolution visual encoder can be determined as the target visual encoder. Alternatively, if the length of the resolution of the image is greater than 768 or the width is greater than 768, the high-resolution visual encoder can be determined as the target visual encoder.

[0150] Otherwise, the low-resolution visual encoder is determined as the target visual encoder. For example, if the length of the resolution of the received image is not greater than 768, or the width is not greater than 768, the low-resolution visual encoder can be determined as the target visual encoder.

[0151] After the input layer determines the target encoder, the image can be input into the target visual encoder for feature extraction to obtain a first feature matrix. How the visual encoder processes the received image is a prior art, and the embodiments of the present application will not be described in detail.

[0152] After obtaining the first feature matrix, the first feature matrix can be input into the visual adapter for feature mapping to obtain a second feature matrix. That is to say, the visual adapter performs feature mapping on the first feature matrix according to the input requirements of the large language model to obtain a second feature matrix. In an embodiment of the present application, one visual adapter may be included, or multiple visual adapters may be included. When the target recognition large model includes only one visual adapter, the visual adapter needs to process the first feature matrix output by each visual encoder. In an embodiment of the present application, a visual adapter may also be configured for each visual encoder. For example, a high-resolution visual encoder is correspondingly configured with a visual adapter, and a low-resolution visual encoder is correspondingly configured with a visual adapter. The unused visual adapter is only used to process the first feature matrix output by the corresponding visual encoder.

[0153] When processing an image using a large model, it is necessary not only to input the image but also to input the corresponding prompt text, so that the large model can process the received image based on the requirements of the prompt text. Therefore, in the embodiment of the present application, the input layer can also input the prompt text received when receiving the image into the text encoder, and the text encoder processes the received prompt text to obtain a text feature matrix.

[0154] After obtaining the text feature matrix and the second feature matrix, the second feature matrix and the text feature matrix can be input into a large language model, and the large language model processes the second feature matrix based on the text feature matrix to obtain description information for describing the target included in the image.

[0155] In an embodiment of the present application, when the large model performs inference, the corresponding visual encoder is adaptively selected according to the resolution size of the received image, so that different visual encoders focus on processing images of different resolution sizes, thereby improving the accuracy of feature extraction by the visual encoder, and thereby improving the ability of the large target recognition model to recognize the received image.

[0156] Figure 6 A schematic diagram of the structure of a target recognition large model provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the target recognition large model includes a hybrid visual encoding, two visual adapters, a text encoder and a large language model. Among them, the hybrid visual encoder includes SAM-B and SigLIP-L, SAM-B is a high-resolution visual encoder, and SigLIP-L is a low-resolution visual encoder. During the reasoning process of the target recognition large model, the input layer of the target recognition large model can adaptively select an image encoder according to the size of the received image. If the resolution of the image is lower than the specified resolution size, the SigLIP-L encoder can be used for image encoding processing. If the resolution of the image is not lower than the specified resolution size, the SAM-B encoder is used for encoding processing. In other words, for the image to be identified and the target sub-image, the SAM-B image encoder is used for processing, and for each sub-image of the target object cut out, the SigLIP-L image encoder is used for processing. For different input images, the visual encoder and its corresponding visual adapter are adaptively selected for processing. After obtaining the first feature matrix output by SAM-B or SigLIP-L, the first feature matrix can be input into the corresponding visual adapter for feature mapping to obtain a second feature matrix. At the same time, the input layer inputs the prompt text into the text encoder to obtain a text feature matrix. The second feature matrix and the text feature matrix are input into the large language model to obtain description information for describing the target object included in the image.

[0157] In order to further improve the accuracy of target object recognition, based on the above embodiments, in the embodiment of the present application, the target recognition large model is obtained by fine-tuning training based on the instruction data set;

[0158] The construction process of the instruction data set includes:

[0159] Get a sample image;

[0160] Recognize the sample image based on the pre-trained large model to obtain a global description;

[0161] Based on the small target recognition model, identifying the position information of the target included in the sample image;

[0162] Inputting the sub-image corresponding to the position information in the sample image into a universal recognition model to obtain a description of the category to which the target object corresponding to the sub-image belongs;

[0163] Input the global description, the location information and the corresponding category description into the pre-trained large model to obtain answer information corresponding to each preset question;

[0164] Each preset question and the corresponding answer information are obtained and stored in the instruction data set.

[0165] In order to further improve the accuracy of target object recognition, the target recognition large model described in the above embodiments is a multimodal large model after fine-tuning training. Based on the specified task scenario of the business, an instruction data set is constructed in the scenario, and then a multimodal vertical domain large model is obtained by fine-tuning LORA (Low-Rank Adaptation), which can be used for target object recognition described in the above embodiments. The construction process of the instruction data set is described below.

[0166] In the embodiment of the present application, a detailed instruction data set can be constructed by multi-level and multi-scale fusion. First, a sample image can be obtained, and the sample image can be collected in advance by relevant staff.

[0167] After obtaining the sample image, the sample image can be identified based on the pre-trained large model to obtain a global description. The pre-trained large model can be ChatGPT4o, of course, it can also be other large models, and technicians in this field can configure it as needed. That is to say, after obtaining the sample image, a global description is generated based on the pre-trained large model. In order to ensure the quality of the instruction data set construction, after obtaining the global description, the global description can be manually corrected.

[0168] In an embodiment of the present application, other models can also be used to determine the target objects included in the sample image, and the category to which each target object belongs. Specifically, based on the target recognition small model, the position information of the target included in the sample image can be identified, and the sub-image corresponding to the position information in the sample image is input into the universal recognition model to obtain the category description to which the target object corresponding to the sub-image belongs. Among them, the universal recognition model can be a recognition model obtained by training based on universal data. Exemplarily, the universal recognition model can be a BLIP-2 model. Similarly, in order to ensure the quality of the construction of the instruction data set, after obtaining the position information and category description of each target, manual correction can be performed.

[0169] Specifically, the target region box can be generated based on the trained small model, and manual corrections can be made to add missed region boxes, delete misdetected region boxes, send the confirmed region boxes to BLIP-2 to generate category names, and then manually correct the integrated information again.

[0170] After obtaining the global description, location information and corresponding category description, the global description, location information and corresponding category description can be input into the pre-trained large model to obtain the answer information corresponding to each preset question. Among them, the pre-trained large model can be ChatGPT4o, of course, it can also be other pre-trained large models, and technicians in this field can configure it as needed. After obtaining each preset question and the corresponding answer information, it can be saved in the instruction data set.

[0171] In a possible implementation, after obtaining the position information of each target, a sub-image corresponding to each target may be cut out from the sample image, and each sub-image and the corresponding position information and category description may be saved in correspondence to construct small-scale instruction data.

[0172] Exemplarily, the constructed instruction data set may be as follows:

[0173] {

[0174] “question”: “How many tools are there in the picture?”

[0175] "answer": "There are six tools in the picture, namely, a dragon saw, a sprayer, a repair hammer, a pile driver, a small shovel, and a portable tool light."

[0176] “question”: “What tools are in the picture?”

[0177] "answer": "There are six tools in the picture: a dragon saw, a sprayer, a repair hammer, a pile driver, a small shovel, and a portable tool light."

[0178] “question”: “Please box each tool in the picture.”

[0179] "answer": <ref> Dragon Saw< / ref> <box> (333,159),(947,1011)< / box> ,

[0180] <ref> Maintenance hammer< / ref> <box> (756,62),(1045,211)< / box> ,

[0181] <ref> sprayer< / ref> <box> (671,127),(951,459)< / box> ,

[0182] <ref> Pile Driver< / ref> <box> (1079,235),(1434,524)< / box> ,

[0183] <ref> Small shovel< / ref> <box> (958,525),(1047,612)< / box> ,

[0184] <ref> Portable Utility Light Sprayer< / ref> <box> (1027,571),(1443,947)< / box> . "

[0185] "question": "In the picture <box> (671,127),(951,459)< / box> What tool is in the box?

[0186] “answer”: “It’s a blue spray can.”

[0187] "question": "Please frame the sprayer in the picture."

[0188] "answer": <ref> sprayer< / ref> <box> (671,127),(951,459)< / box> . "

[0189] "question": "What tool is in the picture?" (The corresponding picture here is the small cutout picture, that is, the sub-image corresponding to the target object)

[0190] "answer": "Portable tool light."

[0191] }.

[0192] The target object recognition process based on the large model is described below with reference to a specific embodiment. Figure 7 A schematic diagram of a target object recognition process based on a large model provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, an input image to be identified is received, and identification is performed simultaneously based on a small target identification model and a large target identification model to obtain the position of each candidate target, and based on the positions of the identified candidate targets, the target sub-image in the image to be identified is determined.

[0193] The target sub-image is input into the target recognition model for secondary recognition.

[0194] Based on the recognition results of the target recognition large model and the target recognition small model, the sub-image corresponding to each target object is determined. Specifically, through the above target object recognition strategy, three recognition results can be obtained:

[0195] (1) Candidate targets identified by the small target recognition model;

[0196] (2) The target recognition model processes the image to be recognized to obtain candidate targets;

[0197] (3) The target object is obtained by processing the target sub-image with the target recognition large model.

[0198] Based on the preset screening strategy, the target object can be screened in the above three recognition results. In the embodiment of the present application, the preset screening strategy is as follows:

[0199] ① For candidate targets identified by the target recognition model, when the confidence of a candidate target is greater than a preset threshold, the candidate target can be directly retained. In other words, the candidate target is determined as the target object. When the confidence of a candidate target is not greater than the preset threshold, if one of the recognition results of (2) and (3) above also recognizes the candidate target, the candidate target can be retained. If none of the recognition results of (2) and (3) above recognizes the candidate target, the candidate target can be deleted.

[0200] ② For candidate targets that are only identified by the above-mentioned identification method (2), the candidate targets are deleted.

[0201] ③ For a target that is identified only by the above identification method (3), the target is directly determined as the target object.

[0202] ④ All candidate targets identified by the above identification methods (2) and (3) are retained.

[0203] Specifically, the process of determining the target object based on the above preset screening strategy may include the following steps:

[0204] ① Sort the recognition results of the target recognition model from high to low according to the confidence level, and divide them into two level sequences: high confidence sequence - confidence level greater than the threshold c 1 ; Low confidence sequence - confidence is lower than the threshold c 1 Among them, c 1 =0.8.

[0205] ②Match the recognition results of the target recognition model with the candidate targets in the recognition results of the second and third recognition methods respectively. If a candidate target recognized by the target recognition model belongs to a high confidence sequence, it will be retained regardless of whether it is matched successfully. At the same time, the candidate targets that are successfully matched in the recognition results of the second and third recognition methods are marked so that the candidate targets will not be matched repeatedly in the future.

[0206] ③ If a candidate target identified by the target recognition model belongs to a low confidence sequence, it is determined whether there is a matching candidate target in the recognition results of the second and third recognition methods. If so, the candidate target is retained. At the same time, the successfully matched candidate targets in the recognition results of the second and third recognition methods are marked so that the candidate target will not be matched repeatedly in the future.

[0207] ④ Match the unmarked candidate targets in the recognition result of the second recognition method with the unmarked candidate targets in the recognition result of the third recognition method. If the match is successful, the corresponding candidate targets are retained. At the same time, the candidate targets that are successfully matched in the recognition result of the third recognition method are marked. If the match is unsuccessful, the corresponding candidate targets are deleted.

[0208] ⑤Retain the unmarked candidate targets in the recognition results of the third recognition method.

[0209] In the above manner, the target objects can be screened out, and after the target objects are determined, the sub-images corresponding to each target object can be determined. After the sub-images are obtained, the target recognition model can be used to recognize each sub-image, determine the category to which the target object in each sub-image belongs, and thus determine the number of target objects included in the image to be recognized.

[0210] In a possible implementation, in order to improve the efficiency of target object recognition, the target recognition large model can output the category to which each target (including candidate targets and target objects) belongs when processing the image to be recognized and the target sub-image, without the need to subsequently call the target recognition large model for secondary recognition of the category to which it belongs. In an embodiment of the present application, among the target objects finally determined, only the sub-image corresponding to the target object recognized by the target small model can be input into the target recognition large model for recognition of the category to which it belongs.

[0211] In an embodiment of the present application, when each sub-image is processed using the target recognition large model, not only the category name of each target object can be obtained, but also a detailed description of each target object can be obtained using the target recognition large model.

[0212] The present invention provides a method for identifying a target object based on a large model, which has the following advantages:

[0213] (1) Combination of large and small models. Only the known classes can be detected by the small model, but the unknown classes cannot be detected. The large model has the ability to recognize the unknown classes, but the detection ability of the large model is slightly weaker than that of the small model. Based on this, the combination of large and small models can detect every target in the image;

[0214] (2) Large model category name recognition. The large model has the ability to recognize a wide range of targets. For targets that the small model cannot recognize, the model category can be obtained through the large model. In addition, the large model can not only recognize its category, but also recognize its related attributes;

[0215] (3) Input key areas into the large model for recognition to improve recognition accuracy;

[0216] (4) A method for constructing a multi-level and multi-scale instruction dataset, by fine-tuning the instruction dataset constructed in this way, to improve the recognition effect of the large model on this type of target;

[0217] (5) Hybrid visual encoder structure: Adaptively select the encoder structure based on the input image to further improve the recognition effect of the large model.

[0218] Using the recognition method proposed in this application, it is possible to obtain results that are better than those obtained by simply using a small model, which meets the characteristics of reliability in the trustworthy characteristic; using the recognition method proposed in this application is suitable for all needs for target quantity and category recognition, which meets the characteristics of generalizability in the trustworthy characteristic; the recognition method designed in this application can maintain characteristics that are better than the results of simply using a small model under any circumstances, which meets the characteristics of robustness in the trustworthy characteristic.

[0219] Based on the same inventive concept, the embodiment of the present application also provides a target object recognition device based on a large model. Figure 8 A schematic diagram of the structure of a target object recognition device based on a large model provided in an embodiment of the present application, the device comprises:

[0220] The determination module 801 is used to identify the position of the candidate target in the image to be identified based on the target recognition large model and the target recognition small model respectively, determine the target sub-image in the image to be identified, and the target sub-image includes each candidate target; input the target sub-image into the target recognition large model to obtain a sub-image set, and the sub-image set includes a sub-image corresponding to each target object;

[0221] The recognition module 802 is used to input each sub-image and the first prompt text into the target recognition model to obtain the description text of the target object included in each sub-image; and determine the category to which each target object belongs based on the description text of each target object.

[0222] In a possible implementation, the determination module 801 is specifically used to input the image to be identified into the small target recognition model to identify the position of each first candidate target; input the image to be identified and the second prompt text into the large target recognition model to identify the position of each second candidate target; and determine the target sub-image in the image to be identified based on the position of each first candidate target and the position of each second candidate target.

[0223] In one possible implementation, the determination module 801 is specifically used to input the image to be identified into the target recognition small model to identify the position and confidence of each first candidate target; if the confidence corresponding to any first candidate target is greater than a preset threshold, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if not, the sub-image corresponding to the first candidate target is added to the sub-image set.

[0224] In one possible implementation, the determination module 801 is specifically used to input the image to be identified into the target recognition small model to identify the position and confidence of each first candidate target; if the confidence corresponding to any first candidate target is not greater than the preset threshold, then determine whether the first candidate target exists in each second candidate target based on the degree of overlap between the position of the first candidate target and the position of each second candidate target; if so, determine whether there is a sub-image corresponding to the first candidate target in the sub-image set; if not, add the sub-image corresponding to the first candidate target to the sub-image set.

[0225] In a possible implementation, the determination module 801 is specifically used to input the target sub-image and the third prompt text into the target recognition model to identify the position of each target object; and determine the sub-image corresponding to each target object based on the position of each target object.

[0226] In a possible implementation, the determination module 801 is further used to expand the coverage area of ​​the target sub-image in the image to be identified by a set multiple to obtain a complete sub-image, so as to ensure that the complete sub-image includes a complete image of all candidate targets; and use the complete sub-image to update the target sub-image.

[0227] In one possible implementation, the target recognition large model includes an input layer, a high-resolution visual encoder, a low-resolution visual encoder, a visual adapter, a text encoder, and a large language model;

[0228] The recognition module 802 is specifically used for the input layer to determine the target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, and input the image into the target visual encoder for feature extraction to obtain a first feature matrix; input the first feature matrix into the visual adapter for feature mapping to obtain a second feature matrix; input the prompt text received when receiving the image into the text encoder to obtain a text feature matrix; input the second feature matrix and the text feature matrix into the large language model to obtain description information for describing the target included in the image.

[0229] In one possible implementation, the recognition module 802 is specifically used for the input layer to determine whether the resolution of the image is greater than a preset resolution threshold; if so, the high-resolution visual encoder is determined as the target visual encoder; if not, the low-resolution visual encoder is determined as the target visual encoder.

[0230] In a possible implementation, the target recognition large model is obtained by fine-tuning training based on the instruction data set; the device also includes:

[0231] Construction module 803 is used to obtain a sample image; recognize the sample image based on the pre-trained large model to obtain a global description; recognize the location information of the target included in the sample image based on the target recognition small model; input the sub-image corresponding to the location information in the sample image into the general recognition model to obtain the category description to which the target object corresponding to the sub-image belongs; input the global description, the location information and the corresponding category description into the pre-trained large model to obtain the answer information corresponding to each preset question; each preset question and the corresponding answer information are obtained and saved in the instruction data set.

[0232] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, it includes: a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904;

[0233] The memory 903 stores a computer program. When the program is executed by the processor 901, the processor 901 performs the following steps:

[0234] Recognize the positions of the candidate targets in the image to be identified based on the target recognition large model and the target recognition small model respectively, and determine the target sub-image in the image to be identified, wherein each candidate target is included in the target sub-image;

[0235] The target sub-image is input into the target recognition large model to obtain a sub-image set, wherein the sub-image set includes a sub-image corresponding to each target object; each sub-image and a first prompt text are input into the target recognition large model to obtain a description text of the target object included in each sub-image; and the category to which each target object belongs is determined based on the description text of each target object.

[0236] In a possible implementation manner, determining the target sub-image in the image to be recognized based on the positions of the candidate targets recognized by the target recognition large model and the target recognition small model in the image to be recognized, respectively, includes:

[0237] Inputting the image to be recognized into the target recognition small model to recognize the position of each first candidate target;

[0238] Inputting the image to be recognized and the second prompt text into the target recognition large model to recognize the position of each second candidate target;

[0239] According to the position of each first candidate target and the position of each second candidate target, a target sub-image in the image to be identified is determined.

[0240] In a possible implementation, inputting the image to be recognized into the small target recognition model to recognize the position of each first candidate target includes:

[0241] Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target;

[0242] Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes:

[0243] If the confidence corresponding to any first candidate target is greater than a preset threshold, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if not, the sub-image corresponding to the first candidate target is added to the sub-image set.

[0244] In a possible implementation, inputting the image to be recognized into the small target recognition model to recognize the position of each first candidate target includes:

[0245] Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target;

[0246] Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes:

[0247] If the confidence level corresponding to any first candidate target is not greater than the preset threshold, determining whether the first candidate target exists in each of the second candidate targets according to the degree of overlap between the position of the first candidate target and the position of each of the second candidate targets;

[0248] If yes, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if no, the sub-image corresponding to the first candidate target is added to the sub-image set.

[0249] In a possible implementation manner, inputting the target sub-image into the target recognition large model to determine the sub-image corresponding to each target object includes:

[0250] Inputting the target sub-image and the third prompt text into the target recognition large model to recognize the position of each target object;

[0251] According to the position of each target object, a sub-image corresponding to each target object is determined.

[0252] In a possible implementation, after determining the target sub-image in the image to be recognized and before inputting the target sub-image into the target recognition large model, the method further includes:

[0253] Expanding the coverage area of ​​the target sub-image in the image to be identified by a set multiple to obtain a complete sub-image, so as to ensure that the complete sub-image includes complete images of all candidate targets;

[0254] The target sub-image is updated using the complete sub-image.

[0255] In one possible implementation, the target recognition large model includes an input layer, a high-resolution visual encoder, a low-resolution visual encoder, a visual adapter, a text encoder, and a large language model;

[0256] The process of the target recognition model recognizing any received image includes:

[0257] The input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, and inputs the image into the target visual encoder for feature extraction to obtain a first feature matrix;

[0258] Inputting the first feature matrix into the visual adapter for feature mapping to obtain a second feature matrix;

[0259] Inputting the prompt text received when receiving the image into the text encoder to obtain a text feature matrix;

[0260] The second feature matrix and the text feature matrix are input into the large language model to obtain description information for describing the target included in the image.

[0261] In a possible implementation, the input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, including:

[0262] The input layer determines whether the resolution of the image is greater than a preset resolution threshold;

[0263] If yes, determining the high-resolution visual encoder as the target visual encoder;

[0264] If not, the low-resolution visual encoder is determined as the target visual encoder.

[0265] In a possible implementation, the target recognition large model is obtained by fine-tuning training based on the instruction data set;

[0266] The construction process of the instruction data set includes:

[0267] Get a sample image;

[0268] Recognize the sample image based on the pre-trained large model to obtain a global description;

[0269] Based on the small target recognition model, identifying the position information of the target included in the sample image;

[0270] Inputting the sub-image corresponding to the position information in the sample image into a universal recognition model to obtain a description of the category to which the target object corresponding to the sub-image belongs;

[0271] Input the global description, the location information and the corresponding category description into the pre-trained large model to obtain answer information corresponding to each preset question;

[0272] Each preset question and the corresponding answer information are obtained and stored in the instruction data set.

[0273] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 902 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0274] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0275] Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large model-based target object recognition method described in the above embodiments.

[0276] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0277] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0278] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0279] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0280] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A target object recognition method based on a large model, characterized in that: The method comprises: Recognize the positions of the candidate targets in the image to be identified based on the target recognition large model and the target recognition small model respectively, and determine the target sub-image in the image to be identified, wherein each candidate target is included in the target sub-image; The target sub-image is input into the target recognition large model to obtain a sub-image set, wherein the sub-image set includes a sub-image corresponding to each target object; each sub-image and a first prompt text are input into the target recognition large model to obtain a description text of the target object included in each sub-image; and the category to which each target object belongs is determined based on the description text of each target object.

2. The method according to claim 1, characterized in that The determining of the target sub-image in the image to be identified based on the positions of the candidate targets identified in the image to be identified by the target identification large model and the target identification small model respectively comprises: Inputting the image to be recognized into the target recognition small model to recognize the position of each first candidate target; Inputting the image to be recognized and the second prompt text into the target recognition large model to recognize the position of each second candidate target; According to the position of each first candidate target and the position of each second candidate target, a target sub-image in the image to be identified is determined.

3. The method according to claim 2, characterized in that Inputting the image to be recognized into the target recognition model to recognize the position of each first candidate target includes: Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target; Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes: If the confidence corresponding to any first candidate target is greater than a preset threshold, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if not, the sub-image corresponding to the first candidate target is added to the sub-image set.

4. The method according to claim 2, characterized in that: Inputting the image to be recognized into the target recognition model to recognize the position of each first candidate target includes: Inputting the image to be recognized into the target recognition small model to recognize the position and confidence of each first candidate target; Before inputting each sub-image and the first prompt text into the target recognition large model, the method further includes: If the confidence level corresponding to any first candidate target is not greater than the preset threshold, determining whether the first candidate target exists in each of the second candidate targets according to the degree of overlap between the position of the first candidate target and the position of each of the second candidate targets; If yes, it is determined whether there is a sub-image corresponding to the first candidate target in the sub-image set; if no, the sub-image corresponding to the first candidate target is added to the sub-image set.

5. The method according to claim 1, characterized in that The step of inputting the target sub-image into the target recognition model to determine the sub-image corresponding to each target object includes: Inputting the target sub-image and the third prompt text into the target recognition large model to recognize the position of each target object; According to the position of each target object, a sub-image corresponding to each target object is determined.

6. The method according to claim 1, characterized in that After determining the target sub-image in the image to be recognized and before inputting the target sub-image into the target recognition large model, the method further includes: Expanding the coverage area of ​​the target sub-image in the image to be identified by a set multiple to obtain a complete sub-image, so as to ensure that the complete sub-image includes complete images of all candidate targets; The target sub-image is updated using the complete sub-image.

7. The method according to claim 1 or 2, characterized in that: The target recognition large model includes an input layer, a high-resolution visual encoder, a low-resolution visual encoder, a visual adapter, a text encoder and a large language model; The process of the target recognition model recognizing any received image includes: The input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, and inputs the image into the target visual encoder for feature extraction to obtain a first feature matrix; Inputting the first feature matrix into the visual adapter for feature mapping to obtain a second feature matrix; Inputting the prompt text received when receiving the image into the text encoder to obtain a text feature matrix; The second feature matrix and the text feature matrix are input into the large language model to obtain description information for describing the target included in the image.

8. The method according to claim 7, characterized in that The input layer determines a target visual encoder in the high-resolution visual encoder and the low-resolution visual encoder according to the resolution of the received image and a preset resolution threshold, including: The input layer determines whether the resolution of the image is greater than a preset resolution threshold; If yes, determining the high-resolution visual encoder as the target visual encoder; If not, the low-resolution visual encoder is determined as the target visual encoder.

9. The method according to claim 1, characterized in that: The target recognition large model is obtained by fine-tuning training based on the instruction data set; The construction process of the instruction data set includes: Get a sample image; Recognize the sample image based on the pre-trained large model to obtain a global description; Based on the small target recognition model, identifying the position information of the target included in the sample image; Inputting the sub-image corresponding to the position information in the sample image into a universal recognition model to obtain a description of the category to which the target object corresponding to the sub-image belongs; Input the global description, the location information and the corresponding category description into the pre-trained large model to obtain answer information corresponding to each preset question; Each preset question and the corresponding answer information are obtained and stored in the instruction data set.

10. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the target object recognition method based on a large model as claimed in any one of claims 1 to 9 when executing a computer program stored in a memory.

Images