Target positioning method and related equipment

By using multimodal large model and area description expert model to process complex natural language descriptions in images, the accurate bounding box positioning data set is generated, which solves the problem of low target positioning accuracy in the prior art, and realizes the complex description Accurate positioning and bounding box detection.

CN118568289BActive Publication Date: 2024-11-26HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411053533.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-02
Publication Date
2024-11-26
Estimated Expiration
2044-08-02

AI Technical Summary

Technical Problem

The prior art has the problem of low accuracy in target positioning of complex natural language descriptions in images, especially the inability to effectively integrate the interaction relationships in the description, resulting in false positive examples and inaccurate detection.

Method used

A target positioning method is proposed. By obtaining the pending image and text prompt information, the bounding box positioning is used to use a preset multimodal model to perform bounding box positioning, and the basic multimodal model is trained through the preset bounding box positioning data set. Combining the regional description expert model and the reference matching expert model, adjusting and fusing triple information to generate accurate bounding box positioning data sets to avoid the occurrence of false positive examples.

Benefits of technology

It improves the accuracy of positioning of complex description targets, reduces the occurrence of false positive examples, can more accurately detect whether there are described targets in the image, and provides corresponding bounding boxes to meet practical application requirements.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118568289B_ABST
    Figure CN118568289B_ABST
Patent Text Reader

Abstract

The present application discloses a target positioning method and related equipment, the method comprising: obtaining an image to be processed and first text prompt information, the first text prompt information being description information of a target to be retrieved from the image to be processed; based on the image to be processed and the first text prompt information, positioning a bounding box of the target in the image to be processed by a preset multimodal large model to obtain a positioning result; wherein the preset multimodal large model is obtained by training a basic multimodal large model based on a preset bounding box positioning data set, each data in the preset bounding box positioning data set comprises a first data set and a bounding box label corresponding to the first data set, the bounding box label and the first data set are in an N-to-1 relationship, N is greater than or equal to 0, the first data set comprises a first sample image and a second text prompt information describing the target to be retrieved; the present application realizes the improvement of the accuracy of positioning targets of arbitrarily complex descriptions.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a target positioning method and related equipment. Background Art

[0002] Detecting objects described by natural language in images has always been an important research area. The current mainstream technologies for detecting objects in images include open-vocabulary object detection (OVD) and referring expression comprehension (REC). Among them, open-vocabulary object detection can detect any category name description, but cannot detect objects referred to by more complex natural language descriptions. Referring expression comprehension can detect objects referred to by natural language descriptions in images, and this technology assumes that there must be and only one object referred to by the current description in the image, but in actual scenarios, this assumption is difficult to hold. For example, when it is necessary to detect the object described by "a person without a helmet", open-vocabulary object detection technology can detect "helmet" and "person", but cannot integrate the interactive relationship between the two "not wearing", resulting in inaccurate object detection; referring expression comprehension technology can fully understand "a person without a helmet", but can only detect and force the detection of a target bounding box, which is easy to cause false positives.

[0003] That is, in actual scenarios, in images, there is a problem of low accuracy in locating objects with arbitrarily complex descriptions. Summary of the invention

[0004] The main purpose of this application is to provide a target positioning method and related equipment, aiming to solve the technical problem of low accuracy in positioning targets of arbitrarily complex descriptions in images.

[0005] To achieve the above objectives, the present application proposes a target positioning method, which includes:

[0006] Acquire an image to be processed and first text prompt information, where the first text prompt information is description information of a target to be retrieved from the image to be processed;

[0007] Based on the image to be processed and the first text prompt information, a bounding box of the target in the image to be processed is located by using a preset multimodal large model, and a positioning result of the bounding box is used as a positioning result of the target;

[0008] Among them, the preset multimodal large model is obtained by training the basic multimodal large model based on the preset bounding box positioning data set, each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, the bounding box label and the first data set are in an N-to-1 relationship, N is greater than or equal to 0, and the first data set includes a first sample image and a second text prompt information describing the target to be retrieved.

[0009] In one embodiment, before the step of locating the bounding box of the target in the image to be processed by using a preset multimodal large model based on the image to be processed and the first text prompt information, the method includes:

[0010] Obtaining a first sample image and bounding box label information of an object in the first sample image;

[0011] Based on the first sample image and the bounding box label information of the target in the first sample image, the bounding box of the target in the image to be processed is inferred by a preset region description expert model to obtain second text prompt information, wherein the preset region description expert model is a pre-trained model;

[0012] Determine a preset bounding box positioning data set based on a first data set consisting of the second text prompt information and the first sample image, and the bounding box label;

[0013] Based on the preset bounding box positioning data set, the basic multimodal large model is trained to obtain the preset multimodal large model.

[0014] In one embodiment, the step of determining a preset bounding box positioning data set based on a first data set consisting of the second text prompt information and the first sample image, and the bounding box label, comprises:

[0015] Adjusting the second text prompt information into triple information described in triple form, wherein the triple information includes subjects and interaction relationships between subjects, wherein one of the subjects in the triple information is a target;

[0016] A preset bounding box positioning data set is determined based on the triplet information, the bounding box label and the first sample image.

[0017] In one embodiment, the step of determining a preset bounding box positioning data set based on the triple information, the bounding box label and the first sample image includes:

[0018] Modifying the subjects and / or the interaction relationship between the subjects in the triple information to obtain first negative sample data, wherein the data in the triple information without adjusting the subjects and the interaction relationship between the subjects is used as the first positive sample data;

[0019] A preset bounding box positioning data set is determined based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image.

[0020] In one embodiment, the step of determining a preset bounding box positioning data set based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image includes:

[0021] The first negative sample data with a preset first matching label, the corresponding bounding box label, and the corresponding first sample image are used as a first combined data set, and the first positive sample data with a preset second matching label, the corresponding bounding box label, and the corresponding first sample image are used as a second combined data set;

[0022] Training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model;

[0023] The first permutation and combination data set is input into the target reference expression matching expert model, and two sets of data in the permutation and combination data set are cross-matched one-to-one based on the target reference expression matching expert model to obtain a first matching result of whether the two sets of data match; wherein the first permutation and combination data set includes a set of data consisting of a preset second sample image and a bounding box of a target corresponding to the preset second sample image, and another set of data consisting of a second negative sample data or a second positive sample data, which are permuted and combined;

[0024] A preset bounding box positioning dataset is determined based on the first matching result, the first combined dataset, and the second combined dataset.

[0025] In one embodiment, the step of determining a preset bounding box positioning dataset based on the first matching result, the first combined dataset, and the second combined dataset includes:

[0026] Based on the first matching result, determine a pseudo label corresponding to each piece of data in the data set after the first arrangement and combination, and a first confidence level corresponding to the corresponding pseudo label;

[0027] A first target data set having a corresponding first confidence higher than a preset confidence threshold is screened out from the first permuted and combined data set, and a preset bounding box positioning data set is determined as the preset bounding box positioning data set by using the first target data set, the first combined data set, and the second combined data set.

[0028] In one embodiment, the step of training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model includes:

[0029] Based on the first combined data set and the second combined data set, predicting whether the corresponding text prompt information of each combined data set matches the corresponding first sample image and its bounding box label through the basic reference expression matching expert model to obtain a prediction result;

[0030] Based on the prediction result and the corresponding first matching label or second matching label, the basic reference expression matching expert model is trained to obtain an initial reference expression matching expert model;

[0031] Inputting the incremental training data into the initial reference expression matching expert model, and performing one-to-one cross matching on two groups of data in the incremental training data based on the initial reference expression matching expert model to obtain a third matching result; wherein the incremental training data includes a group of data consisting of a preset third sample image and a bounding box of a target corresponding to the preset third sample image, and another group of data consisting of a third negative sample data or a third positive sample data;

[0032] Determine a pseudo label corresponding to each piece of data in the incremental training data, and a second confidence level corresponding to the corresponding pseudo label;

[0033] A second target data set having a second confidence level higher than a preset confidence threshold is selected from the incremental training data, and the initial reference expression matching expert model is iteratively trained based on the second target data set until a target reference expression matching expert model is obtained.

[0034] In one embodiment, the step of training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model includes:

[0035] Cropping the first sample image to obtain a sub-image corresponding to the bounding box label;

[0036] Based on the sub-image, the complete first sample image and other data in the corresponding first combined data set and second combined data set, a basic reference expression matching expert model is trained to obtain a target reference expression matching expert model.

[0037] In addition, to achieve the above purpose, the present application also proposes a target positioning device, the device comprising:

[0038] An acquisition module, used for acquiring an image to be processed and first text prompt information, wherein the first text prompt information is description information of a target to be retrieved from the image to be processed;

[0039] A positioning module, configured to locate a bounding box of a target in the image to be processed by using a preset multimodal large model based on the image to be processed and the first text prompt information, and use the positioning result of the bounding box as the positioning result of the target;

[0040] Among them, the preset multimodal large model is obtained by training the basic multimodal large model based on the preset bounding box positioning data set, each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, the bounding box label and the first data set are in an N-to-1 relationship, N is greater than or equal to 0, and the first data set includes a first sample image and a second text prompt information describing the target to be retrieved.

[0041] In one embodiment, the target positioning device is used to achieve:

[0042] Obtaining a first sample image and bounding box label information of an object in the first sample image;

[0043] Based on the first sample image and the bounding box label information of the target in the first sample image, the bounding box of the target in the image to be processed is inferred by a preset region description expert model to obtain second text prompt information, wherein the preset region description expert model is a pre-trained model;

[0044] Determine a preset bounding box positioning data set based on a first data set consisting of the second text prompt information and the first sample image, and the bounding box label;

[0045] Based on the preset bounding box positioning data set, the basic multimodal large model is trained to obtain the preset multimodal large model.

[0046] In one embodiment, the target positioning device is used to achieve:

[0047] Adjusting the second text prompt information into triple information described in triple form, wherein the triple information includes subjects and interaction relationships between subjects, wherein one of the subjects in the triple information is a target;

[0048] A preset bounding box positioning data set is determined based on the triplet information, the bounding box label and the first sample image.

[0049] In one embodiment, the target positioning device is used to achieve:

[0050] Modifying the subjects and / or the interaction relationship between the subjects in the triple information to obtain first negative sample data, wherein the data in the triple information without adjusting the subjects and the interaction relationship between the subjects is used as the first positive sample data;

[0051] A preset bounding box positioning data set is determined based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image.

[0052] In one embodiment, the target positioning device is used to achieve:

[0053] The first negative sample data with a preset first matching label, the corresponding bounding box label, and the corresponding first sample image are used as a first combined data set, and the first positive sample data with a preset second matching label, the corresponding bounding box label, and the corresponding first sample image are used as a second combined data set;

[0054] Training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model;

[0055] The first permutation and combination data set is input into the target reference expression matching expert model, and two sets of data in the permutation and combination data set are cross-matched one-to-one based on the target reference expression matching expert model to obtain a first matching result of whether the two sets of data match; wherein the first permutation and combination data set includes a set of data consisting of a preset second sample image and a bounding box of a target corresponding to the preset second sample image, and another set of data consisting of a second negative sample data or a second positive sample data, which are permuted and combined;

[0056] A preset bounding box positioning dataset is determined based on the first matching result, the first combined dataset, and the second combined dataset.

[0057] In one embodiment, the target positioning device is used to achieve:

[0058] Based on the first matching result, determine a pseudo label corresponding to each piece of data in the data set after the first arrangement and combination, and a first confidence level corresponding to the corresponding pseudo label;

[0059] A first target data set having a corresponding first confidence higher than a preset confidence threshold is screened out from the first permuted and combined data set, and a preset bounding box positioning data set is determined as the preset bounding box positioning data set by using the first target data set, the first combined data set, and the second combined data set.

[0060] In one embodiment, the target positioning device is used to achieve:

[0061] Based on the first combined data set and the second combined data set, predicting whether the corresponding text prompt information of each combined data set matches the corresponding first sample image and its bounding box label through the basic reference expression matching expert model to obtain a prediction result;

[0062] Based on the prediction result and the corresponding first matching label or second matching label, the basic reference expression matching expert model is trained to obtain an initial reference expression matching expert model;

[0063] Inputting the incremental training data into the initial reference expression matching expert model, and performing one-to-one cross matching on two groups of data in the incremental training data based on the initial reference expression matching expert model to obtain a third matching result; wherein the incremental training data includes a group of data consisting of a preset third sample image and a bounding box of a target corresponding to the preset third sample image, and another group of data consisting of a third negative sample data or a third positive sample data;

[0064] Determine a pseudo label corresponding to each piece of data in the incremental training data, and a second confidence level corresponding to the corresponding pseudo label;

[0065] A second target data set having a second confidence level higher than a preset confidence threshold is selected from the incremental training data, and the initial reference expression matching expert model is iteratively trained based on the second target data set until a target reference expression matching expert model is obtained.

[0066] In one embodiment, the target positioning device is used to achieve:

[0067] Cropping the first sample image to obtain a sub-image corresponding to the bounding box label;

[0068] Based on the sub-image, the complete first sample image and other data in the corresponding first combined data set and second combined data set, a basic reference expression matching expert model is trained to obtain a target reference expression matching expert model.

[0069] In addition, to achieve the above objectives, the present application also proposes a target positioning device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target positioning method described above.

[0070] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the target positioning method described above are implemented.

[0071] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the target positioning method described above are implemented.

[0072] The above technical solution proposed in this application has the following technical effects:

[0073] The present application obtains an image to be processed and a first text prompt information, wherein the first text prompt information is the description information of a target to be retrieved from the image to be processed; based on the image to be processed and the first text prompt information, a bounding box of the target in the image to be processed is located by a preset multimodal large model, and the positioning result of the bounding box is used as the positioning result of the target; wherein the preset multimodal large model is obtained by training a basic multimodal large model based on a preset bounding box positioning data set, and each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, and the bounding box label has an N-to-1 relationship with the first data set, and N is greater than or equal to 0, and the first data set includes a first sample image and a second text prompt information describing the target to be retrieved. It can be understood that in the present application, in the training data of the preset multimodal large model, the bounding box The label and the first data set are in an N-to-1 relationship, and N is greater than or equal to 0, that is, N can be 0, 1 or data greater than 1, that is, the bounding box label can be 0, or only 1 or more, and the bounding box label is the bounding box label of the target, which is consistent with the corresponding target. Therefore, when detecting the target described by natural language, there is no need to assume that there must be and only one target referred to by the current description in the image, and there will be no false positive cases, which can be more in line with practical applications and can improve accuracy. In addition, in the present application, since the first data set in the training data of the preset multimodal large model includes a first sample image and a second text prompt information describing the target to be retrieved, and has a target bounding box label, the preset multimodal large model trained based on the preset bounding box positioning data set can accurately locate the target based on the image to be processed and the first text prompt information. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0075] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0076] Figure 1 A first flow chart of the target positioning method embodiment 1 of the present application is provided;

[0077] Figure 2 A second flow chart of the target positioning method according to the present invention;

[0078] Figure 3 This is a schematic diagram of the third process involved in the third embodiment of the target positioning method of the present application;

[0079] Figure 4 A schematic diagram of a first scenario provided in Example 1 of the target positioning method of the present application;

[0080] Figure 5 A schematic diagram of a second scenario provided in Example 2 of the target positioning method of the present application;

[0081] Figure 6 A schematic diagram of a third scenario provided in Embodiment 2 of the target positioning method of the present application;

[0082] Figure 7 A schematic diagram of the fourth scenario provided in Embodiment 3 of the target positioning method of the present application;

[0083] Figure 8 A schematic diagram of the device structure of the hardware operating environment involved in the target positioning method in the embodiment of the present application;

[0084] Fig. 9 This is a schematic diagram of the overall process involved in the target positioning method of the embodiment of the present application;

[0085] Fig.10 This is a schematic diagram of the module structure of the target positioning device according to an embodiment of the present application.

[0086] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0087] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0088] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0089] It should be noted that the execution subject of this embodiment may be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a target positioning device, etc. The following takes the target positioning device as an example to illustrate this embodiment and the following embodiments.

[0090] Based on this, the present application embodiment provides a target positioning method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the target positioning method of the present application.

[0091] In this embodiment, the target positioning method includes steps S10 to S20:

[0092] Step S10, obtaining an image to be processed and first text prompt information, wherein the first text prompt information is description information of a target to be retrieved from the image to be processed;

[0093] It should be noted that the image to be processed can be an image input into the target positioning device through any means. For example, the image to be processed can be an image directly transmitted to the target positioning device by a camera, the image to be processed can also be an image uploaded to the target positioning device by a client, or the image to be processed can also be an image uploaded to the target positioning device by a cloud platform, etc., without specific limitation.

[0094] Furthermore, the image to be processed may be any type of image. For example, the image to be processed may be an image captured by a camera, or the image to be processed may be an image obtained by screenshot.

[0095] Furthermore, the image to be processed may be an image in various formats, such as a gpg format, or a BMP format, etc., which is not specifically limited.

[0096] Furthermore, the image to be processed may include various types of targets, such as the image to be processed may be an image with a person as the target, or the image to be processed may be an image with an object as the target, for example, the target may be a child, or the target may be a vehicle, or the target may be clothing.

[0097] It should be emphasized that the image to be processed may not include any target. For example, if the target is a person, the image to be processed may not include the person. Or, if the target is a vehicle, the image to be processed may not include the vehicle.

[0098] In this embodiment, first text prompt information is also obtained, wherein the first text prompt information may be prompt information obtained based on the user input using natural language, or the first text prompt information may be prompt information obtained based on the user input through editing, without specific limitation.

[0099] Among them, the first text prompt information is the description information of the target that needs to be retrieved from the image to be processed. For example, the first text prompt information may be: "a person without a helmet" is detected, or the first text prompt information may be: "a middle-aged man holding a mobile phone and a black cloth bag" is detected in the corresponding area.

[0100] It should be noted that there may be no target corresponding to the first text prompt information in the image to be processed, or there may be 1 target corresponding to the first text prompt information in the image to be processed, or there may be no target corresponding to the first text prompt information in the image to be processed, or there may be multiple targets corresponding to the first text prompt information in the image to be processed. For example, if the first text prompt information is detected as "a person without a safety helmet", there is 1 person without a safety helmet in the image to be processed, or there may be no person without a safety helmet in the image to be processed, or there are multiple people without a safety helmet in the image to be processed.

[0101] Step S20, based on the image to be processed and the first text prompt information, positioning the bounding box of the target in the image to be processed by using a preset multimodal large model, and using the positioning result of the bounding box as the positioning result of the target;

[0102] Among them, the preset multimodal large model is obtained by training the basic multimodal large model based on the preset bounding box positioning data set, each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, the bounding box label and the first data set are in an N-to-1 relationship, N is greater than or equal to 0, and the first data set includes a first sample image and a second text prompt information describing the target to be retrieved.

[0103] In this embodiment, the basic concepts are first described:

[0104] Open-Vocabulary object Detection (OVD): refers to the object detection technology that can detect object detection described by any category name. The category name is usually a short category name without attribute or relationship description.

[0105] Referring Expression Comprehension (REC): refers to the object detection technology that can understand complex descriptions and detect and enforce the detection of a bounding box result.

[0106] Referring Expression Matching (REM): refers to the technology that can determine whether the content of the bounding box area matches the complex description content.

[0107] Bounding box (bbox): A description of part of the image in coordinate form.

[0108] Region Caption: refers to a detailed description of the contents of the bounding box area.

[0109] Tokenizer: refers to the technology that converts natural language into tokens that can be understood by large language models.

[0110] In this embodiment, the preset multimodal large model is obtained by training the basic multimodal large model based on the preset bounding box positioning data set. Since the model is a trained model, and each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, and the first data set includes a first sample image and a second text prompt information describing the target to be retrieved, the model can accurately predict the bounding box of the target based on the corresponding image and the prompt information describing the target to be retrieved, and then accurately locate the target. In addition, it should be emphasized that the bounding box label and the first data set are in an N-to-1 relationship. , the N is greater than or equal to 0, that is, N can be 0, 1 or data greater than 1, that is, the bounding box label can be 0, or only 1 or more, and the bounding box label is the bounding box label of the target, which is consistent with the corresponding target. Therefore, the training process in the preset multimodal large model takes into account that there is no target described in the second text prompt information in the first sample image, or there is only one target described in the second text prompt information in the first sample image, or there are multiple targets described in the second text prompt information in the first sample image, that is, the training process of the preset multimodal large model takes into account various actual situations, and therefore, the target can be accurately located to avoid false positives.

[0111] It should be noted that although there are open set target detection technology and referential expression understanding technology in the current technology, the open set target detection technology can only detect arbitrary category name descriptions, but cannot detect targets referred to by more complex natural language descriptions. The referential expression understanding technology can detect the target referred to by the natural language description in the image, and the technology assumes that there must be and only one target referred to by the current description in the image, but in actual scenarios, this assumption is difficult to establish. For example, when it is necessary to detect the target described by "a person without a helmet", the open set target detection technology can detect "helmet" and "person", but cannot integrate the interactive relationship between the two "not wearing"; the referential expression understanding technology can fully understand "a person without a helmet", but can only detect and force the detection of a target bounding box, which is easy to cause false positives. In this application, based on the trained model, descriptions of arbitrary complexity, i.e., text prompt information, can be understood, and whether the described target exists in the current image is correctly detected. If it exists, the bounding box of the corresponding target should be detected, thereby achieving accurate positioning of the target.

[0112] Specifically, before the step of positioning the bounding box of the image to be processed by using a preset multimodal large model based on the image to be processed and the first text prompt information, the method includes:

[0113] Obtaining the preset bounding box positioning data set;

[0114] Based on the preset bounding box positioning data set, the basic multimodal large model is trained to obtain the preset multimodal large model.

[0115] Or before the step of positioning the boundary box of the image to be processed by a preset multimodal large model based on the image to be processed and the first text prompt information, the following steps S01-S04 are included:

[0116] Step S01, obtaining a first sample image and bounding box label information of an object in the first sample image;

[0117] Step S02, based on the first sample image and the bounding box label information of the target in the first sample image, the bounding box of the target in the image to be processed is inferred by a preset region description expert model to obtain second text prompt information, wherein the preset region description expert model is a pre-trained model;

[0118] It can be understood that in this embodiment, the preset multimodal large model is obtained by training the basic multimodal large model based on the preset bounding box positioning data set, and in this application, the process of how to obtain the preset bounding box positioning data set is also explained.

[0119] Specifically, the target positioning device first obtains a first sample image with bounding box label information of the corresponding target, and then infers description information of the first sample image with bounding box label information of the corresponding target through a trained preset area description expert model to obtain second text prompt information. That is, the input of the preset area description expert model is the first sample image with bounding box label information of the corresponding target, and the output is a complex description of the first sample image, that is, the second text prompt information.

[0120] It should be noted that the preset area description expert model is trained accordingly by using the next token prediction loss function, and is a trained model.

[0121] Specifically, Figure 4As shown, the multimodal large model structure used by the region description expert model (preset region description expert model) includes a visual encoder, an adapter, a tag parser and a large language model. Among them, ViT with a parameter amount of 1B can be used as the visual encoder of the region description expert model, and the purpose of the visual encoder is to extract the semantic features in the first sample image; and the structure of the corresponding adapter of the region description expert model is an Adapter structure, and the adapter of the Adapter structure can reduce the dimension of the image semantic features as the input signal (visual signal) of the large language model. In specific implementation, the adapter of the Adapter structure is implemented using a 2-layer fully connected layer; further, the region description expert model can use InternLMv2 with a parameter amount of 7B as the large language model, and the large language model takes the visual signal (image semantic features after feature dimensionality reduction) and the corresponding bounding box label information as input, and outputs a detailed description of the bounding box area, that is, the second text prompt information.

[0122] Step S03, determining a preset bounding box positioning data set based on a first data set consisting of the second text prompt information and the first sample image, and the bounding box label;

[0123] In this embodiment, after obtaining the second text prompt information, the second text prompt information and the first sample image are used as a first data set, and the first data set and the bounding box label are integrated to obtain a preset bounding box positioning data set.

[0124] Specifically, the second text prompt information can be described by a description mode of <subject, attribute>, and then the information described by the description mode and the first sample image are used as a first data set, and the first data set and the bounding box label are integrated to obtain a preset bounding box positioning data set.

[0125] The step of determining a preset bounding box positioning data set based on the first data set composed of the second text prompt information and the first sample image, and the bounding box label, includes the following steps S031-S032:

[0126] Step S031, adjusting the second text prompt information into triple information described in triple form, wherein the triple information includes subjects and interaction relationships between subjects, wherein one of the subjects in the triple information is a target;

[0127] In practical applications, the target can be summarized by a triple form <subject 1, relationship, subject 2>, that is, specifically, in this embodiment, the second text prompt information can be adjusted to a triple information described in the triple form <subject 1, relationship, subject 2>. Specifically, a large language model can be used to extract triple information for the subject in each area description, wherein one of the subjects in the triple information is the target, such as subject 1 is the target, and subject 2 is an object / person related to target 1. For example, the second text prompt information is: a boy wearing blue clothes, then the boy is subject 1, clothes are subject 2, and clothes are a relationship.

[0128] Step S032: determining a preset bounding box positioning data set based on the triplet information, the bounding box label and the first sample image.

[0129] After obtaining the triplet information, a preset bounding box positioning data set is determined based on the triplet information, the bounding box label and the first sample image.

[0130] Specifically, the step of determining a preset bounding box positioning data set based on the triple information, the bounding box label and the first sample image includes the following steps S0321-S0322:

[0131] Step S0321, modifying the subject and / or the interaction relationship between the subjects in the triple information to obtain first negative sample data, wherein the data in the triple information without adjusting the subject and the interaction relationship between the subjects is used as the first positive sample data;

[0132] Step S0322: determining a preset bounding box positioning data set based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image.

[0133] In this embodiment, it is explained how to determine a preset bounding box positioning data set based on triple information. Specifically, it involves triple information extraction, modification and fusion. That is, after obtaining the <subject 1, relationship, subject 2> triple information containing the interaction relationship between subjects, the triple information is then fused into positive / negative sample data.

[0134] The following describes how to fuse triple information into positive / negative sample data:

[0135] That is, the step of determining a preset bounding box positioning data set based on the triple information, the bounding box label and the first sample image includes:

[0136] Modifying the subjects and / or the interaction relationship between the subjects in the triple information to obtain first negative sample data, wherein the data in the triple information without adjusting the subjects and the interaction relationship between the subjects is used as the first positive sample data;

[0137] A preset bounding box positioning data set is determined based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image.

[0138] like Figure 5 As shown, by modifying the subject and / or the interaction relationship between the subjects in the triple information, the first negative sample data is obtained, and the data without adjusting the subject and the interaction relationship between the subjects is the first positive sample data.

[0139] Specifically, Figure 5 As shown, the first negative sample data can be complex description negative sample 1, complex description negative sample 2, and complex description negative sample 3, wherein the complex description negative sample 1 is obtained by modifying subject 1, the complex description negative sample 2 is obtained by modifying the interaction relationship between subject 1 and subject 2, and the complex description negative sample 3 is obtained by modifying subject 2.

[0140] The following is a specific description using an embodiment. After determining the second text prompt information (area description) "This area is a middle-aged male wearing a blue shirt and dark trousers. He is holding a mobile phone in his left hand and a black cloth bag in his right hand, walking on a crosswalk.", the corresponding triplet information is extracted. The extracted triplet information may be <middle-aged male, holding a mobile phone and a black cloth bag> and <male, wearing a blue shirt and dark trousers>, etc.

[0141] Then, a large language model is used to fuse the triplet information into a phrase description. For example, <middle-aged male, holding, mobile phone and black cloth bag> is fused into "middle-aged male holding mobile phone and black cloth bag", and the triplet information <male, wearing, blue top and dark trousers> is fused into "male wearing blue top and dark trousers". The description obtained by fusion in this step is the positive sample of the description of this area (the first positive sample data).

[0142] Further, a large language model is used (the large language model used is the Qwen1.5-72B-Chat model, but this is not limited to this. Any large language model capable of completing the above work can be used as a tool for this step) to modify "subject 1", "subject 2" and "relationship" in the triple information into opposite or semantically different descriptions, and merge them into new phrase descriptions. For example, the triple information <middle-aged male, holding, mobile phone and black cloth bag> is modified to <middle-aged male, holding, white handbag>, <female, holding, mobile phone and black cloth bag> and <middle-aged male, not holding, mobile phone and black cloth bag>, and merged into short sentence descriptions "middle-aged male holding white handbag", "female holding mobile phone and black cloth bag" and "middle-aged male not holding mobile phone and black cloth bag". The description obtained by fusion in this step is the negative sample of the region (first negative sample data).

[0143] It should be noted that compared with the <subject, attribute> pattern, such as descriptions such as "a person with long hair" and "a pink hat", the triple description constructed in this application is more fine-grained and has more information, which is sufficient to construct complex descriptive data.

[0144] It can be understood that in this embodiment, using triples as an intermediate step to obtain complex descriptions can make triple modification more controllable and reduce the probability of erroneous responses of large language models.

[0145] Step S04: Based on the preset bounding box positioning data set, the basic multimodal large model is trained to obtain the preset multimodal large model.

[0146] In this embodiment, after obtaining the preset bounding box positioning data set, the basic multimodal large model is trained based on the preset bounding box positioning data set to obtain the preset multimodal large model.

[0147] It can be understood that in the present application, in the training data of the preset multimodal large model, the bounding box label and the first data set are in an N-to-1 relationship, and N is greater than or equal to 0, that is, N can be 0, 1 or data greater than 1, that is, there may be no bounding box label, or there may be only 1 or more bounding box labels, and the bounding box label and the corresponding target are consistent. Therefore, when detecting targets described in natural language, there is no need for the technology to assume that there must be and only one target referred to by the current description in the image, and there will be no false positive cases, which can be more in line with practical applications and can improve accuracy. In addition, in the present application, since the first data set in the training data of the preset multimodal large model includes a first sample image and a second text prompt information describing the target to be retrieved, and has a bounding box label (reflecting the target to be located), the preset multimodal large model that has been trained can accurately locate the target based on the image to be processed and the first text prompt information.

[0148] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above embodiment can be referred to the above introduction, and will not be repeated in the following. Figure 2 The step of determining a preset bounding box positioning data set based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image comprises the following steps A1-A4:

[0149] Step A1, taking the first negative sample data with a preset first matching label, the corresponding bounding box label and the corresponding first sample image as a first combined data set, and taking the first positive sample data with a preset second matching label, the corresponding bounding box label and the corresponding first sample image as a second combined data set;

[0150] Step A2, training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model;

[0151] In this embodiment, the first negative sample data with a preset first matching label, the corresponding bounding box label and the corresponding first sample image are used as a first combined data set, and the first positive sample data with a preset second matching label, the corresponding bounding box label and the corresponding first sample image are used as a second combined data set, for example, the first combined data set is <first sample image, bounding box label, first negative sample data>, and the second combined data set is <first sample image, bounding box label, first positive sample data>. Wherein, the first combined data set has a preset first matching label, and the second combined data set has a preset second matching label, for example, <first sample image, bounding box label, first positive sample data>---whether it matches---<yes>, then <yes> is the second preset matching label, <first sample image, bounding box label, first negative sample data>---whether it matches---<no>, then <no> is the second preset matching label, and the basic reference expression matching expert model is trained based on the first combined data set and the second combined data set with corresponding matching labels to obtain a target reference expression matching expert model.

[0152] Step A3, inputting the first permutation and combination data set into the target reference expression matching expert model, performing one-to-one cross matching on two sets of data in the permutation and combination data set based on the target reference expression matching expert model, and obtaining a first matching result of whether the two sets of data match; wherein the first permutation and combination data set includes a set of data consisting of a preset second sample image and a bounding box of a target corresponding to the preset second sample image, and another set of data consisting of a second negative sample data or a second positive sample data, which are permuted and combined;

[0153] After obtaining the target indication expression matching expert model, a first arranged and combined data set obtained by arranging and combining a group of data consisting of a preset second sample image and a bounding box of the target corresponding to the preset second sample image, and another group of data consisting of the second negative sample data or the second positive sample data is input into the target indication expression matching expert model to obtain a first matching result of whether the two groups of data match.

[0154] For example, <second sample image, A bounding box label, second positive sample data>, <second sample image, B bounding box label, second negative sample data>, <second sample image, A bounding box label, second negative sample data>, <second sample image, B bounding box label, second positive sample data>, etc. are input into the target reference expression matching expert model to obtain a first matching result of whether the two sets of data match.

[0155] Step A4: determining a preset bounding box positioning dataset based on the first matching result, the first combined dataset and the second combined dataset.

[0156] After obtaining a first matching result of yes or no, a preset bounding box positioning data set is determined based on the first matching result, the first combined data set, and the second combined data set.

[0157] Specifically, the step of determining a preset bounding box positioning data set based on the first matching result, the first combined data set, and the second combined data set includes the following steps B1-B2:

[0158] Step B1, based on the first matching result, determining a pseudo label corresponding to each piece of data in the first permuted and combined data set, and a first confidence level corresponding to the corresponding pseudo label;

[0159] In this embodiment, after obtaining the first matching result, the pseudo-label corresponding to each data in the data set after the first arrangement and combination can be determined (since the label corresponding to each data is predicted by the target reference expression matching expert model, it is called a pseudo-label), and the first confidence corresponding to the corresponding pseudo-label can be determined, such as the first confidence of a certain pseudo-label is 90%, and the first confidence of another pseudo-label is 60%.

[0160] Step B2, screening out a first target data set whose corresponding first confidence is higher than a preset confidence threshold from the first permuted and combined data set, and determining a preset bounding box positioning data set as the preset bounding box positioning data set by using the first target data set, the first combined data set and the second combined data set.

[0161] A first target data set having a corresponding first confidence level higher than a preset confidence threshold, such as 70%, is selected from the first permutation and combination data set, such as Figure 6 As shown, the first target data set, the first combined data set and the second combined data set are used to determine a preset bounding box positioning data set as the preset bounding box positioning data set.

[0162] Specifically, for example, after inputting <second sample image, A bounding box label, second positive sample data>, <second sample image, B bounding box label, second negative sample data>, <second sample image, A bounding box label, second negative sample data>, <second sample image, B bounding box label, second positive sample data>, etc. into the target reference expression matching expert model, the first confidence level of <second sample image, A bounding box label, second positive sample data>---whether it matches---<yes> is 90%, the first confidence level of <second sample image, B bounding box label, second negative sample data> whether it matches---<no> is 60%, the first confidence level of <second sample image, A bounding box label, second negative sample data>---whether it matches---<no> is 80%, and the first confidence level of <second sample image, B bounding box label, second positive sample data>---whether it matches---<yes> is 70%. If the preset confidence threshold is 70%, then whether <second sample image, B bounding box label, second negative sample data> matches---the first confidence of <No> is 60% and cannot be used as the preset bounding box positioning data set, and the others can be used as the preset bounding box positioning data set.

[0163] In this embodiment, the accuracy of the preset bounding box positioning data set is ensured, thereby indirectly improving the accuracy of target positioning.

[0164] The step of training the basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain the target reference expression matching expert model includes:

[0165] Step C1, based on the first combined data set and the second combined data set, using the basic reference expression matching expert model to predict whether the corresponding text prompt information of each combined data set matches the corresponding first sample image and its bounding box label, and obtain a prediction result;

[0166] Step C2, based on the prediction result and the corresponding first matching label or second matching label, training the basic reference expression matching expert model to obtain an initial reference expression matching expert model;

[0167] Step C3, inputting the incremental training data into the initial reference expression matching expert model, performing one-to-one cross matching on two sets of data in the incremental training data based on the initial reference expression matching expert model, and obtaining a third matching result; wherein the incremental training data includes a set of data consisting of a preset third sample image and a bounding box of a target corresponding to the preset third sample image, and another set of data consisting of a third negative sample data or a third positive sample data;

[0168] Step C4, determining a pseudo label corresponding to each piece of data in the incremental training data, and a second confidence level corresponding to the corresponding pseudo label;

[0169] Step C5, selecting a second target data set whose second confidence is higher than a preset confidence threshold from the incremental training data, and iteratively training the initial reference expression matching expert model based on the second target data set until a target reference expression matching expert model is obtained.

[0170] In this embodiment, in order to improve the performance of the referential expression matching expert model, after each training to obtain a version of the referential expression expert model, the model is used to infer more pseudo labels (samples) of <image, bounding box, positive / negative sample> and the corresponding second confidence. Figure 6 As shown, samples with values ​​above the confidence threshold are then selected as incremental training data for the next version of the model. By iterating this approach, an expert model of referential expression with steadily improved performance can be obtained.

[0171] Specifically, based on the prediction result and the corresponding first matching label or the second matching label, the basic reference expression matching expert model is trained to obtain the initial reference expression matching expert model, and then the incremental training data is input into the initial reference expression matching expert model, and the two sets of data in the incremental training data are cross-matched one-to-one based on the initial reference expression matching expert model to obtain a third matching result; wherein the incremental training data includes a set of data consisting of a preset third sample image and a bounding box of a target corresponding to the preset third sample image, and another set of data consisting of a third negative sample data or a third positive sample data, wherein one set of data can be: <third sample image, C bounding box label>, <third sample image, D bounding box label>, <third sample image, E bounding box label>, and the other set of data can be <third negative Sample data 1>, <third positive sample data 2>, <third negative sample data 3>, etc. After obtaining the incremental training data, the incremental training data is input into the initial reference expression matching expert model, and the two groups of data in the incremental training data are cross-matched one-to-one based on the initial reference expression matching expert model to obtain a third matching result. Specifically, based on the initial reference expression matching expert model, it is judged whether <third sample image, C boundary box label> matches <third negative sample data 1>, whether <third sample image, D boundary box label> matches <third negative sample data 3>, whether <third sample image, E boundary box label> matches <third negative sample data 3>, whether <third sample image, C boundary box label> matches <third positive sample data 2>, etc., to obtain a matching judgment result, and the matching judgment result is used as the third matching result.

[0172] After obtaining the third matching result, determine the pseudo-label corresponding to each data in the incremental training data and the second confidence corresponding to the corresponding pseudo-label, screen out a second target data set whose second confidence is higher than a preset confidence threshold such as 70% from the incremental training data, and iteratively train the initial reference expression matching expert model based on the second target data set until the target reference expression matching expert model is obtained. For example, if the second confidence of <third sample image, C bounding box label, third negative sample data 1>---whether it matches---<yes> is 90%, the second confidence of <third sample image, D bounding box label, third negative sample data 3> whether it matches---<no> is 60%, the second confidence of <third sample image, E bounding box label, third negative sample data 3>---whether it matches---<no> is 80%, etc., then the second confidence of <third sample image, D bounding box label, third negative sample data 3> whether it matches---<no> is 60% and cannot be used as the second target data set, and the others can be used as the second target data set.

[0173] In this embodiment, the initial reference expression matching expert model is iteratively trained based on the second target data set until a target reference expression matching expert model is obtained.

[0174] In this embodiment, the first negative sample data with a preset first matching label, the corresponding bounding box label and the corresponding first sample image are taken as a first combined data set, and the first positive sample data with a preset second matching label, the corresponding bounding box label and the corresponding first sample image are taken as a second combined data set; a basic reference expression matching expert model is trained based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model; the first permuted and combined data set is input into the target reference expression matching expert model, and a one-to-one cross-matching is performed on the two groups of data in the permuted and combined data set based on the target reference expression matching expert model to obtain a first matching result of whether the two groups of data match; wherein the first permuted and combined data set includes a group of data consisting of a preset second sample image and a bounding box of a target corresponding to the preset second sample image, and another group of data consisting of the second negative sample data or the second positive sample data; based on the first matching result, the first combined data set and the second combined data set, a preset bounding box positioning data set is determined. It can be understood that the present application can produce a complex description target positioning data set, and then, through the complex description target positioning data set, a preset multimodal large model can be trained more accurately, and then, the target positioning can be performed more accurately.

[0175] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and will not be described in detail later. Figure 3 The step of training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model includes:

[0176] Step D1, cropping the first sample image to obtain a sub-image corresponding to the bounding box label;

[0177] Step D2: training a basic reference expression matching expert model based on the sub-image, the complete first sample image and other data parts in the corresponding first combined data set and second combined data set to obtain a target reference expression matching expert model.

[0178] In this embodiment, if Figure 7 As shown, in order to improve the model's fine-grained perception ability of the bounding box area, a sub-image corresponding to the bounding box label is also cropped from the first sample image, that is, the bounding box area of ​​the target is cropped from the complete image into a separate sub-image, and used together with the complete image (the complete first sample image) as image input. Specifically, based on the sub-image, the complete first sample image and other data parts in the corresponding first combined data set and the second combined data set (excluding the complete first sample image in the first combined data set or the second combined data set as the other data parts), the basic reference expression matching expert model is trained to obtain the target reference expression matching expert model.

[0179] In this embodiment, it can be understood that using the cropped individual sub-image as part of the image input can amplify the influence of the sub-image (increase the proportion of the sub-image corresponding to the target), so that the corresponding model can better perceive the fine-grained differences between different samples, and give a correct answer based on the correctness of the description of the bounding box area of ​​each sample, thereby improving the accuracy of target positioning.

[0180] In this embodiment, a sub-image corresponding to the bounding box label is obtained by cropping from the first sample image; a basic reference expression matching expert model is trained based on the sub-image, the complete first sample image, and other data parts in the corresponding first combined data set and second combined data set to obtain a target reference expression matching expert model. In this application, the influence of the sub-image is expanded to improve the accuracy of the target reference expression matching expert model.

[0181] Based on the first, second and third embodiments of the present application, in the third embodiment of the present application, the same or similar contents as those in the above embodiments can be referred to the above introduction, and no further description will be given later. Fig. 9 As shown, the application-related process can include 5 stages:

[0182] 1) Use the trained preset region description expert model (region description expert model) to generate a detailed region description (second text prompt information) for each bounding box: Specifically, a multimodal large model is trained through the <image, bounding box, region description> dataset as the preset region description expert model, that is, the preset region description expert model is a trained model, and the trained preset region description expert model is used to infer the bounding boxes of all targets in the image data to obtain a detailed region description of each bounding box, that is, the second text prompt information.

[0183] 2) Triple information extraction, modification and fusion (core improvement part): Use a large language model to extract <subject 1, relationship, subject 2> triples from each region description (second text prompt information) produced in 1), and fuse each triple into a positive / negative sample (first negative sample data / first positive sample data) for the region it describes.

[0184] 3) Training of the referent expression matching expert model (core improvement part): Use the <image, bounding box, positive / negative sample> data produced in 2) to train a multimodal large model to obtain the target referent expression matching expert model.

[0185] 4) Production of complex description target positioning dataset: Use the target reference expression matching expert model trained in 3) to perform one-to-one cross matching on all <images, bounding boxes> and positive / negative sample data to produce a complex description target positioning dataset (preset bounding box positioning dataset).

[0186] 5) Training of an expert model that can locate targets with complex descriptions: Using the data produced in 4), train a large multimodal model, and ultimately obtain an expert model (preset large multimodal model) that can accurately locate targets with complex descriptions.

[0187] This application also provides a target positioning device, please refer to Fig.10 , the device comprises:

[0188] The acquisition module 10 is used to acquire the image to be processed and the first text prompt information, wherein the first text prompt information is the description information of the target to be retrieved from the image to be processed;

[0189] A positioning module 20, configured to locate a bounding box of a target in the image to be processed by using a preset multimodal large model based on the image to be processed and the first text prompt information, and use the positioning result of the bounding box as the positioning result of the target;

[0190] Among them, the preset multimodal large model is obtained by training the basic multimodal large model based on the preset bounding box positioning data set, each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, the bounding box label and the first data set are in an N-to-1 relationship, N is greater than or equal to 0, and the first data set includes a first sample image and a second text prompt information describing the target to be retrieved.

[0191] In one embodiment, the target positioning device is used to achieve:

[0192] Obtaining a first sample image and bounding box label information of an object in the first sample image;

[0193] Based on the first sample image and the bounding box label information of the target in the first sample image, the bounding box of the target in the image to be processed is inferred by a preset region description expert model to obtain second text prompt information, wherein the preset region description expert model is a pre-trained model;

[0194] Determine a preset bounding box positioning data set based on a first data set consisting of the second text prompt information and the first sample image, and the bounding box label;

[0195] Based on the preset bounding box positioning data set, the basic multimodal large model is trained to obtain the preset multimodal large model.

[0196] In one embodiment, the target positioning device is used to achieve:

[0197] Adjusting the second text prompt information into triple information described in triple form, wherein the triple information includes subjects and interaction relationships between subjects, wherein one of the subjects in the triple information is a target;

[0198] A preset bounding box positioning data set is determined based on the triplet information, the bounding box label and the first sample image.

[0199] In one embodiment, the target positioning device is used to achieve:

[0200] Modifying the subjects and / or the interaction relationship between the subjects in the triple information to obtain first negative sample data, wherein the data in the triple information without adjusting the subjects and the interaction relationship between the subjects is used as the first positive sample data;

[0201] A preset bounding box positioning data set is determined based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image.

[0202] In one embodiment, the target positioning device is used to achieve:

[0203] The first negative sample data with a preset first matching label, the corresponding bounding box label, and the corresponding first sample image are used as a first combined data set, and the first positive sample data with a preset second matching label, the corresponding bounding box label, and the corresponding first sample image are used as a second combined data set;

[0204] Training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model;

[0205] The first permutation and combination data set is input into the target reference expression matching expert model, and two sets of data in the permutation and combination data set are cross-matched one-to-one based on the target reference expression matching expert model to obtain a first matching result of whether the two sets of data match; wherein the first permutation and combination data set includes a set of data consisting of a preset second sample image and a bounding box of a target corresponding to the preset second sample image, and another set of data consisting of a second negative sample data or a second positive sample data, which are permuted and combined;

[0206] A preset bounding box positioning dataset is determined based on the first matching result, the first combined dataset, and the second combined dataset.

[0207] In one embodiment, the target positioning device is used to achieve:

[0208] Based on the first matching result, determine a pseudo label corresponding to each piece of data in the data set after the first arrangement and combination, and a first confidence level corresponding to the corresponding pseudo label;

[0209] A first target data set having a corresponding first confidence higher than a preset confidence threshold is screened out from the first permuted and combined data set, and a preset bounding box positioning data set is determined as the preset bounding box positioning data set by using the first target data set, the first combined data set, and the second combined data set.

[0210] In one embodiment, the target positioning device is used to achieve:

[0211] Based on the first combined data set and the second combined data set, predicting whether the corresponding text prompt information of each combined data set matches the corresponding first sample image and its bounding box label through the basic reference expression matching expert model to obtain a prediction result;

[0212] Based on the prediction result and the corresponding first matching label or second matching label, the basic reference expression matching expert model is trained to obtain an initial reference expression matching expert model;

[0213] Inputting the incremental training data into the initial reference expression matching expert model, and performing one-to-one cross matching on two groups of data in the incremental training data based on the initial reference expression matching expert model to obtain a third matching result; wherein the incremental training data includes a group of data consisting of a preset third sample image and a bounding box of a target corresponding to the preset third sample image, and another group of data consisting of a third negative sample data or a third positive sample data;

[0214] Determine a pseudo label corresponding to each piece of data in the incremental training data, and a second confidence level corresponding to the corresponding pseudo label;

[0215] A second target data set having a second confidence level higher than a preset confidence threshold is selected from the incremental training data, and the initial reference expression matching expert model is iteratively trained based on the second target data set until a target reference expression matching expert model is obtained.

[0216] In one embodiment, the target positioning device is used to achieve:

[0217] Cropping the first sample image to obtain a sub-image corresponding to the bounding box label;

[0218] Based on the sub-image, the complete first sample image and other data in the corresponding first combined data set and second combined data set, a basic reference expression matching expert model is trained to obtain a target reference expression matching expert model.

[0219] The target positioning device provided by the present application adopts the target positioning method in the above embodiment to solve the technical problem of target positioning. Compared with the prior art, the beneficial effects of the target positioning device provided by the present application are the same as the beneficial effects of the target positioning method provided by the above embodiment, and other technical features in the target positioning device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0220] The present application provides a target positioning device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target positioning method in the above-mentioned embodiment 1.

[0221] Reference below Figure 8 , which shows a schematic diagram of the structure of a target positioning device suitable for implementing the embodiment of the present application. The target positioning device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The target positioning device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0222] like Figure 8 As shown, the target positioning device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the target positioning device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the target positioning device to communicate with other devices wirelessly or by wire to exchange data. Although the target positioning device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0223] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0224] The target positioning device provided by the present application adopts the target positioning method in the above embodiment to solve the technical problem of target positioning. Compared with the prior art, the beneficial effects of the target positioning device provided by the present application are the same as the beneficial effects of the target positioning method provided by the above embodiment, and other technical features in the target positioning device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0225] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0226] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0227] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the target positioning method in the above-mentioned embodiment.

[0228] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0229] The computer-readable storage medium may be included in the target positioning device; or may exist independently without being assembled into the target positioning device.

[0230] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the target positioning device, the target positioning device is enabled to: execute the target positioning method.

[0231] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the client computer, partially on the client computer, as a separate software package, partially on the client computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the client computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0232] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0233] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0234] The readable storage medium provided in the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned target positioning method, and can solve the technical problem of target positioning. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the present application are the same as the beneficial effects of the target positioning method provided in the above-mentioned embodiment, and will not be described in detail here.

[0235] The present application also provides a computer program product, including a computer program, which implements the steps of the target positioning method as described above when executed by a processor.

[0236] The computer program product provided by the present application can solve the technical problem of target positioning. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the target positioning method provided by the above embodiment, which will not be repeated here.

[0237] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A target positioning method, It is characterized in that The method comprises: Acquire an image to be processed and first text prompt information, where the first text prompt information is description information of a target to be retrieved from the image to be processed; Based on the image to be processed and the first text prompt information, a bounding box of the target in the image to be processed is located by using a preset multimodal large model, and a positioning result of the bounding box is used as a positioning result of the target; The preset multimodal large model is obtained by training a basic multimodal large model based on a preset bounding box positioning data set, each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, and the bounding box label and the first data set are in an N-to-1 relationship, where N is greater than or equal to 0. In the training process of the preset multimodal large model, there is a situation where there is no target described in the second text prompt information in the first sample image, or there is only one target described in the second text prompt information in the first sample image, or there are multiple targets described in the second text prompt information in the first sample image. The first data set includes a first sample image and second text prompt information describing the target to be retrieved, and the second text prompt information is a complex description of the first sample image, wherein a triple is used as an intermediate step to obtain the complex description; The target positioning method further includes: Crop a sub-image corresponding to the bounding box label from the first sample image; Based on the sub-image, the complete first sample image and other data in the corresponding first combined data set and the second combined data set, the basic reference expression matching expert model is trained to obtain a target reference expression matching expert model, and the target reference expression matching expert model is used to determine a preset bounding box positioning data set.

2. The method according to claim 1, It is characterized in that Before the step of locating the bounding box of the target in the image to be processed by using a preset multimodal large model based on the image to be processed and the first text prompt information, the method includes: Obtaining a first sample image and bounding box label information of an object in the first sample image; Based on the first sample image and the bounding box label information of the target in the first sample image, the bounding box of the target in the image to be processed is inferred by a preset region description expert model to obtain second text prompt information, wherein the preset region description expert model is a pre-trained model; Determine a preset bounding box positioning data set based on a first data set consisting of the second text prompt information and the first sample image, and the bounding box label; Based on the preset bounding box positioning data set, the basic multimodal large model is trained to obtain the preset multimodal large model.

3. The method according to claim 2, It is characterized in that The step of determining a preset bounding box positioning data set based on a first data set consisting of the second text prompt information and the first sample image, and the bounding box label, comprises: Adjusting the second text prompt information into triple information described in triple form, wherein the triple information includes subjects and interaction relationships between subjects, wherein one of the subjects in the triple information is a target; A preset bounding box positioning data set is determined based on the triplet information, the bounding box label and the first sample image.

4. The method according to claim 3, It is characterized in that The step of determining a preset bounding box positioning data set based on the triple information, the bounding box label and the first sample image comprises: Modifying the subjects and / or the interaction relationship between the subjects in the triple information to obtain first negative sample data, wherein the data in the triple information without adjusting the subjects and the interaction relationship between the subjects is used as the first positive sample data; A preset bounding box positioning data set is determined based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image.

5. The method according to claim 4, It is characterized in that The step of determining a preset bounding box positioning data set based on the first negative sample data, the first positive sample data, the corresponding bounding box label and the corresponding first sample image comprises: The first negative sample data with a preset first matching label, the corresponding bounding box label, and the corresponding first sample image are used as a first combined data set, and the first positive sample data with a preset second matching label, the corresponding bounding box label, and the corresponding first sample image are used as a second combined data set; Training a basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain a target reference expression matching expert model; The first permutation and combination data set is input into the target reference expression matching expert model, and two sets of data in the permutation and combination data set are cross-matched one-to-one based on the target reference expression matching expert model to obtain a first matching result of whether the two sets of data match; wherein the first permutation and combination data set includes a set of data consisting of a preset second sample image and a bounding box of a target corresponding to the preset second sample image, and another set of data consisting of a second negative sample data or a second positive sample data, which are permuted and combined; A preset bounding box positioning dataset is determined based on the first matching result, the first combined dataset, and the second combined dataset.

6. The method according to claim 5, It is characterized in that The step of determining a preset bounding box positioning dataset based on the first matching result, the first combined dataset, and the second combined dataset comprises: Based on the first matching result, determine a pseudo label corresponding to each piece of data in the data set after the first arrangement and combination, and a first confidence level corresponding to the corresponding pseudo label; A first target data set having a corresponding first confidence higher than a preset confidence threshold is screened out from the first permuted and combined data set, and the first target data set, the first combined data set, and the second combined data set are used to determine a preset bounding box positioning data set as the preset bounding box positioning data set.

7. The method according to claim 5, It is characterized in that The step of training the basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain the target reference expression matching expert model includes: Based on the first combined data set and the second combined data set, predicting whether the corresponding text prompt information of each combined data set matches the corresponding first sample image and its bounding box label through the basic reference expression matching expert model to obtain a prediction result; Based on the prediction result and the corresponding first matching label or second matching label, the basic reference expression matching expert model is trained to obtain an initial reference expression matching expert model; Inputting the incremental training data into the initial reference expression matching expert model, and performing one-to-one cross matching on two groups of data in the incremental training data based on the initial reference expression matching expert model to obtain a third matching result; wherein the incremental training data includes a group of data consisting of a preset third sample image and a bounding box of a target corresponding to the preset third sample image, and another group of data consisting of a third negative sample data or a third positive sample data; Determine a pseudo label corresponding to each piece of data in the incremental training data, and a second confidence level corresponding to the corresponding pseudo label; A second target data set having a second confidence level higher than a preset confidence threshold is selected from the incremental training data, and the initial reference expression matching expert model is iteratively trained based on the second target data set until a target reference expression matching expert model is obtained.

8. The method according to claim 5, It is characterized in that The step of training the basic reference expression matching expert model based on the first combined data set and the second combined data set to obtain the target reference expression matching expert model includes: Cropping the first sample image to obtain a sub-image corresponding to the bounding box label; Based on the sub-image, the complete first sample image and other data in the corresponding first combined data set and second combined data set, a basic reference expression matching expert model is trained to obtain a target reference expression matching expert model.

9. A target positioning device, It is characterized in that The device comprises: An acquisition module, used for acquiring an image to be processed and first text prompt information, wherein the first text prompt information is description information of a target to be retrieved from the image to be processed; A positioning module, configured to locate a bounding box of a target in the image to be processed by using a preset multimodal large model based on the image to be processed and the first text prompt information, and use the positioning result of the bounding box as the positioning result of the target; The preset multimodal large model is obtained by training a basic multimodal large model based on a preset bounding box positioning data set, each data in the preset bounding box positioning data set includes a first data set and a bounding box label corresponding to the first data set, and the bounding box label and the first data set are in an N-to-1 relationship, where N is greater than or equal to 0. In the training process of the preset multimodal large model, there is a situation where there is no target described in the second text prompt information in the first sample image, or there is only one target described in the second text prompt information in the first sample image, or there are multiple targets described in the second text prompt information in the first sample image. The first data set includes a first sample image and second text prompt information describing the target to be retrieved, and the second text prompt information is a complex description of the first sample image, wherein a triple is used as an intermediate step to obtain the complex description; The device is also used to achieve: Crop a sub-image corresponding to the bounding box label from the first sample image; Based on the sub-image, the complete first sample image and other data in the corresponding first combined data set and the second combined data set, the basic reference expression matching expert model is trained to obtain a target reference expression matching expert model, and the target reference expression matching expert model is used to determine a preset bounding box positioning data set.

10. A target positioning device, It is characterized in that The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the target positioning method according to any one of claims 1 to 8.

11. A storage medium, It is characterized in that The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the target positioning method according to any one of claims 1 to 8 are implemented.

12. A computer program product, It is characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the target positioning method according to any one of claims 1 to 8 are implemented.