Image annotation method and device based on multi-modal large model and related equipment
Through a multimodal large model, the image and text information are predicted, the annotation area is generated and the annotation is marked, which solves the problem of long image annotation time and improves the model training efficiency.
Patent Information
- Application Number
- CN202410946576.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-22
AI Technical Summary
In the prior art, the image annotation time is long, resulting in low model training efficiency.
By obtaining text information of the initial image and the target object, input it to the preset multimodal model for prediction, generating annotation area and confidence, and marking when the confidence is greater than the set threshold, improving the efficiency of image annotation.
When annotating a large number of initial images, it significantly reduces time-consuming and improves the efficiency of model training.
Smart Images

Figure CN119006773A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning and multimodal large models, and in particular to an image annotation method, device and related equipment based on a multimodal large model. Background Art
[0002] Multimodal Large Models refer to models that combine multimodal information such as text, images, videos, and audio for training, including but not limited to multimodal large language models. "Large Language Model" refers to a large-scale language model (LLM), which is a language model with a large parameter scale. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc. The rapid development of large language model technology has brought new opportunities for economic and social development. How to realize the practical application of large language models is an issue that needs to be studied urgently.
[0003] In the field of machine learning and multimodal large model technology, the target object area on the image is annotated, and then the annotated image is used for model training. In related technologies, the target object is mainly annotated by painting a mask of the target object on the image, or using a polygonal box to select the location of the target object.
[0004] However, in related technologies, each image needs to be labeled by a person in turn. When there are many images, it takes a long time to label the images, which will lead to low efficiency of model training. Summary of the invention
[0005] The embodiments of the present invention provide an image annotation method, apparatus and related equipment based on a multimodal large model to solve the problem in the related art that image annotation takes a long time, resulting in low efficiency of model training.
[0006] To solve the above problems, the present invention is achieved as follows:
[0007] In a first aspect, an embodiment of the present invention provides an image annotation method based on a multimodal large model, comprising:
[0008] Step S1, obtaining an initial image and text information of a target object, wherein the target object is an object of a first type that needs to be labeled, and the text information is used to describe the target object;
[0009] Step S2: inputting the initial image and the text information into a preset multimodal large model for prediction to obtain a first annotated area and a first confidence level, wherein the first annotated area is a predicted area where the target object exists, and the first confidence level is a confidence level of the first annotated area;
[0010] Step S3: generating a first annotated image, where the first annotated image is an image obtained by annotating the initial image based on the first annotated area;
[0011] Step S4: When the first confidence level is greater than a first set confidence level threshold, setting the first annotated image as a target annotated image.
[0012] In a second aspect, an embodiment of the present invention further provides an image annotation device based on a multimodal large model, comprising:
[0013] A first acquisition module, used to acquire an initial image and text information of a target object, wherein the target object is an object of a first type that needs to be labeled, and the text information is used to describe the target object;
[0014] A first prediction module, used for inputting the initial image and the text information into a preset multimodal large model for prediction, to obtain a first annotated area and a first confidence level, wherein the first annotated area is a predicted area where the target object exists, and the first confidence level is a confidence level of the first annotated area;
[0015] A first generating module, configured to generate a first annotated image, wherein the first annotated image is an image obtained by annotating the initial image based on the first annotated area;
[0016] The first setting module is configured to set the first annotated image as a target annotated image when the first confidence level is greater than a first set confidence level threshold.
[0017] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the image annotation method based on a multimodal large model as described in the first aspect above.
[0018] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the image annotation method based on a multimodal large model as described in the first aspect above.
[0019] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps in the image annotation method based on a multimodal large model as described in the first aspect above.
[0020] In an embodiment of the present application, by obtaining the text information of the initial image and the target object, the target object is an object of the first type that needs to be annotated, and the text information is used to describe the target object; the initial image and the text information are input into the preset multimodal large model for prediction, and a first annotation area and a first confidence level are obtained, the first annotation area is the predicted area where the target object exists, and the first confidence level is the confidence level of the first annotation area; a first annotated image is generated, and the first annotated image is an image obtained by annotating the initial image based on the first annotated area; when the first confidence level is greater than the first set confidence level threshold, the first annotated image is set as the target annotated image, so as to realize the rapid annotation of the target object in the initial image through the preset multimodal large model. In this way, the initial image is annotated by the preset multimodal large model, and less time is spent when annotating a large number of initial images, thereby improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0022] Figure 1 is a flow chart of an image annotation method based on a multimodal large model provided by an embodiment of the present invention;
[0023] Figure 2 is a schematic diagram of a correction process provided by an embodiment of the present invention;
[0024] Figure 3 It is a schematic diagram of the process of predicting a preset multimodal large model provided by an embodiment of the present invention;
[0025] Figure 4 is a structural diagram of an image annotation device based on a multimodal large model provided by an embodiment of the present invention;
[0026] Figure 5 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] The "Multimodal Large Models" mentioned in the present invention refer to models that combine multimodal information such as text, images, videos, audio, etc. for training, including but not limited to multimodal large language models.
[0029] The “large language model” mentioned in the present invention refers to a large language model (LLM), which is a language model with a large parameter scale. It is designed to understand and generate human language. It is trained with a large amount of text data and can perform a wide range of tasks including text summarization, translation, sentiment analysis, etc.
[0030] See also Figure 1 , Figure 1 is a flowchart of an image annotation method based on a multimodal large model provided by an embodiment of the present invention, such as Figure 1 As shown, the following steps are included:
[0031] Step S1: Acquire an initial image and text information of a target object, wherein the target object is an object of a first type that needs to be labeled, and the text information is used to describe the target object.
[0032] The target object is an object that needs to be marked in the initial image, such as a water cup. The target object is a first type of object, and the first type is an attribute type of the target object. For example, if the target object is a water cup, the first type can be a water cup or a water cup filled with water.
[0033] The above-mentioned initial image is an image that needs to be marked with the target object. The target object may be present in its entirety, in part, or not in the initial image.
[0034] The above text information is used to describe the first type of target object, which can be the name of the target object or a description of the characteristics of the target object. For example, if the target object is a water cup, the text information can be "water cup" or "water cup placed on the table". The target object is described by text information, and then the area where the target object is located is marked in the initial image. By obtaining the text information and subsequently inputting the text information into the preset multimodal large model, the preset multimodal large model predicts the area where the first type of target object is located from the initial image.
[0035] Step S2: input the initial image and the text information into a preset multimodal large model for prediction to obtain a first annotated area and a first confidence level, wherein the first annotated area is a predicted area where the target object exists, and the first confidence level is a confidence level of the first annotated area.
[0036] The above-mentioned preset multimodal large model is used for image annotation and description, as well as cross-modal retrieval of input text information. Among them, the preset multimodal large model is used for image annotation and description, specifically combining the image and text to generate the description and annotation area of the image; the preset multimodal large model is used for cross-modal retrieval of the input text information, specifically querying or retrieving according to the input information (text information, image information or voice information, etc.), and obtaining the relevant information or information vector corresponding to the output information. For example, the input text information is "a cup filled with water", and the preset multimodal large model performs cross-modal retrieval on the text information, first generates a text vector corresponding to the "cup filled with water", and then searches in the initial image based on the text vector to obtain the information of the area related to the text vector in the initial image.
[0037] The above-mentioned first annotated area is a predicted area where the first type of target object is located, wherein the first annotated area is used to annotate a part of the initial image, and the annotation method can be to mask the area where the first type of target object is located, or to draw a polygon, and the area where the first type of target object is located is inside the polygon.
[0038] The first confidence level is the confidence level of the first type of target object predicted in the first annotated area, and is used to characterize the probability that the first type of target object exists in the first annotated area. Through the first confidence level, it can be determined whether the first annotated area is the area where the first type of target object is located. If the first annotated area is the area where the first type of target object is located, the initial image annotation can be completed directly; if the first annotated area is not the area where the first type of target object is located, it can be re-annotated through correction information until the annotated area where the first type of target object is located is generated, as described in the subsequent embodiments.
[0039] Step S3: Generate a first annotated image, where the first annotated image is an image obtained by annotating the initial image based on the first annotated area.
[0040] Step S4: When the first confidence level is greater than a first set confidence level threshold, setting the first annotated image as a target annotated image.
[0041] The first set confidence threshold is used to determine whether the image marked in the first annotation area in the first annotation image is a target object of the first type. Specifically, when the first confidence is greater than the first set confidence threshold, the image marked in the first annotation area in the first annotation image is determined to be a target object of the first type, in which case the first annotation image is set as the target annotation image; and when the first confidence is less than or equal to the first set confidence threshold, the probability that the image marked in the first annotation area in the first annotation image is a target object of the first type is low, in which case the first annotation image is unreliable and needs to be further predicted through correction information.
[0042] It should be understood that in the present application, a first annotated image is first generated, and then when the first confidence is greater than a first set confidence threshold, the first annotated image is set as the target annotated image. Alternatively, the relationship between the first confidence and the first set confidence threshold may be first determined, and when the first confidence is greater than the first set confidence threshold, the first annotated image is generated. In this case, the first annotated image is the target annotated image.
[0043] In an embodiment of the present application, by obtaining the text information of the initial image and the target object, the target object is an object of the first type that needs to be annotated, and the text information is used to describe the target object; the initial image and the text information are input into the preset multimodal large model for prediction, and a first annotation area and a first confidence level are obtained, the first annotation area is the predicted area where the target object exists, and the first confidence level is the confidence level of the first annotation area; a first annotated image is generated, and the first annotated image is an image obtained by annotating the initial image based on the first annotated area; when the first confidence level is greater than the first set confidence level threshold, the first annotated image is set as the target annotated image, so as to realize the rapid annotation of the target object in the initial image through the preset multimodal large model. In this way, the initial image is annotated by the preset multimodal large model, and less time is spent when annotating a large number of initial images, thereby improving the efficiency of model training.
[0044] In one embodiment, the method further comprises:
[0045] Step S5: when the first confidence level is less than or equal to the first set confidence level threshold, obtaining correction information corresponding to the first annotated image;
[0046] Step S6: input the correction information and the first annotated image into the preset multimodal large model for prediction to obtain a second annotated area and a second confidence level, wherein the second annotated area is an area where the target object is predicted to exist, and the second confidence level is a confidence level of the second annotated area;
[0047] Step S7: generating a second annotated image, where the second annotated image is an image obtained by annotating the initial image based on the second annotated area;
[0048] Step S8: When the second confidence level is greater than the first set confidence level threshold, the second annotated image is set as the target annotated image.
[0049] The above-mentioned correction information is made based on the first annotated image and is used to correct the annotated area. When the preset multimodal large model receives the correction information, it makes a prediction based on the correction information and the first annotated image to obtain a corrected second annotated area and a second confidence level. The second annotated area is an annotated area adjusted based on the correction information, and can more accurately annotate the area where the first type of target object is located compared to the first annotated area.
[0050] Among them, the correction information can be a marking layer, a geometric border, a type identifier, a target coordinate and / or identification information. By inputting the correction information and the first annotated image into a preset multimodal large model for prediction, a second annotated area and a second confidence level are obtained, as detailed in subsequent embodiments.
[0051] Furthermore, the correction information is information about the correction of the first annotated image by the operator. The schematic diagram of the correction process is as follows: Figure 2 As shown. Among them, the front-end page is used to input correction information, including marking layers, geometric borders, type identification, target coordinates and identification information, and text information can also be input. The input method can be input in the form of a mouse or a brush. After the correction information is entered on the front-end page, the preset multimodal large model processes the correction information, merges the marking layer, the layer of the geometric border and / or the layer of the first annotated image, and then inputs the type identification, target coordinates and / or identification information, and the merged layer into the preset multimodal large model for prediction to obtain a second annotated area and a second confidence level.
[0052] The above-mentioned second annotation area is the area where the first type of target object is located, which is predicted based on the correction information and the first annotation image, wherein the second annotation area is used to annotate part of the area in the initial image, and the annotation method can be to mask the area where the first type of target object is located, or to draw a polygon, and the area where the first type of target object is located is inside the polygon.
[0053] The second confidence level is the confidence level of the first type of target object predicted in the second annotated area, and is used to characterize the probability that the first type of target object exists in the second annotated area. Through the first confidence level, it can be determined whether the second annotated area is the area where the first type of target object is located. In the case where the second annotated area is the area where the first type of target object is located, that is, when the second confidence level is greater than the first confidence level threshold, the initial image annotation can be completed; if the second annotated area is not the area where the first type of target object is located, it can continue to be re-annotated through correction information until a annotation area where the first type of target object is located is generated.
[0054] In an embodiment of the present application, when the first confidence is less than or equal to the first set confidence threshold, correction information corresponding to the first annotated image is obtained; the correction information and the first annotated image are input into a preset multimodal large model for prediction to obtain a second annotated area and a second confidence, the second annotated area is an area predicted to have a target object, and the second confidence is the confidence of the second annotated area; a second annotated image is generated, the second annotated image is an image obtained by annotating the initial image based on the second annotated area; when the second confidence is greater than the first set confidence threshold, the second annotated image is set as the target annotated image, the second annotated image is an annotated image after the annotation position is adjusted based on the correction information, thereby achieving adjustment of the annotation area through the correction information and improving the accuracy of the annotation area in the annotated image.
[0055] In one embodiment, the correction information includes target coordinates and identification information, and the identification information is used to indicate whether the target coordinates are within the marked area;
[0056] The step S6 comprises:
[0057] Step S61: when the identification information indicates that the target coordinates are within the annotated area, input the target coordinates, the identification information and the first annotated image into the preset multimodal large model for prediction, to obtain the second annotated area and the second confidence level, and the target coordinates are within the second annotated area;
[0058] Step S62: When the identification information indicates that the target coordinates are outside the marked area, the target coordinates, the identification information and the first marked image are input into the preset multimodal large model for prediction to obtain the second marked area and the second confidence level, and the target coordinates are outside the second marked area.
[0059] The above-mentioned target coordinates are the coordinates of a point in the initial image, and the identification information represents whether the target coordinates are within the annotated area. By selecting the target coordinates in the initial image and setting the identification information, the preset multimodal large model can include or exclude the target coordinates in the second annotated area predicted based on the first annotated image, the target coordinates and the identification information.
[0060] When the identification information indicates that the target coordinates are within the marked area, the target coordinates are within the second marked area; when the identification information indicates that the target coordinates are outside the marked area, the target coordinates are outside the second marked area.
[0061] In an embodiment of the present application, the correction information includes target coordinates and identification information, and the identification information is used to characterize whether the target coordinates are within the marked area; when the identification information characterizes that the target coordinates are within the marked area, the target coordinates, the identification information and the first marked image are input into a preset multimodal large model for prediction, and a second marked area and a second confidence are obtained, and the target coordinates are within the second marked area; when the identification information characterizes that the target coordinates are outside the marked area, the target coordinates, the identification information and the first marked image are input into a preset multimodal large model for prediction, and a second marked area and a second confidence are obtained, and the target coordinates are outside the second marked area. By setting a target coordinate and identification information, and then predicting the second marked area through the preset multimodal large model, the marked area can be quickly corrected. At the same time, only one or more coordinates need to be input, and there is no need to precisely mark the position of the target object, so the correction information can be quickly obtained, which improves the speed of adjusting the marked area.
[0062] In one embodiment, the correction information includes a marking layer, and the marking layer is a layer for marking the initial image;
[0063] The step S6 comprises:
[0064] Step S63: input the marked layer and the first annotated image into the preset multimodal large model for prediction to obtain the second annotated area and the second confidence level, and the mark of the marked layer is within the second annotated area.
[0065] The above-mentioned marking layer is a layer that marks part or all of the target object in the initial image. By inputting the marking layer and the first marked image into the preset multimodal large model, the second marked area includes the marking layer, thereby realizing the correction of the marked area.
[0066] The marked layer may be a layer of the target object outside the first marked area, or may be a layer of the target object within the first marked area. When the marked layer is a layer of the target object outside the first marked area, the second marked area is enlarged relative to the first marked area to include the marked layer; when the marked layer is a layer of the target object within the first marked area, the second marked area is shrunk relative to the first marked area to exclude areas within the first marked area that are not the target object.
[0067] In an embodiment of the present application, the correction information includes a marking layer, which is a layer for marking the initial image; by inputting the marking layer and the first annotated image into a preset multimodal large model for prediction, a second annotated area and a second confidence level are obtained, so that the mark of the marking layer is within the second annotated area, thereby realizing rapid correction of the annotated area.
[0068] In one embodiment, the correction information includes a geometric bounding box;
[0069] The step S6 comprises:
[0070] Step S64: input the geometric bounding box and the first annotated image into the preset multimodal large model for prediction to obtain the second annotated area and the second confidence level, wherein the second annotated area is within the geometric bounding box.
[0071] The geometric frame is the position of the edge of the target object in the initial image. The geometric frame may be a frame corresponding to the edge of the target object outside the first annotated area, or may be a frame corresponding to the edge of the target object within the first annotated area. By inputting the geometric frame and the first annotated image into a preset multimodal large model for prediction, a second annotated area including the geometric frame is obtained, thereby realizing correction of the annotated area.
[0072] Among them, when the geometric border may be a border corresponding to the edge of the target object outside the first annotation area, the predicted second annotation area is enlarged relative to the first annotation area to include the geometric border; when the geometric border may be a border corresponding to the edge of the target object within the first annotation area, the predicted second annotation area is relative to the first annotation area clue to exclude the area within the first annotation area that is not the target object.
[0073] In the embodiment of the present application, the correction information includes a geometric border; the geometric border and the first annotated image are input into a preset multimodal large model for prediction, and a second annotated area and a second confidence level are obtained, so that the second annotated area is within the geometric border, thereby realizing rapid correction of the annotated area. At the same time, the geometric border is the border of the edge of the target, and the second annotated area includes the geometric border, which can be used to determine the accuracy of the second annotated area.
[0074] In one embodiment, the correction information includes a type identifier, and the type identifier is used to indicate that the object in the first marked area is an object of the second type, and the first type and the second type are different types;
[0075] The step S6 comprises:
[0076] Step S65: input the type identifier and the first annotated image into the preset multimodal large model for prediction to obtain the second annotated area and the second confidence level, wherein the second annotated area does not include the first annotated area.
[0077] It should be understood that the object in the first annotated area can be a target object, but it is not a target object of the first type, but a target object of the second type. For example, the target object of the first type is an "empty water cup", and the target object of the second type is a "water cup filled with water". The target object is a water cup, but it does not belong to the same type. When the target object is predicted by the preset multimodal large model, there is a case where the object in the first annotated area is predicted to be a target object of the second type, in which case re-prediction is required.
[0078] The above-mentioned type identification is the type identification of the target object. By setting the object in the first annotated area to an object of the second type, and then inputting the type identification and the first annotated image into the preset multimodal large model for prediction, the second annotated area does not include the first annotated area, that is, does not include the target object of the second type, thereby realizing the correction of the annotated area.
[0079] In the embodiment of the present application, the correction information includes a type identifier, and the type identifier is used to characterize that the object in the first annotated area is an object of the second type, and the first type and the second type are different types. By inputting the type identifier and the first annotated image into a preset multimodal large model for prediction, a second annotated area and a second confidence level are obtained, so that the second annotated area does not include the first annotated area, thereby realizing rapid correction of the annotated area.
[0080] Specifically, the process of correcting the first marked area by using the correction information is as follows: Figure 3As shown, after the preset multimodal large model predicts the initial image and obtains the first annotated area and the first confidence, it is determined whether the first annotated area (or the first annotated image) meets the requirements through the first set preset confidence. If the requirements are not met, correction information is obtained, including corrected image information (such as marked layers and geometric borders), type identification, target coordinates and identification information, and at least one of text information. The correction information is then input into the preset multimodal large model for prediction to obtain a second annotated area and a second confidence. This is repeated until the confidence is greater than the first set preset confidence. At this time, the annotated area corresponding to the confidence is the target area.
[0081] In one embodiment, the preset multimodal large model is obtained by:
[0082] Acquire historical image data and historical annotation data, wherein the historical annotation data is used to annotate a region of the target object in the historical image data;
[0083] The initial multimodal large model is trained based on the historical image data and the historical annotation data to obtain the preset multimodal large model and the first confidence threshold corresponding to the preset multimodal large model.
[0084] The above historical image data is a plurality of pictures including a first type of target object, and the above historical annotation area is used to annotate the area of the first type of target object in the picture. The initial multimodal large model is trained by the historical image data and the historical annotation data to obtain a preset multimodal large model and a first confidence threshold corresponding to the preset multimodal large model, so that the preset multimodal large model can annotate the area in the initial picture where the first type of target object exists.
[0085] Furthermore, historical correction information, a historical first annotated image, and a historical second annotated image can also be obtained. The confidence corresponding to the first annotated area in the historical first annotated image is less than a first set confidence threshold. By inputting the historical correction information, the historical first annotated image, and the historical second annotated image into the initial multimodal large model for training, a multimodal large model and a first confidence threshold corresponding to the preset multimodal large model are preset, so that the preset multimodal large model can predict the correction information and the first annotated image to obtain the second annotated image.
[0086] In an embodiment of the present application, historical image data and historical annotation data are obtained, and the historical annotation data is used to annotate the area of the target object in the historical image data; the initial multimodal large model is trained based on the historical image data and the historical annotation data to obtain the training model and the third confidence level corresponding to the training model; when the third confidence level is greater than the second set confidence level threshold, the training model is set to the preset multimodal large model, so that the preset multimodal large model can mark the area in the initial image where the first type of target object exists.
[0087] See also Figure 4 , Figure 4 is a structural diagram of an image annotation device based on a multimodal large model provided by an embodiment of the present invention, such as Figure 4 As shown, the image annotation device 400 based on the multimodal large model includes:
[0088] A first acquisition module 401 is used to acquire an initial image and text information of a target object, where the target object is an object of a first type that needs to be labeled, and the text information is used to describe the target object;
[0089] A first prediction module 402 is used to input the initial image and the text information into a preset multimodal large model for prediction, and obtain a first annotated area and a first confidence level, wherein the first annotated area is a predicted area where the target object exists, and the first confidence level is a confidence level of the first annotated area;
[0090] A first generating module 403 is used to generate a first annotated image, where the first annotated image is an image obtained by annotating the initial image based on the first annotated area;
[0091] The first setting module 404 is configured to set the first annotated image as a target annotated image when the first confidence level is greater than a first set confidence level threshold.
[0092] In one embodiment, the multimodal large model-based image annotation apparatus 400 further includes:
[0093] A second acquisition module, configured to acquire correction information corresponding to the first annotated image when the first confidence level is less than or equal to the first set confidence level threshold;
[0094] a second prediction module, configured to input the correction information and the first annotated image into the preset multimodal large model for prediction, to obtain a second annotated area and a second confidence level, wherein the second annotated area is an area where the target object is predicted to exist, and the second confidence level is a confidence level of the second annotated area;
[0095] A second generating module, configured to generate a second annotated image, where the second annotated image is an image obtained by annotating the initial image based on the second annotated area;
[0096] The second setting module is used to set the second annotated image as the target annotated image when the second confidence level is greater than the first set confidence level threshold.
[0097] In one embodiment, the correction information includes target coordinates and identification information, and the identification information is used to indicate whether the target coordinates are within the marked area;
[0098] The second prediction module comprises:
[0099] a first prediction unit, configured to input the target coordinates, the identification information, and the first annotated image into the preset multimodal large model for prediction when the identification information indicates that the target coordinates are within the annotated area, to obtain the second annotated area and the second confidence, and the target coordinates are within the second annotated area;
[0100] The second prediction unit is used to input the target coordinates, the identification information and the first annotated image into the preset multimodal large model for prediction when the identification information indicates that the target coordinates are outside the annotated area, so as to obtain the second annotated area and the second confidence level, and the target coordinates are outside the second annotated area.
[0101] In one embodiment, the correction information includes a marking layer, and the marking layer is a layer for marking the initial image;
[0102] The second prediction module comprises:
[0103] A third prediction unit is used to input the marked layer and the first annotated image into the preset multimodal large model for prediction, to obtain the second annotated area and the second confidence, and the mark of the marked layer is within the second annotated area.
[0104] In one embodiment, the correction information includes a geometric bounding box;
[0105] The second prediction module comprises:
[0106] The fourth prediction unit is used to input the geometric bounding box and the first annotated image into the preset multimodal large model for prediction to obtain the second annotated area and the second confidence level, wherein the second annotated area is within the geometric bounding box.
[0107] In one embodiment, the correction information includes a type identifier, and the type identifier is used to indicate that the object in the first marked area is an object of the second type, and the first type and the second type are different types;
[0108] The second prediction module comprises:
[0109] A fifth prediction unit is used to input the type identifier and the first annotated image into the preset multimodal large model for prediction, to obtain the second annotated area and the second confidence level, wherein the second annotated area does not include the first annotated area.
[0110] In one embodiment, the preset multimodal large model is obtained by:
[0111] Acquire historical image data and historical annotation data, wherein the historical annotation data is used to annotate a region of the target object in the historical image data;
[0112] The initial multimodal large model is trained based on the historical image data and the historical annotation data to obtain the preset multimodal large model and the first confidence threshold corresponding to the preset multimodal large model.
[0113] The image annotation device based on a multimodal large model provided in an embodiment of the present invention is capable of implementing each process of each embodiment of the above-mentioned image annotation method based on a multimodal large model. The technical features correspond one to one and can achieve the same technical effect. To avoid repetition, they will not be described here.
[0114] It should be noted that the image annotation device based on the multimodal large model in the embodiment of the present invention can be a device, or a component, an integrated circuit, or a chip in an electronic device.
[0115] The embodiment of the present invention further provides an electronic device, see Figure 5 , Figure 5 is a schematic diagram of the structure of an electronic device provided by the present invention, the electronic device includes a memory 501, a processor 502 and a program or instruction stored in the memory 501 and running on the memory 501. When the program or instruction is executed by the processor 502, Figure 1 Any steps in the corresponding embodiment of the image annotation method based on the multimodal large model and the same beneficial effects are achieved, which will not be repeated here.
[0116] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or a complex programmable logic device (CPLD).
[0117] A person skilled in the art can understand that all or part of the steps of implementing the above-mentioned multimodal large model-based image annotation method embodiment can be completed by hardware related to program instructions, and the program can be stored in a readable medium.
[0118] The embodiment of the present invention further provides a readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the above Figure 1 The corresponding steps in the embodiment of the method for labeling an image based on a multimodal large model can achieve the same technical effect, and will not be repeated here to avoid repetition. The storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0119] The embodiment of the present invention also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 Corresponding to any step in the embodiment of the image annotation method based on the multimodal large model, and being able to achieve the same technical effect, will not be described again here to avoid repetition.
[0120] The terms "first", "second" etc. in the embodiments of the present invention are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. In addition, the terms "include" and "have" and any variation thereof are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment. In addition, "and / or" is used in the present application to represent at least one of the connected objects, such as A and / or B and / or C, indicating that A alone, B alone, C alone, and A and B all exist, B and C all exist, A and C all exist, and 7 situations in which A, B and C all exist.
[0121] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a second terminal device, etc.) to execute the methods of each embodiment of the present application.
[0123] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. An image annotation method based on a multimodal large model, characterized in that: include: Step S1, obtaining an initial image and text information of a target object, wherein the target object is an object of a first type that needs to be labeled, and the text information is used to describe the target object; Step S2: inputting the initial image and the text information into a preset multimodal large model for prediction to obtain a first annotated area and a first confidence level, wherein the first annotated area is a predicted area where the target object exists, and the first confidence level is a confidence level of the first annotated area; Step S3: generating a first annotated image, where the first annotated image is an image obtained by annotating the initial image based on the first annotated area; Step S4: When the first confidence level is greater than a first set confidence level threshold, setting the first annotated image as a target annotated image.
2. The method according to claim 1, characterized in that The method further comprises: Step S5: when the first confidence level is less than or equal to the first set confidence level threshold, obtaining correction information corresponding to the first annotated image; Step S6: input the correction information and the first annotated image into the preset multimodal large model for prediction to obtain a second annotated area and a second confidence level, wherein the second annotated area is an area where the target object is predicted to exist, and the second confidence level is a confidence level of the second annotated area; Step S7: generating a second annotated image, where the second annotated image is an image obtained by annotating the initial image based on the second annotated area; Step S8: When the second confidence level is greater than the first set confidence level threshold, the second annotated image is set as the target annotated image.
3. The method according to claim 2, characterized in that The correction information includes target coordinates and identification information, and the identification information is used to indicate whether the target coordinates are within the marked area; The step S6 comprises: Step S61: when the identification information indicates that the target coordinates are within the annotated area, input the target coordinates, the identification information and the first annotated image into the preset multimodal large model for prediction, to obtain the second annotated area and the second confidence level, and the target coordinates are within the second annotated area; Step S62: When the identification information indicates that the target coordinates are outside the marked area, the target coordinates, the identification information and the first marked image are input into the preset multimodal large model for prediction to obtain the second marked area and the second confidence level, and the target coordinates are outside the second marked area.
4. The method according to claim 2, characterized in that: The correction information includes a marking layer, and the marking layer is a layer for marking the initial image; The step S6 comprises: Step S63: input the marked layer and the first marked image into the preset multimodal large model for prediction to obtain the second marked area and the second confidence level, and the mark of the marked layer is within the second marked area.
5. The method according to claim 2, characterized in that: The correction information includes a geometric border; The step S6 comprises: Step S64: input the geometric bounding box and the first annotated image into the preset multimodal large model for prediction to obtain the second annotated area and the second confidence level, wherein the second annotated area is within the geometric bounding box.
6. The method according to claim 2, characterized in that The correction information includes a type identifier, where the type identifier is used to indicate that the object in the first marked area is an object of the second type, and the first type and the second type are different types; The step S6 comprises: Step S65: input the type identifier and the first annotated image into the preset multimodal large model for prediction to obtain the second annotated area and the second confidence level, wherein the second annotated area does not include the first annotated area.
7. The method according to claim 1, characterized in that The preset multimodal large model is obtained in the following manner: Acquire historical image data and historical annotation data, wherein the historical annotation data is used to annotate a region of the target object in the historical image data; The initial multimodal large model is trained based on the historical image data and the historical annotation data to obtain the preset multimodal large model and the first confidence threshold corresponding to the preset multimodal large model.
8. An image annotation device based on a multimodal large model, characterized in that: include: A first acquisition module, used to acquire an initial image and text information of a target object, wherein the target object is an object of a first type that needs to be labeled, and the text information is used to describe the target object; A first prediction module, used for inputting the initial image and the text information into a preset multimodal large model for prediction, to obtain a first annotated area and a first confidence level, wherein the first annotated area is a predicted area where the target object exists, and the first confidence level is a confidence level of the first annotated area; A first generating module, configured to generate a first annotated image, where the first annotated image is an image obtained by annotating the initial image based on the first annotated area; The first setting module is configured to set the first annotated image as a target annotated image when the first confidence level is greater than a first set confidence level threshold.
9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the image annotation method based on a multimodal large model as described in any one of claims 1 to 7.
10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps of the image annotation method based on a multimodal large model as described in any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Model training method and device, computer equipment and storage medium
CN111210024A
Neural network model training method, electronic device and computer program product
CN114359618A
