Image annotation method, device, equipment, program product and storage medium

Abstract

The embodiment of the present invention discloses an image annotation method, device, equipment, program product and storage medium, including: obtaining an image to be processed, and obtaining annotation task information of the image to be processed according to the image to be processed and a predetermined multimodal large model; for each initial annotation information, performing image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result; determining whether the segmentation result meets a preset condition, if the segmentation result does not meet the preset condition, optimizing the current initial annotation information according to the segmentation result, the image segmentation model and the multimodal large model to obtain target annotation information corresponding to the current initial annotation information; annotating the image to be processed according to the target annotation information corresponding to each initial annotation information. The method of the present invention realizes an efficient, accurate and automated image annotation process, reduces the cost of manual annotation, and improves the user experience.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence, and in particular to an image annotation method, apparatus, device, program product and storage medium. Background Art

[0002] In the field of computer vision, image object detection and positioning has become a basic and key research direction. Currently, image object detection methods based on deep learning have become the mainstream of research. For example, image object detection methods based on the YOLO model and neural network model have achieved remarkable results.

[0003] However, the successful application of these models depends on a large amount of high-quality data that needs to be manually annotated. In the actual data annotation process, a lot of manpower and material resources are needed, especially the accurate annotation of the target location (i.e. segmentation annotation). Therefore, how to accurately annotate images efficiently and automatically has become an urgent problem to be solved. Summary of the invention

[0004] The embodiments of the present invention provide an image annotation method, apparatus, device, program product and storage medium, which can automatically and accurately obtain the image annotation position by using a multimodal large model and an image segmentation model, thereby realizing precise annotation of images without human intervention and improving the efficiency of image annotation.

[0005] In a first aspect, an embodiment of the present invention provides an image annotation method, comprising:

[0006] Acquire an image to be processed, and obtain labeling task information of the image to be processed according to the image to be processed and a predetermined multimodal large model; wherein the labeling task information includes at least one initial labeling information;

[0007] For each initial annotation information, performing image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result;

[0008] Determine whether the segmentation result satisfies a preset condition. If the segmentation result does not satisfy the preset condition, optimize the current initial annotation information according to the segmentation result, the image segmentation model and the multimodal large model to obtain target annotation information corresponding to the current initial annotation information;

[0009] The image to be processed is annotated according to the target annotation information corresponding to each initial annotation information.

[0010] In a second aspect, an embodiment of the present invention provides an image annotation device, the device comprising:

[0011] An image acquisition module, used to acquire an image to be processed, and obtain annotation task information of the image to be processed according to the image to be processed and a predetermined multimodal large model; wherein the annotation task information includes at least one initial annotation information;

[0012] An image segmentation module is used to perform image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model for each initial annotation information to obtain a segmentation result;

[0013] an information optimization module, used to determine whether the segmentation result satisfies a preset condition; if the segmentation result does not satisfy the preset condition, optimize the current initial annotation information according to the segmentation result, the image segmentation model and the multimodal large model to obtain target annotation information corresponding to the current initial annotation information;

[0014] The image annotation module is used to annotate the image to be processed according to the target annotation information corresponding to each initial annotation information.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, an image annotation method as described in any one of the embodiments of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image annotation method as described in any one of the embodiments of the present invention.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the image annotation method as described in any one of the embodiments of the present invention.

[0018] In an embodiment of the present invention, an image to be processed is obtained, and the annotation task information of the image to be processed is obtained according to the image to be processed and a predetermined multimodal large model; wherein the annotation task information includes at least one initial annotation information; for each initial annotation information, the image to be processed is segmented based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result; it is determined whether the segmentation result meets the preset conditions, and if the segmentation result does not meet the preset conditions, the current initial annotation information is optimized according to the segmentation result, the image segmentation model and the multimodal large model to obtain the target annotation information corresponding to the current initial annotation information; the image to be processed is annotated according to the target annotation information corresponding to each initial annotation information. The method of the embodiment of the present invention can automatically generate the annotation information required by the image segmentation model using the multimodal large model, reducing the process of manual annotation. The initial annotation information is iteratively optimized according to the multimodal large model and the image segmentation model, and the image is accurately annotated through this coarse-to-fine optimization method, thereby improving the accuracy of image annotation. The image annotation method of this scheme realizes an efficient, accurate and automated image annotation process, reduces the cost of manual annotation, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A first flow chart of an image annotation method provided by an embodiment of the present invention;

[0021] Figure 2 A schematic diagram of an image to be processed provided by an embodiment of the present invention;

[0022] Figure 3 A flowchart of iterative optimization of initial annotation information provided by an embodiment of the present invention;

[0023] Figure 4 A second flow chart of an image annotation method provided by an embodiment of the present invention;

[0024] Figure 5 A schematic diagram of drawing an image provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the structure of an image annotation device provided by an embodiment of the present invention;

[0026] Figure 7A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Figure 1 This is a first flow chart of an image annotation method provided by an embodiment of the present invention. The method of the embodiment of the present invention can use a multimodal large model and an image segmentation model to automatically and accurately obtain the image annotation position, thereby achieving accurate annotation of images without human intervention, thereby improving the efficiency of image annotation. The method can be executed by an image annotation device provided by an embodiment of the present invention, and the device can be implemented in software and / or hardware. The following embodiments will be described by taking the device integrated in an electronic device as an example. The electronic device can be a server or a computer device, etc. Figure 1 , the method may specifically include the following steps:

[0029] Step 101: Acquire an image to be processed, and obtain labeling task information of the image to be processed according to the image to be processed and a predetermined multimodal large model.

[0030] Among them, the annotation task information includes at least one initial annotation information, and the initial annotation information includes at least one initial annotation point and the position information of each initial annotation point in the image to be processed. The initial annotation information is the annotation information obtained by the multimodal large model for image annotation of the image to be processed. The image to be processed is an image that needs to be annotated. The multimodal large model is a model that can process and understand various types of input data (such as text and images, etc.) and generate comprehensive outputs about these data. In this scheme, the multimodal large model can be a multimodal large model of each network connected to each model interface according to a pre-set one, or it can be a pre-established and trained visual question-answering large model. Specifically, after obtaining the image to be processed, the information to be input into the multimodal large model can be determined according to the basic information of the image to be processed (such as size information and image content, etc.), and the information and the image to be processed are input into the multimodal large model. The multimodal large model can intelligently generate annotation task information based on the input information and the image to be processed. In this scheme, optionally, labeling task information of the image to be processed is obtained according to the image to be processed and a predetermined multimodal large model, including: determining labeling guidance information according to the image to be processed and pre-acquired labeling requirement information; inputting the labeling guidance information into the multimodal large model to obtain the labeling task information output by the multimodal large model.

[0031] The annotation guidance information is used to guide the multimodal large model to annotate the image to be processed. The annotation requirement information includes the object information and annotation requirements that need to be annotated. The object information that needs to be annotated includes the objects, specific points, scenes or specific areas that need to be annotated in the image to be processed; the annotation requirements include the format requirements and the accuracy requirements of the annotation.

[0032] Specifically, when it is necessary to annotate the image to be processed, the image to be processed and the annotation requirement information are obtained, and the annotation guidance information of the image to be processed is generated according to the basic information of the image to be processed and the annotation requirement information. Exemplarily, the image to be processed is an image containing multiple pieces of clothing, and the annotation requirement information is: annotate the center point of each piece of clothing in the form of coordinates. According to the basic information of the image to be processed and the annotation requirement information, the annotation guidance information is determined as "the image size is 800 mm wide and 600 mm high, identify the clothes in the image, and output the coordinates of the center point of each piece of clothing".

[0033] After obtaining the annotation guidance information, the annotation guidance information and the image to be processed are input into the multimodal large model. After receiving the annotation guidance information, the multimodal large model performs text analysis on the annotation guidance information to analyze how to process the image to be processed. The image to be processed is processed according to the text analysis result of the annotation guidance information, and the image processing result is determined as the annotation task information. Exemplarily, Figure 2 Schematic diagram of an image to be processed provided by an embodiment of the present invention. Figure 2 As shown in the figure, the image to be processed includes two pieces of clothing. After the image to be processed and the annotation guidance information are input into the multimodal large model, the multimodal large model can analyze the image to be processed, and the annotation task information obtained includes two initial annotation information, namely, the initial annotation information of clothing 1 and the initial annotation information of clothing 2, that is, the coordinates of the center point of each clothing are obtained: the coordinates of the center point of the red clothing 1 are: (300, 400); the coordinates of the center point of the blue clothing 2 are: (400, 350).

[0034] In the above steps, the generalized image recognition capability of the multimodal large model can be used to perform preliminary recognition of the image to be processed and obtain approximate initial annotation information, which lays the foundation for obtaining accurate annotation points, saves manpower, and improves the efficiency of image annotation.

[0035] Step 102: for each initial annotation information, perform image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result.

[0036] Among them, the initial annotation information includes at least one initial annotation point and the position information of each initial annotation point in the image to be processed. The image segmentation model is used to perform fine segmentation on the image to be processed according to the initial annotation information output by the multimodal large model to obtain more accurate image annotation information. The segmentation result of the image to be processed includes the image area corresponding to the initial annotation point. The image segmentation model in this scheme can be a universal segmentation model (Segment Anything Model, SAM). In an optional embodiment, after obtaining each initial annotation information, the image segmentation model can be called multiple times to process each initial annotation information. For each initial annotation information, the current initial annotation information is input into SAM, and SAM performs image segmentation on the image to be processed according to the position information of each initial annotation point in the current initial annotation information, and obtains one or more binary masks (segmentation results) corresponding to the image to be processed, and the binary mask represents the pixel set belonging to the image area corresponding to the initial annotation point in the image to be processed.

[0037] Step 103: determine whether the segmentation result meets the preset conditions. If the segmentation result does not meet the preset conditions, optimize the current initial annotation information according to the segmentation result, the image segmentation model and the multimodal large model to obtain the target annotation information corresponding to the current initial annotation information.

[0038] Among them, the preset conditions are used to determine whether the initial annotation points corresponding to the segmentation results are accurate enough, or whether they meet the annotation requirements of the annotation requirement information. If the segmentation result meets the preset conditions, it means that the initial annotation points are accurate enough, and the initial annotation points can be directly determined as the target annotation points of the image to be processed. If the segmentation result does not meet the preset conditions, it means that the initial annotation points are not accurate enough, and it is necessary to continuously optimize the initial annotation information according to the segmentation result image segmentation model and the multimodal large model until the initial annotation information that meets the preset conditions is obtained, and the initial annotation information that meets the preset conditions is determined as the target annotation information. In an optional implementation, after obtaining the segmentation result, the multimodal large model can be used to evaluate the accuracy of the segmentation result to obtain an accuracy evaluation result of the segmentation result. Determine whether the segmentation result meets the preset conditions based on the accuracy evaluation result of the segmentation result.

[0039] In an optional embodiment, if the segmentation result does not meet the preset conditions, the initial annotation information is adjusted based on the accuracy evaluation information of the segmentation result to obtain candidate annotation information; the current initial annotation information is updated according to the candidate annotation information, and the step of performing image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model is repeated until the segmentation result meets the preset conditions, thereby obtaining the target annotation information of the image to be processed. Figure 3 Flow chart of iterative optimization of initial annotation information provided by an embodiment of the present invention. Figure 3 As shown, after obtaining the image to be processed, the image to be processed is input into the multimodal large model to obtain initial annotation information. The initial annotation information is input into the image segmentation model for image segmentation to obtain a segmentation result. The segmentation result is input into the multimodal large model to determine whether the segmentation result meets the preset conditions. If so, the initial annotation information is determined as the target annotation information. If the segmentation result does not meet the preset conditions, candidate annotation information is obtained according to the segmentation result, and the candidate annotation information is added to the initial annotation information. The step of inputting the initial annotation information into the image segmentation model for image segmentation to obtain the segmentation result is repeated until a segmentation result that meets the preset conditions is obtained to obtain the target annotation information.

[0040] Step 104 : Annotate the image to be processed according to the target annotation information corresponding to each initial annotation information.

[0041] The target annotation information may be coordinate information of the target annotation point, bounding box information of the target area, or other forms of information. After obtaining the target annotation information, the image to be processed is annotated according to the target annotation information and the annotation requirement information.

[0042] The technical solution of this embodiment obtains the image to be processed, and obtains the annotation task information of the image to be processed according to the image to be processed and the predetermined multimodal large model; wherein the annotation task information includes at least one initial annotation information; for each initial annotation information, the image to be processed is segmented based on the current initial annotation information and the pre-established image segmentation model to obtain the segmentation result; determines whether the segmentation result meets the preset conditions, and if the segmentation result does not meet the preset conditions, optimizes the current initial annotation information according to the segmentation result, the image segmentation model and the multimodal large model to obtain the target annotation information corresponding to the current initial annotation information; and annotates the image to be processed according to the target annotation information corresponding to each initial annotation information. The technical solution of this embodiment can use the multimodal large model to automatically generate the annotation information required by the image segmentation model, reducing the process of manual annotation. The initial annotation information is iteratively optimized according to the multimodal large model and the image segmentation model, and the image is accurately annotated through this coarse-to-fine optimization method, thereby improving the accuracy of image annotation. The image annotation method of this scheme realizes an efficient, accurate and automated image annotation process, reduces the cost of manual annotation, and improves the user experience.

[0043] Figure 4 This is a second flow chart of an image annotation method provided by an embodiment of the present invention. This embodiment is a refinement based on the above embodiment. The specific method can be as follows Figure 4 As shown, the method may include the following steps:

[0044] Step 401: Acquire an image to be processed, and obtain labeling task information of the image to be processed according to the image to be processed and a predetermined multimodal large model.

[0045] The labeling task information includes at least one initial labeling information.

[0046] Step 402: for each initial annotation information, perform image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result.

[0047] Step 403: update the image to be processed according to the segmentation result to obtain a rendered image of the image to be processed; analyze the rendered image through the multimodal large model to obtain the annotation offset information of the segmentation result.

[0048] Among them, the annotation offset information includes an offset and an offset direction. The size of the offset can indicate the accuracy of the initial annotation point corresponding to the segmentation result. The smaller the offset, the higher the accuracy of the segmentation result, and also indicates that the accuracy of the initial annotation point corresponding to the segmentation result is higher. The drawn image is used to intuitively display the segmentation result on the image to be processed so that the multimodal large model can evaluate the accuracy of the segmentation result. Specifically, after obtaining the segmentation result, the segmentation result (such as a binary mask) is superimposed on the image to be processed to obtain a new image, that is, the drawn image of the image to be processed. After obtaining the drawn image, the drawn image is analyzed using the multimodal large model to obtain the annotation offset information of the segmentation result. In this scheme, optionally, the drawn image is analyzed by the multimodal large model to obtain the annotation offset information of the segmentation result, including: determining the evaluation guidance information according to the drawn image; inputting the drawn image and the evaluation guidance information into the multimodal large model to obtain the annotation offset information output by the multimodal large model.

[0049] The evaluation guidance information is used to instruct the multimodal large model on how to evaluate the accuracy of the segmentation results. The evaluation guidance information in this solution includes the description information of the drawn image, the accuracy evaluation criteria of the segmentation results, and the evaluation accuracy. After obtaining the evaluation guidance information, the evaluation guidance information and the drawn image are input into the multimodal large model together. The multimodal large model can analyze the drawn image according to the evaluation guidance information and calculate the offset and offset direction of the segmentation result.

[0050] For example, Figure 5 A schematic diagram of a drawing image provided by an embodiment of the present invention. Figure 2 After obtaining the segmentation result corresponding to the first initial annotation information (the initial annotation information of clothing 1), the segmentation result is drawn on the image to be processed, and the result is as follows: Figure 5 The drawn image shown. Figure 5The dotted line in represents the contour of clothing 1 obtained according to the segmentation result. The evaluation guidance information is determined according to the drawn image as "confirm whether the contour of clothing drawn by the dotted line is accurate. If it is accurate, output no deviation. If it is inaccurate, output that there is deviation, and output the offset direction and offset of the center of the contour". The drawn image and the evaluation guidance information are input into the multimodal large model to obtain the labeled offset information output by the multimodal large model. If the multimodal large model outputs "no deviation", the offset is determined to be 0 and the offset direction is "none".

[0051] In the above steps, the accuracy of the segmentation results can be evaluated through a multimodal large model, which provides valuable feedback for further optimizing the annotation information and helps to gradually improve the overall accuracy of image segmentation and annotation.

[0052] Step 404: If the offset is greater than or equal to the preset offset, it is determined that the segmentation result does not meet the preset condition.

[0053] Among them, the preset offset is pre-determined based on domain big data and annotation requirement information, and is used to determine whether the segmentation result is accurate. The preset condition is used to determine whether the initial annotation point needs to be adjusted. If the segmentation result meets the preset condition, it means that there is no need to adjust the initial annotation point corresponding to the segmentation result, and the initial annotation point can be directly determined as the target annotation point. If the segmentation result does not meet the preset condition, it means that the initial annotation point corresponding to the segmentation result needs to be optimized. If the offset is greater than or equal to the preset offset, it means that the accuracy of the segmentation result is low, and it can be further determined that the segmentation result does not meet the preset condition.

[0054] Step 405: If the segmentation result does not meet the preset condition, candidate annotation information is obtained based on the annotation offset information of the segmentation result.

[0055] Specifically, if the segmentation result does not meet the preset conditions, it means that the initial annotation points are not accurate enough. After the segmentation result is obtained, the annotation points of the image to be processed are recalculated according to the offset amount and offset direction of the annotation offset information of the segmentation result to obtain candidate annotation information.

[0056] Exemplarily, the coordinates of the initial annotation points of the image to be processed are The offset of the annotation offset information is , the offset direction is (dx, dy), dx and dy represent the unit vector components in the x and y directions . , further obtained .

[0057] Step 406: update the current initial annotation information according to the candidate annotation information, and repeat the step of performing image segmentation on the image to be processed based on the current initial annotation information and the pre-established image segmentation model until the segmentation result meets the preset conditions, thereby obtaining the target annotation information of the image to be processed.

[0058] Among them, the initial annotation information includes at least one initial annotation point and the position information of each initial annotation point in the image to be processed; the candidate annotation information includes at least one candidate annotation point and the position information of each candidate annotation point in the image to be processed. In an optional embodiment, after obtaining the candidate annotation information, the candidate annotation information is added to the current initial annotation information to obtain the updated current initial annotation information. That is, the candidate annotation points and the initial annotation points before the update are both used as input information of the image segmentation model (updated initial identification information). The candidate annotation information and the initial annotation information before the update are input into the image segmentation model, and the image to be processed is segmented to obtain the segmentation result. The accuracy of the segmentation result is evaluated using a multimodal large model to determine whether the segmentation result meets the preset conditions. If the segmentation result does not meet the preset conditions, the updated initial annotation information is continued to be updated until a segmentation result that meets the preset conditions is obtained, and the initial annotation information corresponding to the segmentation result is determined as the target annotation information.

[0059] By iteratively updating the initial annotation information based on the multimodal large model and the image segmentation model, the accuracy of the segmentation results can be gradually improved, and ultimately high-precision target annotation information can be obtained.

[0060] Step 407: If the offset is less than the preset offset, determine that the segmentation result meets the preset condition.

[0061] Step 408: If the segmentation result meets the preset conditions, the current initial annotation information is determined as the target annotation information.

[0062] Step 409: Annotate the image to be processed according to the target annotation information.

[0063] In the technical solution of this embodiment, an image to be processed is obtained, and the labeling task information of the image to be processed is obtained according to the image to be processed and a predetermined multimodal large model; wherein the labeling task information includes at least one initial labeling information. For each initial labeling information, the image to be processed is segmented based on the current initial labeling information and a pre-established image segmentation model to obtain a segmentation result; the image to be processed is updated according to the segmentation result to obtain a drawn image of the image to be processed; the drawn image is analyzed by the multimodal large model to obtain the labeling offset information of the segmentation result, wherein the labeling offset information includes an offset and an offset direction; if the offset is greater than or equal to a preset offset, it is determined that the segmentation result does not meet the preset condition. If the segmentation result does not meet the preset condition, candidate labeling information is obtained based on the labeling offset information of the segmentation result; the current initial labeling information is updated according to the candidate labeling information, and the step of performing image segmentation on the image to be processed based on the current initial labeling information and a pre-established image segmentation model is repeatedly performed until the segmentation result meets the preset condition, and the target labeling information of the image to be processed is obtained. If the offset is less than the preset offset, it is determined that the segmentation result meets the preset condition. The technical solution of this embodiment uses a multimodal large model to automatically generate initial labeling information, thereby reducing the cost of manually labeling images. The image is finely segmented according to the initial annotation information and the image segmentation model, which improves the accuracy of image annotation. The initial annotation information is continuously updated based on the annotation offset information of the segmentation result, and the annotation information is gradually and automatically optimized to ensure the accuracy of image annotation while reducing the dependence on manual annotation. Especially when processing a large amount of image data, it significantly reduces the labor cost and improves the efficiency of image annotation.

[0064] Figure 6 FIG. 1 is a schematic diagram of the structure of an image annotation device provided in an embodiment of the present invention, and the device is suitable for executing the image annotation method provided in an embodiment of the present invention. Figure 6 As shown, the device may specifically include:

[0065] The image acquisition module 601 is used to acquire an image to be processed, and obtain the labeling task information of the image to be processed according to the image to be processed and a predetermined multimodal large model; wherein the labeling task information includes at least one initial labeling information;

[0066] An image segmentation module 602 is used to perform image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model for each initial annotation information to obtain a segmentation result;

[0067] An information optimization module 603 is used to determine whether the segmentation result meets a preset condition. If the segmentation result does not meet the preset condition, the current initial annotation information is optimized according to the segmentation result, the image segmentation model and the multimodal large model to obtain target annotation information corresponding to the current initial annotation information;

[0068] The image annotation module 604 is used to annotate the image to be processed according to the target annotation information corresponding to each initial annotation information.

[0069] Optionally, the image acquisition module 601 is specifically configured to: determine the annotation guidance information according to the image to be processed and the annotation requirement information acquired in advance;

[0070] The labeling guidance information is input into the multimodal large model to obtain labeling task information output by the multimodal large model.

[0071] Optionally, the information optimization module 603 is specifically used to: update the image to be processed according to the segmentation result to obtain a drawn image of the image to be processed;

[0072] Analyzing the drawn image by using the multimodal large model to obtain the annotation offset information of the segmentation result, wherein the annotation offset information includes an offset amount and an offset direction;

[0073] If the offset is less than the preset offset, it is determined that the segmentation result meets the preset condition; if the offset is greater than or equal to the preset offset, it is determined that the segmentation result does not meet the preset condition.

[0074] Optionally, the information optimization module 603 is further used to: determine evaluation guidance information according to the drawn image;

[0075] The drawn image and the evaluation guidance information are input into the multimodal large model to obtain the annotation offset information output by the multimodal large model.

[0076] Optionally, the information optimization module 603 is further configured to: if the segmentation result does not meet the preset condition, obtain candidate annotation information based on the annotation offset information of the segmentation result;

[0077] The current initial annotation information is updated according to the candidate annotation information, and the step of performing image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model is repeatedly performed until the segmentation result meets the preset condition, thereby obtaining the target annotation information of the image to be processed.

[0078] Optionally, the initial annotation information includes at least one initial annotation point and position information of each initial annotation point in the image to be processed; the candidate annotation information includes at least one candidate annotation point and position information of each candidate annotation point in the image to be processed; the information optimization module 603 is also used to: add the candidate annotation information to the current initial annotation information to obtain the updated current initial annotation information.

[0079] The image annotation device provided in the embodiment of the present invention can execute the image annotation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method. For the contents not described in detail in this embodiment, reference can be made to the description in any method embodiment of the present invention.

[0080] An embodiment of the present invention also provides a computer program product.

[0081] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer program products, which can include one or more computer programs, which can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor, which can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, referring to Figure 7 , Figure 7 The electronic device 12 shown is only an example and should not limit the functions and scope of use of the embodiments of the present application. Figure 7 As shown, the electronic device 12 is in the form of a general purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0083] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0084] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0085] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 7 not shown, usually called a "hard drive"). Although Figure 7 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0086] A program / utility 40 having a set (at least one) of program modules 46 may be stored, for example, in the memory 28, such program modules 46 including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 46 generally perform the functions and / or methods of the embodiments described herein.

[0087] The electronic device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the electronic device 12, and / or may communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the electronic device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that although Figure 7 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0088] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing an image annotation method provided by an embodiment of the present invention: obtaining an image to be processed, and obtaining annotation task information of the image to be processed according to the image to be processed and a predetermined multimodal large model; wherein the annotation task information includes at least one initial annotation information; for each initial annotation information, performing image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result; determining whether the segmentation result meets a preset condition, if the segmentation result does not meet the preset condition, optimizing the current initial annotation information according to the segmentation result, the image segmentation model and the multimodal large model to obtain target annotation information corresponding to the current initial annotation information; and annotating the image to be processed according to the target annotation information corresponding to each initial annotation information.

[0089] The embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, an image annotation method provided by all the embodiments of the present invention is implemented: obtaining an image to be processed, and obtaining annotation task information of the image to be processed according to the image to be processed and a predetermined multimodal large model; wherein the annotation task information includes at least one initial annotation information; for each initial annotation information, performing image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result; determining whether the segmentation result meets a preset condition, if the segmentation result does not meet the preset condition, optimizing the current initial annotation information according to the segmentation result, the image segmentation model and the multimodal large model to obtain the target annotation information corresponding to the current initial annotation information; performing image annotation on the image to be processed according to the target annotation information corresponding to each initial annotation information. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction-executing electronic device, apparatus, or device.

[0090] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction-executing electronic device, apparatus, or device.

[0091] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0092] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0093] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image annotation method, characterized in that: The method comprises: Acquire an image to be processed, and determine annotation guidance information according to the image to be processed and the annotation requirement information acquired in advance; input the annotation guidance information and the image to be processed into a multimodal large model to obtain annotation task information output by the multimodal large model; wherein the annotation task information includes at least one initial annotation information; For each initial annotation information, performing image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model to obtain a segmentation result; Determine whether the segmentation result meets the preset condition, and determine whether the segmentation result meets the preset condition, including: updating the image to be processed according to the segmentation result to obtain a drawn image of the image to be processed; determining evaluation guidance information according to the drawn image; inputting the drawn image and the evaluation guidance information into the multimodal large model to obtain the annotation offset information output by the multimodal large model, and obtaining the annotation offset information of the segmentation result, wherein the annotation offset information includes an offset and an offset direction; if the offset is less than a preset offset, it is determined that the segmentation result meets the preset condition; if the offset is greater than or equal to the preset offset, it is determined that the segmentation result does not meet the preset condition; if the segmentation result does not meet the preset condition, candidate annotation information is obtained based on the annotation offset information of the segmentation result; update the current initial annotation information according to the candidate annotation information, and repeatedly perform the step of performing image segmentation on the image to be processed based on the current initial annotation information and the pre-established image segmentation model until the segmentation result meets the preset condition, and the target annotation information of the image to be processed is obtained; The image to be processed is annotated according to the target annotation information corresponding to each initial annotation information.

2. The method according to claim 1, characterized in that The initial annotation information includes at least one initial annotation point and position information of each initial annotation point in the image to be processed; the candidate annotation information includes at least one candidate annotation point and position information of each candidate annotation point in the image to be processed; Updating the current initial annotation information according to the candidate annotation information includes: The candidate annotation information is added to the current initial annotation information to obtain updated current initial annotation information.

3. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements an image annotation method according to any one of claims 1 to 2.

4. An image annotation device, characterized in that: include: An image acquisition module, used to acquire an image to be processed, and determine annotation guidance information according to the image to be processed and the annotation requirement information acquired in advance; Inputting the annotation guidance information and the image to be processed into the multimodal large model to obtain the annotation task information output by the multimodal large model; wherein the annotation task information includes at least one initial annotation information; An image segmentation module is used to perform image segmentation on the image to be processed based on the current initial annotation information and a pre-established image segmentation model for each initial annotation information to obtain a segmentation result; The information optimization module is used to determine whether the segmentation result meets the preset conditions. The determination of whether the segmentation result meets the preset conditions includes: updating the image to be processed according to the segmentation result to obtain a drawn image of the image to be processed; determining evaluation guidance information according to the drawn image; inputting the drawn image and the evaluation guidance information into the multimodal large model to obtain the annotation offset information output by the multimodal large model, and obtaining the annotation offset information of the segmentation result, wherein the annotation offset information includes an offset and an offset direction; if the offset is less than a preset offset, it is determined that the segmentation result meets the preset conditions; if the offset is greater than or equal to the preset offset, it is determined that the segmentation result does not meet the preset conditions; if the segmentation result does not meet the preset conditions, candidate annotation information is obtained based on the annotation offset information of the segmentation result; updating the current initial annotation information according to the candidate annotation information, and repeatedly performing the step of performing image segmentation on the image to be processed based on the current initial annotation information and the pre-established image segmentation model until the segmentation result meets the preset conditions, and the target annotation information of the image to be processed is obtained; The image annotation module is used to annotate the image to be processed according to the target annotation information corresponding to each initial annotation information.

5. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the image annotation method according to any one of claims 1 to 2 is implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image annotation method as claimed in any one of claims 1 to 2 is implemented.

Images