Interactive image segmentation automatic labeling method supporting continuous clicking and electronic equipment

Abstract

The invention relates to an interactive image segmentation automatic labeling method supporting continuous clicking and electronic equipment, and belongs to the field of data labeling and artificial intelligence. The method comprises the following steps: inputting to-be-labeled image data into an automatic labeling system; the user judges whether a new image labeling task is started or not; if a new image labeling task is started, the system obtains continuous mouse click interaction information of the user; if the to-be-annotated image does not appear in the system, the system calls a back-end interactive image segmentation model supporting continuous clicking to calculate and obtain image feature vectors, the automatic annotation system calculates and obtains a segmentation result of the to-be-annotated image through the correlation degree between the image feature vectors, and then the system returns an automatic annotation result; in order to ensure the labeling quality, a labeling person checks a result obtained by the automatic labeling system, confirms a labeling result and completes labeling. According to the method, the data labeling time efficiency can be improved; the labor cost is reduced; and the annotation data volume is expanded, and more downstream artificial intelligence models are efficiently driven.

Description

Technical Field

[0001] The present invention belongs to the field of data annotation and artificial intelligence, and in particular relates to an image segmentation automatic annotation method and electronic device supporting continuous click interaction. Background Art

[0002] Artificial intelligence models require large-scale labeled data for training, and manual data labeling usually consumes a lot of manpower and material resources. The introduction of automatic data labeling products is precisely to help data labelers increase labeling efficiency and shorten labeling time. Although artificial intelligence models have shown strong migration and deductive learning capabilities, they still have problems such as the inability to recognize unknown classes and poor zero-sample recognition when facing the real physical world. Summary of the invention

[0003] 1. Technical issues to be resolved

[0004] The technical problem to be solved by the present invention is how to provide an image segmentation and automatic annotation method and electronic device that supports continuous click interaction, so as to solve the problems of unknown classes being unable to be recognized and poor zero-sample recognition effect when facing the real physical world.

[0005] (II) Technical solution

[0006] In order to solve the above technical problems, the present invention proposes an image segmentation and automatic annotation method supporting continuous click interaction, which comprises the following steps:

[0007] Step S101, the image data to be annotated is input into the automatic annotation system;

[0008] Step S102: The user determines whether to start a new image annotation task;

[0009] Step S103: if a new image annotation task is started, the system obtains the user's continuous mouse click interaction information;

[0010] Step S104: If the image to be annotated has not appeared in the system, the system calls the backend image segmentation model that supports continuous click interaction to calculate the image feature vector, and the automatic annotation system calculates the correlation between the image feature vectors to obtain the segmentation result of the image to be annotated, and then the system returns the automatic annotation result;

[0011] Step S105: To ensure the quality of annotation, the annotator reviews the results obtained by the automatic annotation system, confirms the annotation results and completes the annotation.

[0012] (III) Beneficial effects

[0013] The present invention proposes an image segmentation and automatic labeling method and electronic device that support continuous click interactivity. The present invention provides an image segmentation and automatic labeling method that supports continuous click interactivity, which can be used in the field of data labeling and can bring the following expected effects: 1) improving the time efficiency of data labeling; 2) reducing labor cost overhead; 3) expanding the amount of labeled data and efficiently driving more downstream artificial intelligence models. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flow chart of an image segmentation and automatic annotation method supporting continuous click interaction provided by the present invention;

[0015] Figure 2 It is a system framework diagram of the present invention;

[0016] Figure 3 is a flow chart of the system operation of the present invention;

[0017] Figure 4 is a schematic diagram of the structure of an electronic device;

[0018] Figure 5 A schematic diagram of a computer storable medium. DETAILED DESCRIPTION

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] The present invention adopts a click-based interactive image segmentation method for automatic image annotation, and uses the prior knowledge of the annotator as an auxiliary input to achieve a better automatic annotation effect. Based on this, the present invention applies for an image segmentation automatic annotation method, system and storage medium that supports continuous click interaction.

[0021] The present invention adopts an image segmentation model that supports continuous click interaction to provide data labeling personnel with automatic image segmentation labeling results, which can effectively simplify the user labeling process and improve data labeling efficiency.

[0022] The present invention proposes an image segmentation automatic annotation method, system and storage medium that support continuous click interaction, aiming to help annotation personnel improve the data annotation efficiency in image segmentation tasks.

[0023] The present invention provides potential results of image segmentation labels to annotators through an image segmentation model that supports continuous click interaction. Annotators can continuously add or delete feature points of potential mask areas of objects through mouse interaction. Finally, the annotators manually review the automatic labeling results and use the reviewed automatic labeling results as image labeling results.

[0024] The present invention relates to an image segmentation and automatic annotation system that supports continuous click interaction, which adopts B / S architecture mode, asynchronous resource scheduling, Docker container deployment and other technologies, and is committed to providing users with efficient, convenient and fast-response data automatic annotation services. While ensuring the accuracy of the annotation results, it can effectively reduce the annotation work cost and ultimately improve the data annotation efficiency of the annotation personnel.

[0025] Embodiment 1:

[0026] Figure 1 The present invention provides an image segmentation and automatic annotation method supporting continuous click interaction, the method comprising the following steps:

[0027] Step S101, the image data to be annotated is input into the automatic annotation system;

[0028] Step S102: The user determines whether to start a new image annotation task;

[0029] Step S103: if a new image annotation task is started, the system obtains the user's continuous mouse click interaction information;

[0030] Step S104: If the image to be annotated has not appeared in the system, the system calls the backend image segmentation model that supports continuous click interaction to calculate the image feature vector, and the automatic annotation system calculates the correlation between the image feature vectors to obtain the segmentation result of the image to be annotated, and then the system returns the automatic annotation result;

[0031] Step S105: To ensure the quality of annotation, the annotator reviews the results obtained by the automatic annotation system, confirms the annotation results and completes the annotation.

[0032] Figure 3 The implementation process of this application is demonstrated, and the above method steps are described in detail.

[0033] The present invention relates to an image segmentation model. The image segmentation model mainly includes three core components, namely: an image encoder, a prompt encoder and a mask decoder. The image encoder adopts a pre-trained visual model ViT architecture, which is responsible for generating a vectorized feature representation of an image; the prompt encoder can encode the interactive prompt information (mouse point position information) between the user and the image through the mouse into a feature vector; the mask encoder can combine the output features of the image encoder and the prompt encoder to predict the final image segmentation mask. The image segmentation model is trained on a large-scale image and mask dataset, so that the model has good data migration capabilities and can effectively adapt to custom datasets.

[0034] Embodiment 2:

[0035] The present invention also provides an electronic device corresponding to the method provided in the above-mentioned embodiment. Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the method provided by any of the aforementioned embodiments of the present application when running the computer program.

[0036] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0037] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0038] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0039] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0040] Embodiment 3:

[0041] The present invention also provides a computer-readable storage medium corresponding to the method provided in the above-mentioned embodiment, please refer to Figure 5 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0042] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0043] Embodiment 4:

[0044] A method, system and storage medium for automatic image segmentation annotation supporting continuous click interaction, which specifically provide automatic data annotation services for users.

[0045] Furthermore, the automatic annotation capability of image segmentation tasks is provided for the original image data to be annotated.

[0046] Furthermore, the automatic annotation results provided by the data automatic annotation product are adopted as the final annotation results after manual review.

[0047] Furthermore, the data automatic labeling capability is provided by an image segmentation model that supports continuous click interaction, which can support users' continuous mouse click information as model input, and the model outputs image segmentation automatic labeling results.

[0048] Furthermore, the data automatic annotation system adopts a B / S architecture model to provide users with a service calling interface.

[0049] Furthermore, the data automatic annotation system uses resource asynchronous scheduling on the server side to call the automatic annotation service.

[0050] Furthermore, the data automatic labeling product uses Docker containerization to deploy the image data automatic labeling service based on the multimodal model.

[0051] The present invention also provides a computer interface calling service program, which can satisfy the method described in any one of claims 1-7.

[0052] The present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any one of claims 1 to 8.

[0053] The present invention also provides a computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

[0054] The present invention adopts click-interactive image segmentation automatic annotation technology to provide automatic annotation capability for image segmentation tasks;

[0055] The present invention adopts a technology that supports continuous click interactive image segmentation and automatic annotation, so that the user can click continuously to mark the same target object. After the user completes the annotation of the same target object, he clicks the end button and selects the target object again to start a new round of annotation.

[0056] The present invention adopts the automatic annotation technology based on continuous click interactive image segmentation and has the ability to annotate unknown domain data;

[0057] The automatic image annotation system involved in the present invention adopts a B / S architecture mode, wherein the automatic annotation service uses an image segmentation model that supports continuous click interaction as a service backend, the model takes the mouse click point position information input by the user as input, and outputs the automatic annotation result of the image segmentation task, and the browser side serves as a client to receive the automatic annotation service result;

[0058] The automatic image annotation system involved in the present invention uses an asynchronous call method on the server side to schedule the automatic annotation service based on the image segmentation model that supports continuous click interaction;

[0059] The automatic image annotation system involved in the present invention adopts Docker containerization to deploy automatic annotation services, and realizes rapid deployment and operation across devices and platforms by mirroring the image segmentation model and the automatic annotation system.

[0060] The present invention provides an image segmentation automatic annotation method that supports continuous click interaction, which can be used in the field of data annotation and can bring the following expected effects: 1) improving the time efficiency of data annotation; 2) reducing labor cost overhead; 3) expanding the amount of annotated data and efficiently driving more downstream artificial intelligence models.

[0061] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for automatic image segmentation and annotation supporting continuous click interaction, characterized in that: The method comprises the following steps: Step S101, the image data to be annotated is input into the automatic annotation system; Step S102: The user determines whether to start a new image annotation task; Step S103: if a new image annotation task is started, the system obtains the user's continuous mouse click interaction information; Step S104: If the image to be annotated has not appeared in the system, the system calls the backend image segmentation model that supports continuous click interaction to calculate the image feature vector, and the automatic annotation system calculates the correlation between the image feature vectors to obtain the segmentation result of the image to be annotated, and then the system returns the automatic annotation result; Step S105: To ensure the quality of annotation, the annotator reviews the results obtained by the automatic annotation system, confirms the annotation results and completes the annotation.

2. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 1, characterized in that: Annotators can continuously add or delete feature points of the object's potential mask area through mouse interaction.

3. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 1, characterized in that: The image segmentation model consists of three core components: image encoder, hint encoder and mask decoder.

4. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 3, characterized in that: The image encoder uses the pre-trained visual model ViT architecture, which is responsible for generating vectorized feature representations of the image; The hint encoder encodes the interactive hint information between the user and the image through the mouse into a feature vector; the mask encoder combines the output features of the image encoder and the hint encoder to predict the final image segmentation mask.

5. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 4, characterized in that: The image segmentation model is trained on large-scale image and mask datasets, so that the model has good data migration capabilities and can effectively adapt to custom datasets.

6. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 3, characterized in that: The automatic annotation system adopts a B / S architecture model to provide users with a service call interface. The automatic annotation service uses a continuous click interactive image segmentation model as the service backend. The model takes the mouse click point location information input by the user as input and outputs the automatic annotation results of the image segmentation task. The browser acts as the client to receive the automatic annotation service results.

7. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 6, characterized in that: The automatic annotation system uses resource asynchronous scheduling on the server side to call the automatic annotation service.

8. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 6, characterized in that: The automatic annotation system uses Docker containerization to deploy automatic image data annotation services based on multimodal models.

9. The image segmentation and automatic annotation method supporting continuous click interaction as claimed in claim 6, characterized in that: The method adopts a technology that supports continuous click interactive image segmentation and automatic labeling, allowing users to continuously click to mark the same target object. After the user completes labeling of the same target object, he clicks the end button and selects the target object again to start a new round of labeling.

10. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 9.

Images