Image data labeling method and device

Abstract

The invention provides an image data annotation method and device, and belongs to the technical field of remote sensing satellite image processing, and the method comprises the steps: creating a sample annotation task, and adding at least one piece of image data into the sample annotation task; creating an entity object of the image data, and meanwhile, analyzing the image data to obtain target information of the image data; calling a map service engine to perform slice publishing on the image data, and obtaining a publishing server ID; updating the release server ID and the target information of the image data to the entity object, and loading and presenting the image data; marking a target in the image data, and carrying out quality inspection on the marked image data; and if the quality inspection is passed, generating sample annotation data according to the annotated image data. According to the image data labeling method and the labeling task, the image data is added, the asynchronous multi-thread task is adopted to analyze information such as width, height and geographic coordinates of the image data, and the information is published to map service, so that the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of remote sensing satellite image processing, and in particular to an image data labeling method and device. Background Art

[0002] At present, the commercialization of artificial intelligence has reached a stage of basic maturity in terms of computing power and algorithms. In order to better meet the needs of implementation and solve the specific pain points of the industry, a large amount of labeled sample data is needed to support algorithm training. With the acceleration of the commercialization of AI, AI companies have also put forward new requirements for data service providers. High-quality, refined, and customized data sets are increasingly favored by demanders, which has brought new challenges to the technical strength, refined management capabilities, and process control capabilities of the sample labeling industry.

[0003] Currently, more forward-looking dataset products and highly customized data services have become the mainstream of industry development. However, the current level of industry development is far from meeting these new demands. The sample labeling industry faces the following challenges: (1) Different industries and different business scenarios have different needs for data labeling. Existing labeling tasks are not detailed enough and lack customized labeling capabilities; (2) Labeling efficiency and data quality are both low, and there is a lack of human-computer collaboration capabilities; (3) Brand data labeling service providers rely on crowdsourcing and subcontracting models, resulting in uneven quality of labeling results. Summary of the invention

[0004] In view of this, the present invention provides an image data labeling method and device, which are used to solve the problems that the current data labeling method is single, cannot meet the personalized needs of users, has low processing efficiency and poor user experience.

[0005] In order to solve the above technical problems, in a first aspect, the present invention provides an image data labeling method, the method comprising: Creating a sample annotation task, adding at least one image data in the image data table to the sample annotation task; Creating an entity object of the image data, and at the same time, parsing the image data to obtain target information of the image data, the target information including geographic coordinate system, width, height, and geometric information; Calling the map service engine to publish the image data in slices and obtain the publishing server ID; updating the publishing server ID and the target information of the image data to the entity object, and loading and presenting the image data; Annotating the target in the image data, and performing quality inspection on the annotated image data; If the quality inspection passes, sample annotation data is generated based on the annotated image data.

[0006] Optionally, the step of creating the entity object of the image data includes: Acquire metadata of the image data from an image data management system; According to the metadata of the image data, an entity object of the image data is created, and entity information is added to the entity object.

[0007] Optionally, the status of the image data includes newly loaded, to be marked, marked, to be quality inspected, passed quality inspection, failed quality inspection, to be corrected, and completed, and the method further includes: The status of the image data is updated according to different stages of the image data in the labeling process.

[0008] Optionally, the status of the sample labeling task includes initialization, pending labeling, pending quality inspection, pending correction, and completion, and the method further includes: According to different stages of the sample labeling task in the labeling process, the status of the labeling task is updated.

[0009] Optionally, the step of performing quality inspection on the annotated image data includes: When the status of all image data in the sample labeling task is labeled, quality inspection is performed on all labeled image data in the sample labeling task.

[0010] Optionally, the step of creating a sample annotation task includes: At least one target label in the target labeling table is added to the sample labeling task, wherein the target label is used to indicate the category of the target required to be labeled under the sample labeling task, wherein different target labels have different colors.

[0011] In a second aspect, the present invention further provides an image data annotation device, the device comprising: A creation module, used for creating a sample annotation task, and adding at least one image data in the image data table to the sample annotation task; An asynchronous processing module, used to create an entity object of the image data, and at the same time, parse the image data to obtain target information of the image data, wherein the target information includes geographic coordinate system, width, height, and geometric information; A publishing module is used to call the map service engine to slice and publish the image data and obtain the publishing server ID; An updating module, used for updating the publishing server ID and the target information of the image data to the entity object, and loading and presenting the image data; A labeling quality inspection module, used to label the objects in the image data and perform quality inspection on the labeled image data; The generation module is used to generate sample annotation data based on the annotated image data if the quality inspection passes.

[0012] Optionally, the asynchronous processing module includes: An acquisition unit, used for acquiring metadata of the image data from an image data management system; The creation unit is used to create an entity object of the image data according to the metadata of the image data, and add entity information to the entity object.

[0013] Optionally, the status of the image data includes newly loaded, to be marked, marked, to be quality inspected, passed quality inspection, failed quality inspection, to be corrected, and completed, and the device further includes: The first updating module is used to update the status of the image data according to different stages of the image data in the labeling process.

[0014] Optionally, the status of the sample labeling task includes initialization, pending labeling, pending quality inspection, pending correction, and completion, and the device further includes: The second updating module is used to update the status of the labeling task according to different stages of the labeling process of the sample labeling task.

[0015] Optionally, the labeling quality inspection module includes: The quality inspection unit is used to perform quality inspection on all the labeled image data in the sample labeling task when the status of all the image data in the sample labeling task is that they are labeled.

[0016] Optionally, the creation module includes: An adding unit is used to add at least one target label in the target labeling table to the sample labeling task, wherein the target label is used to indicate the category of the target required to be labeled under the sample labeling task, wherein different target labels have different colors.

[0017] The beneficial effects of the above technical solution of the present invention are as follows: In the embodiment of the present invention, the sample annotation task table, image data table, and target annotation table are designed separately, with clear logic, convenient for splitting engine service components, and reducing the coupling between components; the annotation task adds image data, and uses asynchronous multi-threaded tasks to parse information such as width, height, and geographic coordinates of the image data, and publishes it to the map service, which speeds up the processing speed and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1A schematic diagram of a process of an image data labeling method in Embodiment 1 of the present invention; Figure 2 A schematic diagram of state changes of a sample labeling task provided in the first embodiment of the present invention; Figure 3 A schematic diagram of state changes of image data provided by Embodiment 1 of the present invention; Figure 4 A schematic diagram of a state change of a target label provided in the first embodiment of the present invention; Figure 5 It is a structural schematic diagram of an image data labeling device in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0020] At present, the commercialization of artificial intelligence has reached a stage of basic maturity in terms of computing power and algorithms. In order to better meet the needs of implementation and solve the specific pain points of the industry, a large amount of labeled sample data is needed to support algorithm training. With the acceleration of the commercialization of AI, AI companies have also put forward new requirements for data service providers. High-quality, refined, and customized data sets are increasingly favored by demanders, which has brought new challenges to the technical strength, refined management capabilities, and process control capabilities of the sample labeling industry.

[0021] Currently, more forward-looking dataset products and highly customized data services have become the mainstream of industry development. However, the current level of industry development is far from meeting these new demands. The sample annotation industry faces the following challenges: (1) Different industries and business scenarios have different requirements for data annotation. Existing annotation tasks are not detailed enough and lack customized annotation capabilities. (2) The labeling efficiency and data quality are both low, and there is a lack of human-machine collaboration capabilities; (3) Brand data labeling service providers rely on crowdsourcing and subcontracting models, resulting in uneven quality of labeling results.

[0022] The existing sample annotation system has imperfect functional design, unsmooth processes, and too strong storage design coupling, which cannot meet the personalized needs of customers. Specifically, there are the following problems: (a) The product function design does not meet user needs and the product logic is not smooth; (b) The storage structure design is highly coupled and difficult to change; (c) The design of the server-side engine component combines the labeling task, image data processing, target labeling, and tag management together, which is highly coupled and not conducive to meeting user customization requirements; (d) Label data is fixed data and does not meet the requirements of label group and classification customization.

[0023] Therefore, please refer to Figure 1 , Figure 1 A flowchart of an image data annotation method provided in Embodiment 1 of the present invention includes the following steps: Step 11: Create a sample annotation task, and add at least one image data in the image data table to the sample annotation task.

[0024] In this embodiment, the image data table records all image data. When some image data need to be labeled, corresponding image data are selected from the image data table and added to the sample labeling task.

[0025] Optionally, the image data table is stored in a data management system. In addition to the image data table, the image data management system also includes an image metadata table, a storage directory table, and the like.

[0026] Step 12: Create an entity object of the image data, and at the same time, parse the image data to obtain target information of the image data, wherein the target information includes geographic coordinate system, width, height, and geometric information.

[0027] In this embodiment, asynchronous multi-threaded processing is used when adding image data to the sample annotation task. Specifically, since it is necessary to parse the CRS (Coordinate Reference System) information, width, height, geometry information / geometric information, etc. of the image data when adding image data to the sample annotation task, and at the same time, it is necessary to publish the image data to the map service and obtain the publishing server ID. This process takes a long time and the user experience is not good, so the server uses asynchronous multi-threaded processing, and the server automatically handles it without affecting the user experience. In other words, when adding image data, while creating the entity object of the image data, the image data is also parsed synchronously, thereby avoiding the problem of slow task interface response and poor user experience caused by waiting for the image data to be parsed before performing the creation action.

[0028] Step 13: Call the map service engine to publish the image data in slices and obtain the publishing server ID.

[0029] Satellite image slicing service is an efficient map data processing method that divides the original satellite image data into multiple small blocks (also called tiles or slices) and stores and publishes them according to certain rules and formats. This can quickly respond to user requests and reduce latency; by adjusting the resolution and level of tiles, it can meet the map display requirements under different devices and network conditions.

[0030] Step 14: Update the publishing server ID and the target information of the image data to the entity object, and load and present the image data.

[0031] In this step, due to asynchronous processing, after the target information is parsed and the publishing server ID is published, the information is updated to the entity object, so that the image data is loaded and presented in the task interface. That is to say, in this embodiment, when the image data is loaded, the image data parsing and processing engine will parse the basic information of the image data and schedule the map service engine to publish the image data. After the image data is published, the front end can load and present the image data.

[0032] Step 15: labeling the targets in the image data, and performing quality inspection on the labeled image data.

[0033] In this embodiment, the targets in the image data can be labeled by manual labeling or automatic labeling based on target recognition, wherein the labeling can be performed according to the specific labeling requirements of the sample labeling task, for example, the labeled targets are only the aircraft in the image data, or the labeled targets include ships, aircraft, etc. in the image data.

[0034] After the labeling is completed, the labeled image data can be quality checked. Similarly, in this embodiment, a manual quality check method can be used to check whether the labeled target in the image data is correct, or an automatic quality check method can be used to check whether the labeled target in the image data is correct, which is not specifically limited in this embodiment.

[0035] Step 16: If the quality inspection passes, generate sample annotation data based on the annotated image data.

[0036] In this embodiment, when the image data quality inspection is completed, the original files corresponding to the sample annotation task can be generated, including shpfile and geojson files. The subsequent generation of sample sets will generate sample set images and labels files based on the original image files, shpfile files and geojson file slices, and store them in the sample library.

[0037] Therefore, the image data annotation method provided by the embodiment of the present invention designs the sample annotation task table, the image data table, and the target annotation table separately, with clear logic, convenient for splitting the engine service components, and at the same time reducing the coupling between components; the annotation task adds image data, and uses asynchronous multi-threaded tasks to parse information such as the width, height, and geographic coordinates of the image data, and publishes it to the map service, which speeds up the processing speed and improves the user experience.

[0038] The following example illustrates the above image data labeling method.

[0039] In one optional specific implementation manner, the step of creating the entity object of the image data includes: Acquire metadata of the image data from an image data management system; According to the metadata of the image data, an entity object of the image data is created, and entity information is added to the entity object.

[0040] In this embodiment, when adding image data, that is, the web end requests to add image data, metadata of the corresponding image data will be obtained from the image data management system, more specifically, metadata of the corresponding image data will be obtained from the image metadata table in the image data management system. Then, based on the metadata of the obtained image data, an entity object of the image data to be annotated is created, and entity information is added to the entity object.

[0041] In some embodiments, optionally, the status of the sample labeling task includes initialization, pending labeling, pending quality inspection, pending correction, and completion, and the method further includes: According to different stages of the sample labeling task in the labeling process, the status of the labeling task is updated.

[0042] In this embodiment, the status of the sample labeling task includes initialization, waiting for labeling, waiting for quality inspection, waiting for correction, and completion, a total of 5 states. The status of the sample labeling task will be updated in real time as it progresses through different stages of the labeling process, making it convenient for users to understand whether the sample labeling task has been completed, whether it is completed, whether it needs correction, etc.

[0043] See also Figure 2 , Figure 2 Schematic diagram of the state change of a sample labeling task provided in the first embodiment of the present invention. Figure 2As shown, when the sample annotation task is just created, it is in the initialization state; after the sample annotation task is edited, it is in the waiting for annotation state to remind the user to annotate it; after all the image data in the sample annotation task are annotated, it can be submitted for quality inspection. After submitting for quality inspection, it is in the waiting for quality inspection state to remind the user to conduct quality inspection on it; after the quality inspection is completed, the quality inspection result can be submitted. If the quality inspection passes, it is in the completion state. If the quality inspection fails, it is in the waiting for correction state, prompting the user to continue to make corrections.

[0044] In some other embodiments, optionally, the status of the image data includes newly loaded, to be labeled, labeled, to be quality inspected, passed quality inspection, failed quality inspection, to be corrected, and completed, and the method further includes: The status of the image data is updated according to different stages of the image data in the labeling process.

[0045] In this embodiment, the status of image data includes newly loaded, to be labeled, labeled, to be quality inspected, passed quality inspection, failed quality inspection, to be corrected, and completed, a total of 8 states. The status of image data will be updated in real time as it progresses through different stages of the labeling process, making it easy for users to understand whether any image data has been labeled, whether it is completed, whether it needs to be corrected, etc.

[0046] See also Figure 3 , Figure 3 The following is a schematic diagram of the state change of image data provided by the first embodiment of the present invention. Figure 3 As shown in the figure, when the image data is just added, it is in the new loading state; after the loading is completed and saved, it is in the waiting for annotation state to remind the user to annotate it; after the preliminary confirmation that the target annotation in the image data is completed, it is in the labeled state; after submitting for quality inspection, if the quality inspection finds that there are unlabeled targets, the state of the image data is switched back to the waiting for annotation state, and a prompt "there are unlabeled data" is prompted; if the quality inspection does not find any unlabeled targets, the state of the image data is marked as waiting for quality inspection; after the quality inspection is completed (the quality inspection here refers to whether the quality inspection annotation is correct), the quality inspection result can be submitted. If the quality inspection passes, it is in the completed state. If the quality inspection fails, it is in the waiting for correction state, prompting the user to continue to make corrections.

[0047] See also Figure 4 , Figure 4 The following is a schematic diagram of the state change of a target label provided in the first embodiment of the present invention. Figure 4As shown, in some embodiments, optionally, the target annotation includes three states: waiting for quality inspection, passing quality inspection, and failing quality inspection. The state of the target annotation determines the state of the image data. When one of the target annotations fails the quality inspection, the image data state is failing the quality inspection; when all the target annotation states pass, the image data state is passing the quality inspection. Among them, the target annotation is the annotation made on the target.

[0048] In some embodiments, optionally, the step of performing quality inspection on the annotated image data includes: When the status of all image data in the sample labeling task is labeled, quality inspection is performed on all labeled image data in the sample labeling task.

[0049] Specifically, when submitting for quality inspection, it can only be submitted for quality inspection after all image data have been labeled.

[0050] Optionally, when submitting the quality inspection results, the quality inspection results can only be submitted after all the image data have been inspected. When submitting the quality inspection results, if the image data status is passed, after submitting the quality inspection results, the image data status changes to completed; if the image data status is failed, after submitting the quality inspection results, the image data status changes to pending correction.

[0051] In some other embodiments, the step of creating a sample annotation task includes: At least one target label in the target labeling table is added to the sample labeling task, wherein the target label is used to indicate the category of the target required to be labeled under the sample labeling task, wherein different target labels have different colors.

[0052] That is to say, the required target labels can be added for the sample labeling task. Specifically, the target labeling table includes various target labels, and different labels have different classifications and colors. In this way, the personalized needs of users can be met. For example, the target labels can be divided into aircraft, ships, etc.

[0053] In this embodiment, optionally, the architecture for implementing this method is divided into four parts: storage layer, engine layer, application service layer, and web front-end presentation. Among them, the storage layer mainly completes the storage of sample annotated image data, thumbnails, shpfile files, geojson files, and business data storage and management. Including shared disk storage directory, postgresql, elasticsearch. The engine layer mainly completes the business processing and state flow of annotation tasks, image data analysis, image data processing and state flow, target annotation data processing, and image data slicing in different formats. The application service layer mainly provides service APIs to the web front-end, including annotation tasks, image data management, and target annotation service APIs. The web front-end mainly completes API data presentation and UI interaction.

[0054] In this embodiment, when creating a labeling task, image data will be loaded, and the image data comes from the image data management system. At the same time, the labeling task engine will synchronize the image data from the image data management system to the image data directory / images, and generate image thumbnails and store them in the / thumb directory. When the image data is loaded, the image data parsing and processing engine will perform basic information parsing of the image data, and schedule the map service engine to publish the image data. After the image data is published, the front end can load and present the image data. Only after the image data is loaded by the front end can the target labeling stage be entered. In the target labeling stage, the target labeling engine processes the target labeling data management, labeling and quality inspection status management of the image data. When the image data quality inspection is completed, the original file of the image labeling can be generated, including shpfile and geojson files. The subsequent generation of the sample set will generate the sample set images and labels files based on the original image file, shpfile file and geojson file slices, and store them in the sample library.

[0055] In the embodiment of the present invention, the sample annotation task table, image data table, and target annotation table are designed separately, with clear logic, convenient for splitting engine service components, and reducing the coupling between components; in the storage layer, the image data, thumbnails, shpfile files, geojson, segmented data, etc. under the same task are placed in the corresponding task directory, which is convenient for subsequent data permission control and data migration; the annotation task adds image data, and uses asynchronous multi-threaded tasks to parse the width, height, geographic coordinates and other information of the image data, and publishes it to the map service, which improves the user experience.

[0056] See also Figure 5 , Figure 5 is a schematic diagram of the structure of an image data labeling device provided in Embodiment 2 of the present invention. The device 50 includes: A creation module 51, used to create a sample annotation task, and add at least one image data in the image data table to the sample annotation task; The asynchronous processing module 52 is used to create a physical object of the image data and parse the image data to obtain target information of the image data, wherein the target information includes geographic coordinate system, width, height and geometric information; The publishing module 53 is used to call the map service engine to slice and publish the image data and obtain the publishing server ID; An updating module 54, used to update the publishing server ID and the target information of the image data to the entity object, and load and present the image data; A labeling quality inspection module 55, used to label the objects in the image data and perform quality inspection on the labeled image data; The generating module 56 is used to generate sample annotation data according to the annotated image data if the quality inspection is passed.

[0057] Optionally, the asynchronous processing module includes: An acquisition unit, used for acquiring metadata of the image data from an image data management system; The creation unit is used to create an entity object of the image data according to the metadata of the image data, and add entity information to the entity object.

[0058] Optionally, the status of the image data includes newly loaded, to be marked, marked, to be quality inspected, passed quality inspection, failed quality inspection, to be corrected, and completed, and the device further includes: The first updating module is used to update the status of the image data according to different stages of the image data in the labeling process.

[0059] Optionally, the status of the sample labeling task includes initialization, pending labeling, pending quality inspection, pending correction, and completion, and the device further includes: The second updating module is used to update the status of the labeling task according to different stages of the labeling process of the sample labeling task.

[0060] Optionally, the labeling quality inspection module includes: The quality inspection unit is used to perform quality inspection on all the labeled image data in the sample labeling task when the status of all the image data in the sample labeling task is that they are labeled.

[0061] Optionally, the creation module includes: An adding unit is used to add at least one target label in the target labeling table to the sample labeling task, wherein the target label is used to indicate the category of the target required to be labeled under the sample labeling task, wherein different target labels have different colors.

[0062] In the embodiment of the present invention, the sample annotation task table, image data table, and target annotation table are designed separately, with clear logic, convenient for splitting engine service components, and reducing the coupling between components; in the storage layer, the image data, thumbnails, shpfile files, geojson, segmented data, etc. under the same task are placed in the corresponding task directory, which is convenient for subsequent data permission control and data migration; the annotation task adds image data, and uses asynchronous multi-threaded tasks to parse the width, height, geographic coordinates and other information of the image data, and publishes it to the map service, which improves the user experience.

[0063] The embodiment of the present invention is a product embodiment corresponding to the above-mentioned method embodiment 1, so it will not be described in detail here. Please refer to the above-mentioned embodiment 1 for details.

[0064] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for labeling image data, characterized in that: include: Creating a sample annotation task, adding at least one image data in the image data table to the sample annotation task; Creating an entity object of the image data, and at the same time, parsing the image data to obtain target information of the image data, the target information including geographic coordinate system, width, height, and geometric information; Calling the map service engine to publish the image data in slices and obtain the publishing server ID; updating the publishing server ID and the target information of the image data to the entity object, and loading and presenting the image data; Annotating the target in the image data, and performing quality inspection on the annotated image data; If the quality inspection passes, sample annotation data is generated based on the annotated image data.

2. The method according to claim 1, characterized in that The step of creating the entity object of the image data comprises: Acquire metadata of the image data from an image data management system; According to the metadata of the image data, an entity object of the image data is created, and entity information is added to the entity object.

3. The method according to claim 1, characterized in that: The states of the image data include newly loaded, to be marked, marked, to be quality inspected, passed quality inspection, failed quality inspection, to be corrected, and completed. The method further includes: The status of the image data is updated according to different stages of the image data in the labeling process.

4. The method according to claim 1, characterized in that The status of the sample labeling task includes initialization, pending labeling, pending quality inspection, pending correction, and completion. The method further includes: According to different stages of the sample labeling task in the labeling process, the status of the labeling task is updated.

5. The method according to claim 3, characterized in that: The step of performing quality inspection on the annotated image data comprises: When the status of all image data in the sample labeling task is labeled, quality inspection is performed on all labeled image data in the sample labeling task.

6. The method according to claim 1, characterized in that The steps of creating a sample annotation task include: At least one target label in the target labeling table is added to the sample labeling task, wherein the target label is used to indicate the category of the target required to be labeled under the sample labeling task, wherein different target labels have different colors.

7. An image data labeling device, characterized in that: include: A creation module, used for creating a sample annotation task, and adding at least one image data in the image data table to the sample annotation task; An asynchronous processing module, used to create an entity object of the image data, and at the same time, parse the image data to obtain target information of the image data, wherein the target information includes geographic coordinate system, width, height, and geometric information; A publishing module is used to call the map service engine to slice and publish the image data and obtain the publishing server ID; An updating module, used for updating the publishing server ID and the target information of the image data to the entity object, and loading and presenting the image data; A labeling quality inspection module, used to label the objects in the image data and perform quality inspection on the labeled image data; The generation module is used to generate sample annotation data based on the annotated image data if the quality inspection passes.

8. The device according to claim 7, characterized in that The asynchronous processing module includes: An acquisition unit, used for acquiring metadata of the image data from an image data management system; The creation unit is used to create an entity object of the image data according to the metadata of the image data, and add entity information to the entity object.

9. The device according to claim 7, characterized in that The states of the image data include newly loaded, to be marked, marked, to be inspected, passed the quality inspection, failed the quality inspection, to be corrected, and completed. The device also includes: The first updating module is used to update the status of the image data according to different stages of the image data in the labeling process.

10. The device according to claim 7, characterized in that The status of the sample labeling task includes initialization, pending labeling, pending quality inspection, pending correction, and completion. The device also includes: The second updating module is used to update the status of the labeling task according to different stages of the labeling process of the sample labeling task.

Images