Image annotation method and device, computing equipment and computer storage medium

By generating an initial mask image and filtering out noisy regions using preset area filtering conditions, a circumscribed rectangle is constructed to generate the target mask image. This solves the problems of high cost and low accuracy in image annotation, and achieves efficient and accurate automated annotation.

CN118968513APending Publication Date: 2024-11-15GUANGDONG AOPUTE TECH CO LTD

Patent Information

Application Number
CN202410872452.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-01
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing technology for image annotation suffers from high cost and low accuracy, especially in semi-automatic annotation algorithms, where manual intervention is required, leading to high annotation costs. Meanwhile, semi-automatic annotation based on AI models is not precise enough.

Method used

By generating an initial mask image and labeling the region of interest, filtering out segmented regions that do not meet the conditions using preset area filtering conditions, constructing a circumscribed rectangle and generating a target mask image, automated and refined image labeling is achieved.

Benefits of technology

It significantly improves the accuracy and efficiency of annotation, reduces manpower requirements, lowers annotation costs, and is suitable for efficient processing of large-scale datasets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118968513A_ABST
    Figure CN118968513A_ABST
Patent Text Reader

Abstract

The invention provides an image annotation method and device, computing equipment and a computer storage medium. The method comprises the following steps: generating an initial mask image according to an input query object and a to-be-labeled image in an acquired to-be-labeled data set, and labeling an initial segmentation region in the initial mask image; filtering out the initial segmentation regions which do not accord with a condition through a preset area filtering condition, and obtaining the remaining initial segmentation regions; and constructing a bounding rectangle corresponding to the residual initial segmentation region, generating a target mask image based on the bounding rectangle and the to-be-labeled image, and labeling a target segmentation region of the query object in the to-be-labeled image in the target mask image. According to the technical scheme provided by the invention, the labeling cost can be reduced, and the labeling accuracy can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image annotation method, apparatus, computing device, and computer storage medium. Background Art

[0002] As a core technology in the field of artificial intelligence, image annotation is crucial to improving the automation and intelligence level of industrial product inspection. It involves manual or automatic labeling of image data to provide guidance for machine learning and deep learning models, enabling the models to identify, classify and locate specific objects, features or defects in images. In the scenario of industrial product inspection, image annotation is widely used in product appearance, size, and defect detection. Through high-precision and high-speed artificial intelligence (AI) visual inspection algorithms, it ensures product quality control and improves production efficiency and customer satisfaction.

[0003] At present, the common labeling scheme usually uses AI visual inspection algorithms to realize intelligent inspection of product appearance, size, defects, etc. It can achieve high-speed, high-precision automated inspection and quality control, greatly shorten the inspection time, reduce the cost of manual inspection, and improve overall production efficiency. At the same time, it can also reduce the inspection errors caused by human errors. Compared with traditional manual inspection, this method is faster, more accurate and unbiased. It greatly improves the inspection efficiency and consistency. However, the performance and effect of the AI ​​model depend to a large extent on the quality and quantity of training data. Large-scale data training can enhance the robustness of the AI ​​model, improve its stability and reliability in complex environments, and reduce the probability of errors. However, large-scale labeling means higher labeling costs, and manual labeling is a time-consuming and labor-intensive task. Through automated or semi-automated labeling, the manpower input can be greatly reduced, thereby reducing the labeling cost.

[0004] However, the current semi-automatic annotation algorithms mainly include the following: based on the segmentation model (SegmentAnythingMode l, SAM), the grab cut algorithm (Grab-cut), the graph cut algorithm (Graph-cut) and other semi-automatic annotation methods. This type of algorithm is to manually set the initial point, box, or area for annotation. The advantage of this method is that the annotation is accurate and it also shortens the time of most manual outline drawing. However, the semi-automatic algorithm still needs manual annotation when encountering unlabeled images, resulting in high annotation costs. The other is a semi-automatic annotation algorithm based on an AI model, which loads a pre-trained model to label the image to be labeled, and manually checks and adjusts the labels. This method can automatically and semi-automatically annotate images without manual intervention, but the disadvantage is that the annotation is not fine enough. Summary of the invention

[0005] The embodiments of the present application provide an image annotation method, apparatus, computing device, and computer storage medium to solve the problems of high annotation cost and low annotation accuracy in the prior art.

[0006] In a first aspect, an embodiment of the present application provides an image annotation method, comprising:

[0007] Generate an initial mask image according to the input query object and the image to be annotated in the acquired dataset to be annotated, and the initial mask image is annotated with a region of interest, where the region of interest refers to an initial segmentation region of the query object in the image to be annotated;

[0008] Filtering out the initial segmentation regions that do not meet the conditions by using a preset area filtering condition to obtain the remaining initial segmentation regions, wherein the preset area filtering condition is constructed based on the mask region area of ​​the query object and the mask region area in the initial segmentation region;

[0009] A bounding rectangle corresponding to the remaining initial segmented area is constructed, a target mask image is generated based on the bounding rectangle and the image to be annotated, and a target segmented area of ​​the query object in the image to be annotated is annotated in the target mask image.

[0010] Optionally, generating an initial mask image according to the input query object and the image to be annotated in the acquired data set to be annotated includes:

[0011] Determining an input query object, the query object is used to retrieve matching images to be annotated from the dataset to be annotated, wherein the query object includes an image segment;

[0012] Taking the query object as input, extracting an object feature vector of the query object through a first feature extraction network;

[0013] Acquire a data set to be labeled, wherein the data set to be labeled includes images to be labeled;

[0014] Taking the image to be annotated in the to-be-annotated data set as input, so as to extract an image feature vector of the image to be annotated through a second feature extraction network;

[0015] For each of the images to be annotated, a corresponding initial mask image is generated according to the object feature vector and the image feature vector of the image to be annotated.

[0016] Optionally, filtering out the initial segmentation areas that do not meet the conditions by using a preset area filtering condition to obtain the remaining initial segmentation areas includes:

[0017] Calculating the mask area of ​​the query object and the mask area in the initial segmented area, wherein the mask area of ​​the query object is determined according to the number of pixels included in the query object, and the mask area in the initial segmented area is determined according to the number of pixels included in the initial segmented area;

[0018] Filter by preset area condition: τ*S Q ≤S k ≤(1 / τ)S Q , filter less than or equal to τ*S Q The initial segmentation area of ​​the area, and greater than or equal to (1 / τ)S Q The initial segmentation region of the area is obtained to obtain the remaining initial segmentation region;

[0019] Among them, τ represents the area filtering threshold coefficient, τ<1, S Q It is represented as the mask area of ​​the query object, S k It is represented as the area of ​​the mask region in the initial segmentation region.

[0020] Optionally, the generating a target mask image based on the bounding rectangle corresponding to the remaining initial segmented area and the image to be labeled includes:

[0021] Determine the coordinate values ​​of the circumscribed rectangle corresponding to the remaining initial segmented area, and input the coordinate values ​​of the circumscribed rectangle into a third feature extraction network to extract a rectangular feature vector of the circumscribed rectangle through the third feature extraction network;

[0022] A target mask image is generated based on the rectangular feature vector and the image feature vector of the image to be labeled.

[0023] Optionally, it also includes:

[0024] Determine a task type of the image annotation task, where the task type includes at least a target detection task and an instance segmentation task;

[0025] If the task type includes a target detection task, updating the minimum bounding rectangle corresponding to the remaining initial segmented area;

[0026] If the task type includes an instance segmentation task, the minimum bounding rectangle corresponding to the remaining initial segmented area is updated, and the target mask image is updated based on the minimum bounding rectangle.

[0027] Optionally, after generating the target mask image and marking the target segmentation area of ​​the query object in the image to be marked in the target mask image, the method further includes:

[0028] Determining an object category of the query object;

[0029] The category of the query object is marked in the target segmentation area for manual identification and calibration.

[0030] Optionally, it also includes:

[0031] The target mask image is adjusted, and the adjustment at least includes: using a preset fine-tuning tool to fine-tune the target mask image or delete an erroneous object category;

[0032] The preset fine-tuning tools include at least: a polygon tool, an area pen tool, an eraser tool, a circle tool, a polyline tool, a fill tool, a three-point drawing tool, an ellipse tool and a rectangle tool.

[0033] In a second aspect, an embodiment of the present application provides an image annotation device, including:

[0034] A generating module, configured to generate an initial mask image according to an input query object and an image to be annotated in the acquired dataset to be annotated, wherein the initial mask image is annotated with a region of interest, and the region of interest refers to an initial segmented region of the query object in the image to be annotated;

[0035] A filtering module, used to filter out the initial segmented areas that do not meet the conditions through a preset area filtering condition to obtain the remaining initial segmented areas, wherein the preset area filtering condition is constructed based on the mask area area of ​​the query object and the mask area area in the initial segmented areas;

[0036] A construction module, used to construct a circumscribed rectangle corresponding to the remaining initial segmented area;

[0037] The generating module is further configured to generate a target mask image based on the circumscribed rectangle and the image to be annotated, and annotate the target segmentation region of the query object in the image to be annotated in the target mask image.

[0038] In a third aspect, an embodiment of the present application provides a computing device, including a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the image annotation method described in the first aspect above.

[0039] In a fourth aspect, an embodiment of the present application provides a computer storage medium storing a computer program, wherein when the computer program is executed by a computer, the image annotation method as described in the first aspect above is implemented.

[0040] In an embodiment of the present application, an initial mask image is generated based on an input query object and an image to be annotated in an acquired data set to be annotated, and a region of interest is annotated in the initial mask image, where the region of interest refers to an initial segmentation region of the query object in the image to be annotated; initial segmentation regions that do not meet the conditions are filtered out through a preset area filtering condition to obtain remaining initial segmentation regions, and the preset area filtering condition is constructed based on the mask region area of ​​the query object and the mask region area in the initial segmentation region; a bounding rectangle corresponding to the remaining initial segmentation region is constructed, and a target mask image is generated based on the bounding rectangle and the image to be annotated, and the target segmentation region of the query object in the image to be annotated is annotated in the target mask image.

[0041] The image annotation method proposed in this application significantly improves the accuracy and efficiency of annotation through a series of intelligent processing steps, while effectively reducing the annotation cost. The specific beneficial effects are reflected in the following aspects:

[0042] Accurately locate the region of interest: By generating an initial mask image and marking the initial segmented area of ​​the query object in the image to be labeled, this method can quickly focus on the part related to the query object and eliminate irrelevant background interference, thereby improving the pertinence and accuracy of the labeling.

[0043] Intelligent screening of connected regions: By calculating the first connected region and its area in the region of interest, and removing irrelevant or misidentified small regions based on preset area filtering conditions, it ensures that only significant regions closely related to the query object are retained, avoiding noise interference and improving the accuracy of annotation.

[0044] Refined labeling of target area: Based on the remaining initial segmented area, a bounding rectangle is constructed and a target mask image is generated accordingly, which enables refined labeling of the target segmented area of ​​the query object in the image and ensures that the labeling result is highly consistent with the actual object morphology.

[0045] Automated processing reduces manpower requirements: The entire process is highly automated, from initial mask generation to precise positioning of the target mask, which greatly reduces dependence on manual labeling and significantly reduces labor costs.

[0046] Efficient filtering mechanism saves time: The introduction of preset area filtering conditions can quickly exclude a large number of irrelevant or minor connected areas in the early stage, greatly improving the annotation efficiency and shortening the annotation cycle, thereby achieving significant savings in time costs.

[0047] Improve large-scale data processing capabilities: This method is not only suitable for the annotation of a single image, but is also effective for large-scale data sets. Its automation and intelligence make it efficient and cost-effective when processing large amounts of image data. It is suitable for industrial application scenarios and further reduces the annotation cost per unit of data.

[0048] In summary, the image annotation method of the present application not only significantly enhances the accuracy of annotation and ensures high-quality annotation results through intelligent technical means, but also greatly reduces the overall cost of annotation by reducing dependence on manual labor, providing an efficient and economical solution for fields such as industrial product inspection.

[0049] These and other aspects of the present application will become more apparent from the description of the following embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 A flowchart of an image annotation method provided in an embodiment of the present application;

[0052] Figure 2 A schematic diagram of extracting a feature vector of an object provided in an embodiment of the present application;

[0053] Figure 3 A schematic diagram of generating an initial mask image provided by an embodiment of the present application;

[0054] Figure 4A schematic diagram of generating a target mask image provided by an embodiment of the present application;

[0055] Figure 5 A schematic diagram of the structure of an image annotation device provided in an embodiment of the present application;

[0056] Figure 6 A schematic diagram of the structure of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0058] In some of the processes described in the specification and claims of this application and the above-mentioned figures, multiple operations that appear in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this article or executed in parallel. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish between different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit the "first" and "second" to be different types.

[0059] The image annotation method of this application is very suitable for application in the field of industrial product inspection due to its significant advantages in improving annotation accuracy and reducing annotation costs. Specific application scenarios include but are not limited to the following aspects:

[0060] Defect detection and classification: On the production line, this method can automatically mark defective areas such as scratches, cracks, stains, etc. on the product surface. By comparing with standard templates, the defect type and severity can be quickly classified to achieve immediate quality control, reduce the outflow of defective products, and improve product quality.

[0061] Size and shape verification: For parts that require strict size control, this method can accurately locate and mark key size features (such as hole diameter, edge length, etc.), automatically compare design drawings, ensure that each component meets tolerance requirements, and improve assembly accuracy and production efficiency.

[0062] Component assembly verification: During the assembly process, image annotation technology is used to automatically identify and annotate the position and posture of assembly parts, and compared with the expected assembly model to promptly detect assembly errors or deviations, reduce assembly failure rates, and optimize the assembly process.

[0063] Appearance consistency inspection: For products with high requirements for appearance consistency such as color and texture, this method can effectively mark and analyze the color distribution and pattern integrity of the product surface to ensure the consistency and aesthetics of the product appearance quality and meet the needs of the high-end market.

[0064] Automated quality report generation: Combined with AI image analysis results, detailed inspection reports are automatically generated, including defect locations, dimensional deviations, appearance anomalies, etc., to facilitate rapid review and decision-making by quality management personnel, while also providing data support for subsequent improvements.

[0065] Remote monitoring and fault prediction: In industrial equipment monitoring, real-time image analysis and annotation technology can be used to detect equipment wear, foreign matter intrusion and other problems in advance, provide a basis for preventive maintenance, and reduce downtime and maintenance costs caused by faults.

[0066] In summary, the image annotation method of the present application has broad application prospects in the field of industrial product inspection due to its high efficiency and accuracy. It can effectively improve production efficiency, reduce costs, ensure product quality, and provide strong technical support for realizing intelligent manufacturing and industrial upgrading.

[0067] The inventors have found that the current image annotation solutions in industrial product inspection mainly include the following categories:

[0068] Manual labeling: Rely on professionals to use labeling tools (such as Label box, CVAT) to manually label images in detail, including but not limited to labeling boxes, segmentation labeling, key point labeling, etc., which is suitable for scenes that require high accuracy.

[0069] Semi-automatic annotation algorithms: such as SAM, GrabCut, GraphCut, etc. based on interactive technology. These methods guide the algorithm to complete the remaining annotation work through the initial manual setting of points, boxes or areas, effectively reducing the time and workload of manual annotation. However, manual intervention is still required for the annotation of new images or complex situations, resulting in high annotation costs.

[0070] Semi-automatic annotation based on AI models: Use pre-trained AI models to automatically annotate images, which are then manually checked and adjusted. This type of solution reduces reliance on manpower, but the precision of annotation is limited by the model's previous learning scope. When new task categories appear, the model needs to be retrained, which limits its versatility and efficiency, resulting in poor annotation accuracy.

[0071] In view of this, the present application provides a new image annotation method, which includes: generating an initial mask image according to an input query object and an image to be annotated in an acquired data set to be annotated, and an area of ​​interest is annotated in the initial mask image, and the area of ​​interest refers to the initial segmentation area of ​​the query object in the image to be annotated; filtering out the initial segmentation areas that do not meet the conditions through a preset area filtering condition to obtain the remaining initial segmentation areas, and the preset area filtering condition is constructed based on the mask area area of ​​the query object and the mask area area in the initial segmentation area; constructing a circumscribed rectangle corresponding to the remaining initial segmentation area, generating a target mask image based on the circumscribed rectangle and the image to be annotated, and annotating the target segmentation area of ​​the query object in the image to be annotated in the target mask image.

[0072] The above method can not only solve the problems existing in the existing labeling scheme, but also significantly enhance the accuracy of labeling and ensure high-quality labeling results. Moreover, by reducing the reliance on manual labor, the overall cost of labeling is greatly reduced, providing efficient and economical solutions for industrial product inspection, medical image analysis and other fields.

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0074] Figure 1 A flowchart of an image annotation method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0075] 101. Generate an initial mask image according to an input query object and an image to be annotated in the acquired dataset to be annotated, wherein the initial mask image is annotated with a region of interest, where the region of interest refers to an initial segmentation region of the query object in the image to be annotated;

[0076] In this step, the query object refers to the specific target that the user wants to identify and annotate in the image to be annotated. For example, in industrial product inspection, the query object may be a specific part, defect type, or feature that needs to be measured (such as threads, holes, etc.).

[0077] The dataset to be labeled contains a large number of industrial product images that have not been manually labeled in detail. The purpose is to find images containing query objects and label them.

[0078] The initial mask image is initially generated by the SAM model, and the initial mask image is annotated with the area where the query object may exist in the image to be annotated (i.e., the initial segmentation area), which is usually determined by comparing the object feature vector and the image feature vector. This area is called the region of interest (ROI), which preliminarily defines the part that needs to be analyzed in detail. Optionally, in the embodiment of the present application, step 101 may include the following process:

[0079] 1011. Determine an input query object, where the query object is used to retrieve matching images to be annotated from a dataset to be annotated, wherein the query object includes an image segment;

[0080] In this step, the image fragment refers to a small-size image that meets the input requirements of the SAM model after preprocessing (such as scaling and cropping). In other words, the query object can be a thumbnail of a specified image or an image region cropped from a specified image.

[0081] Specifically, the query object can be any image provided by the user that only contains the desired query target (such as a defective area of ​​a workpiece), or an image containing the desired query target selected from multiple images to be annotated in the dataset to be annotated. By selecting these specified images as prompts and preprocessing (such as scaling and cropping) to generate small-size images that meet the input requirements of the SAM model, they are used as query objects.

[0082] 1012. Taking the query object as input, extracting an object feature vector of the query object through a first feature extraction network;

[0083] In this step, the query object is sent to the first feature extraction network (QueryEncoder module), which uses its built-in algorithm and structure to convert complex image data into an object feature vector of B*256 dimensions that is easy to process and compare, where B represents the number of queries, which can be determined according to the number of query objects input, such as B can be 1, 2, 3, etc. This process is the basis for subsequent image retrieval or matching tasks, ensuring that the system can quickly and accurately find images that match the query object in large-scale data sets.

[0084] Specifically, the Query Encoder module is a feature extraction backbone network based on the ResNet18 network, or other classification networks, which is constructed by multiple subcomponents such as the convolutional blocks (i.e., CNN Blocks) in the Convolutional Neural Network, the backbone network, and Adapt_Average_Pooling. Figure 2As shown in the figure, the input query object passes through the Query Encoder module constructed by CNN Blocks, backbone network, and Adapt_Average_Pooling, and then outputs the object feature vector corresponding to the query object.

[0085] 1013. Acquire a dataset to be annotated, where the dataset to be annotated includes images to be annotated;

[0086] In this step, obtaining the dataset to be annotated means collecting and organizing a series of image data that have not been annotated in detail. These image data form the basis for subsequent automatic or manual annotation tasks. The image to be annotated is each specific visual data unit in the dataset. They may come from various sources, such as real-time monitoring screenshots of industrial production lines, medical imaging scans, geographic remote sensing images, or image resources on the Internet.

[0087] 1014. Using the image to be annotated in the to-be-annotated data set as input, to extract an image feature vector of the image to be annotated through a second feature extraction network;

[0088] In this step, the second feature extraction network (Image Encoder module) is the image encoding module in SAM, which adopts the transformer structure. It is called "second" which may mean that there is more than one feature extraction network in the system or model architecture, and this network focuses on extracting useful features from the image. The Image Encoder module usually includes a series of convolutional layers, pooling layers, and other possible neural network structures, such as fully connected layers, normalization layers, etc. Its main responsibility is to extract meaningful and abstract feature representations from the input image, which can capture the content information of the image, not just the pixel-level information.

[0089] This application uses the feature extraction network of the Image Encoder module to perform feature extraction operations on each image in the labeled dataset, thereby converting it into an image feature vector that is easy for the model to process and analyze, laying the foundation for subsequent labeling or classification tasks.

[0090] The difference between the second feature extraction network (Image Encoder module) and the first feature extraction network (Query Encoder module) is that both aim to extract key features of the input data, but the Image Encoder module emphasizes visual feature extraction, while the Query Encoder module focuses on understanding semantic content.

[0091] 1015. For each of the to-be-annotated images, generate a corresponding initial mask image according to the object feature vector and the image feature vector of the to-be-annotated image.

[0092] In this step, if Figure 3 As shown in the figure, the object feature vector of the query object and the image feature vector of the image to be labeled in the dataset to be labeled are input into the Decoder module in the SAM model for further processing. The function of the Decoder module is to output a binary mask image (i.e., the initial mask image) based on the object feature vector and the image feature vector, and annotate the initial mask image with the location information of the query object in the image to be labeled (i.e., the initial segmentation area). It should be noted that the initial mask image generated at this time is not accurate. Figure 3 As shown, the initial segmentation region still contains many noise points (ie, the initial segmentation region includes many additional points in addition to the region similar to the query object), so subsequent steps need to be performed to generate a more accurate target mask image.

[0093] For example:

[0094] Assume that the application scenario is tire defect detection in an automobile parts manufacturing plant. There may be multiple defects on the tire surface, such as scratches, bubbles, and debonding, and each defect is defined as a different query object.

[0095] First, it is determined that the input query object is “tire surface scratches.” The operator selects a typical scratch image segment as the query object, and the image segment clearly shows the scratch features on the tire surface (ie, step 1011 ).

[0096] Secondly, the scratched image fragment is input into a pre-trained first feature extraction network (e.g., a model based on a convolutional neural network), and the network outputs a high-dimensional feature vector of the scratched object, which can characterize the key visual features of the scratched image (i.e., step 1012).

[0097] Next, a data set containing thousands of images to be inspected taken from a tire production line is obtained, and these images may contain scratches, no defects or other types of defects (ie, step 1013).

[0098] Furthermore, for each image to be annotated in the dataset, a second feature extraction network (which may be the same as or different from the network used in the first step, depending on task requirements) is also used to extract the global feature vector of the image, which helps in subsequent matching and recognition (i.e., step 1014).

[0099] Finally, an initial mask image is generated for each image by calculating the similarity between the feature vector of the query object and the feature vector of each image to be annotated. For example, if an image highly matches the scratch feature vector, the initial mask generated on the image will highlight the suspected scratch area, which is the area of ​​interest, providing a basis for further accurate annotation or analysis. The mask image may use different colors or grayscale values ​​to distinguish the background and potential defect areas, which is convenient for subsequent visual analysis and manual review (i.e., step 1015).

[0100] Through the above process, this method can quickly locate and mark defect areas of interest in large-scale industrial product image data, greatly improving detection efficiency and accuracy.

[0101] It should be noted that the initial mask image initially generated by the SAM model may have two situations: one is that the annotated initial segmentation area may contain noise areas (or noise points), and the other is that it does not contain noise areas. Generally speaking, for most simple query objects and annotated objects, the generated initial segmentation area will not have noise areas. For complex annotated samples, small areas with poor segmentation effects (i.e., noise areas) may be generated in or around the initial segmentation area, resulting in unsatisfactory segmentation effects.

[0102] In order to solve the above problem, the present application further needs to perform subsequent step 102 processing on the initial segmented area of ​​such noise-generating area.

[0103] 102. Filter out the initial segmentation regions that do not meet the condition by using a preset area filtering condition to obtain the remaining initial segmentation regions, wherein the preset area filtering condition is constructed based on the mask region area of ​​the query object and the mask region area in the initial segmentation region;

[0104] In this step, the area of ​​the mask region refers to the number of pixels in the mask region. Specifically, the area of ​​the mask region in the initial segmentation region can be determined according to the number of pixels contained in the initial segmentation region, and the area of ​​the mask region of the query object is determined according to the number of pixels contained in the query object.

[0105] The area filtering condition is a screening criterion. By comparing the proportional relationship between the mask area area of ​​the query object and the mask area area in the initial segmentation area, the initial segmentation area that is greatly different from the query object is removed. For example, when some initial segmentation areas are surrounded by noise areas, the area of ​​the initial segmentation area will be much larger than the area of ​​the query object. If such initial segmentation areas with noise areas are subsequently processed, it will lead to poor labeling and inaccurate labeling. Therefore, it is necessary to filter out these initial segmentation areas. It can also be understood that the images to be labeled corresponding to these initial segmentation areas need to be filtered out, and only the remaining images to be labeled that match the query object are processed.

[0106] Optionally, the process of "filtering out the initial segmentation areas that do not meet the conditions by using the preset area filtering conditions to obtain the remaining initial segmentation areas" in step 102 may include:

[0107] 1021. Calculate the mask area of ​​the query object and the mask area in the initial segmented area, wherein the mask area of ​​the query object is determined according to the number of pixels included in the query object, and the mask area in the initial segmented area is determined according to the number of pixels included in the initial segmented area;

[0108] 1022. Filter by preset area condition: τ*S Q ≤S k ≤(1 / τ)S Q , filter less than or equal to τ*S Q The initial segmentation area of ​​the area, and greater than or equal to (1 / τ)S Q The initial segmentation region of the area is obtained to obtain the remaining initial segmentation region;

[0109] Among them, τ represents the area filtering threshold coefficient, τ<1, S Q It is represented as the mask area of ​​the query object, S k It is expressed as the area of ​​the mask region in the initial segmentation region. In the embodiment of the present application, taking the field of industrial product inspection as an example, it is assumed that we are using computer vision technology to perform quality control on parts continuously produced on an assembly line, especially to screen out parts whose sizes meet specific requirements. Here, the "query object" can be understood as an ideal template of the part we want to inspect, and its mask area represents the area within the effective and acceptable size range that the part should have. The "initial segmentation area" refers to the candidate area that may represent a single part that is segmented from the actual captured part image through image analysis technology.

[0110] Example:

[0111] Consider a factory producing precision metal washers, where the standard diameter of the washers is 50 mm and the allowable dimensional tolerance is very small, such as ±2%. In order to automatically screen out washers with qualified dimensions, it is necessary to screen out the washers images that meet the requirements through the above step 103.

[0112] Define the query object: Design a digital template of an ideal gasket (i.e., the query object). This template accurately reflects the ideal circular gasket with a diameter of 50 mm. Binarize the query object image to form a mask area, and use the number of valid pixels (representing the gasket part) in the mask area as the area of ​​the mask area of ​​the ideal gasket. For example, if the number of valid pixels in the mask area is 200, then S Q =200mm 2 , where S Q The mask area of ​​the ideal gasket is represented as the mask area of ​​the query object. Of course, a corresponding relationship rule between the number of pixels and the area can also be set, and the mask area is determined according to the corresponding relationship rule.

[0113] Image acquisition and preprocessing: The industrial camera captures multiple gasket images transmitted on the assembly line, and after preprocessing the multiple gasket images (such as correcting the perspective, etc.) and performing preliminary segmentation, a series of initial segmentation areas that may contain a single gasket are obtained.

[0114] Calculate the area: Apply the image processing algorithm to each initial segmented area to calculate the number of pixels in its mask area, thereby obtaining the mask area S of the initial segmented area of ​​a single gasket. k , (k=1,…K), K represents the number of initial segmentation regions, which can also be understood as the number of gasket images (the default number of initial segmentation regions generated by one gasket image is one).

[0115] According to the size tolerance (such as ±2%), set the area filter threshold coefficient τ (τ<1). For example, for a size tolerance of ±2%, τ can be set slightly lower than 0.98 (because it is the lower limit, we hope to exclude too small sizes, so a stricter limit is used). In actual calculations, the conversion accuracy from pixels to actual sizes may also need to be considered. If τ=0.99 is set, the filtering condition becomes (0.99*200)mm 2 ≤S k ≤(1 / 0.99)*200mm 2 Therefore, the area less than or equal to (0.99*200) mm can be filtered out from this series of initial segmentation areas that may contain a single gasket. 2 The initial segmentation area is greater than or equal to (1 / 0.99)*200mm 2 The initial segmentation area of ​​the area is obtained to obtain the remaining initial segmentation area.

[0116] Application of results: After area filtering, only those gasket images whose dimensions fall within the allowable error range are retained and the system marks them as qualified products, while other areas with unqualified dimensions are marked as defective products or items requiring further inspection.

[0117] Through this area-based filtering method, the automated inspection system can efficiently screen out products whose dimensions meet strict standards from a large number of gaskets produced, greatly improving inspection efficiency and accuracy.

[0118] 103. Construct a bounding rectangle corresponding to the remaining initial segmented area, generate a target mask image based on the bounding rectangle and the image to be annotated, and annotate the target segmented area of ​​the query object in the image to be annotated in the target mask image.

[0119] In this step, the bounding rectangle refers to the smallest rectangular frame that completely surrounds the initial segmented area, and its coordinate values ​​define the position and size of the rectangle in the corresponding image to be labeled.

[0120] A target mask image is a special type of image in which the target object is marked as white (or a specified highlight color) and the background is black. It is often used in tasks such as image segmentation and target recognition.

[0121] Rectangle feature vectors are key feature representations extracted from bounding rectangles, which help to identify and distinguish different objects.

[0122] Optionally, in the embodiment of the present application, the process of "generating a target mask image based on the bounding rectangle corresponding to the remaining initial segmented area and the image to be annotated" in step 103 may include:

[0123] 1031. Determine the coordinate values ​​of the circumscribed rectangle corresponding to the remaining initial segmented area;

[0124] In this step, for example, if the number of the remaining initial segmentation regions is M (where M≤K), then the coordinate values ​​of the circumscribed rectangles corresponding to the M initial segmentation regions are m=1,2...M, where the coordinate value They are respectively expressed as the coordinates of the upper left corner and lower right corner of the circumscribed rectangle. The position and size of the circumscribed rectangle can be determined by the coordinates of the upper left corner and lower right corner.

[0125] 1032. Input the coordinate values ​​of the circumscribed rectangle into a third feature extraction network, so as to extract a rectangular feature vector of the circumscribed rectangle through the third feature extraction network;

[0126] In this step, the coordinates of the bounding rectangle can be converted to Encoded into a feature vector Prompt Token (i.e., a rectangular feature vector) with a dimension of 1*256. The difference between the Prompt Encoder module and the Image Encoder module and the QueryEncoder module is that the input of the PromptEncoder module is a text string, an image fragment, or a structured prompt information, and the output is a feature vector related to the prompt, which is usually used to guide the subsequent image processing process. The input of the Image Encoder module is the original image, and the output is the feature representation of the image, which is usually a high-dimensional vector that can reflect the content information of the image. The input of the QueryEncoder module can be a text query, an image query, etc., and the output is a feature vector of the query, which is used to find matching images or other types of data in the feature space.

[0127] 1033. Generate a target mask image based on the rectangular feature vector and the image feature vector of the image to be labeled.

[0128] In this step, the rectangular feature vector and the image feature vector are simultaneously input into the Decoder module of the SAM model, so that the information is exchanged with the rectangular feature vector and the image feature vector through the Ouput Token inside the decoder in the Decoder module, and then the OutputToken is converted into a binary mask image (i.e., the target mask image) through the neural network model, and the refined annotation result (i.e., the target segmentation area) is output. Specifically, Figure 4 As shown, the QueryEncoder module extracts the rectangular feature vectors of the circumscribed rectangles corresponding to the remaining initial segmented regions, combines them with the image feature vectors of the multiple images to be annotated extracted by the ImageEncoder module in the previous step, and inputs them into the SAM model. After the above series of operations, the SAM model finally obtains the target mask image, and the target segmented region of the query object in the image to be annotated is annotated in the target mask image. It should be noted that Figure 4 It can be seen that the generated target segmentation area is smaller than Figure 3 The generated initial segmentation area is more accurate (eg, noise points are filtered out and therefore more accurate).

[0129] Among them, "Output Token" usually refers to the target language vocabulary item generated by the decoder at each step. The decoding process is a sequence generation process. At each moment, the Decoder module predicts the next token to be generated based on the previously generated tokens and the output of the encoder.

[0130] In an embodiment of the present application, in a semiconductor chip defect detection scenario, the goal is to automatically identify and mark microcracks on the chip surface.

[0131] Determine the coordinate values ​​of the bounding rectangle: After area filtering, for each remaining initial segmentation area (assuming that these areas represent potential microcracks), calculate the pixel coordinates of the upper left corner and the lower right corner of its minimum bounding rectangle to define the range of the bounding rectangle.

[0132] Extracting rectangular feature vectors: The coordinate values ​​of the circumscribed rectangle and the pixel information contained therein are input through a pre-trained third feature extraction network to output a high-dimensional vector through the third feature extraction network, which captures key features such as the edge contour and texture of the microcrack as a rectangular feature vector.

[0133] Generate target mask image: At the same time, another feature extraction method is used to extract the image feature vector from the entire chip image to be annotated, which contains the overall structure of the chip, background information, etc. Subsequently, the rectangular feature vector is fused with the image feature vector, and a fusion algorithm (such as weighted summation or multimodal feature fusion strategy) is used to generate a high-quality target mask image. In this mask image, the microcrack area is densely annotated with white pixels, while other areas are black, clearly marking all potential defect locations.

[0134] In this way, by combining local details (the circumscribed rectangular features of microcracks) with global context (the features of the entire chip image), this embodiment improves the accuracy and robustness of defect detection and provides a reliable basis for subsequent automated analysis and processing.

[0135] It should be noted that the above steps 101 to 103 can be to generate an initial mask image for an image to be annotated, and then perform a series of subsequent processing to obtain a target mask image, and then traverse other images to be annotated and perform the same operation until the annotation of all images to be annotated that match the query object is completed. Of course, it is also possible to simultaneously process multiple images to be annotated in multiple threads to generate multiple target mask images to complete the annotation of all images to be annotated that match the query object. The embodiment of the present application does not limit this, and it can be set specifically according to needs.

[0136] Optionally, in an embodiment of the present application, the method also includes: determining a task type of the image annotation task, the task type including at least a target detection task and an instance segmentation task; if the task type includes a target detection task, updating the minimum bounding rectangle corresponding to the remaining initial segmentation area; if the task type includes an instance segmentation task, updating the minimum bounding rectangle corresponding to the remaining initial segmentation area, and updating the target mask image based on the minimum bounding rectangle.

[0137] In this step, the object detection task aims to identify objects of a specific category in the image and provide a bounding box for each detected object to indicate its location, but does not involve precise pixel-level segmentation inside the object. The instance segmentation task requires not only identifying each instance in the image, but also accurately segmenting the complete outline of each instance, that is, generating a target mask image for each instance.

[0138] In the embodiment of the present application, it is assumed that the application scenario is the quality inspection of circuit board solder joints on an electronic product assembly line.

[0139] Determine the task type: First, determine whether the current inspection task is target detection or instance segmentation based on the needs of the production line. If the purpose is to quickly screen out the locations of solder joints with poor soldering, select the target detection task. If the welding quality of each solder joint is required to be evaluated in detail, including the precise measurement of the area of ​​missing or excess parts of the solder joint, select the instance segmentation task.

[0140] For target detection tasks: When the task is to detect the location of bad solder joints, the system only needs to calculate and update the minimum bounding rectangle of each suspicious solder joint. For example, the suspected defective solder joints are identified through the machine learning model, and the bounding box is drawn for each solder joint and its position coordinates are recorded, without the need to generate a fine segmentation mask of the solder joint. This information is sufficient to guide subsequent manual re-inspection or automated rejection processes.

[0141] For instance segmentation tasks: If the task requires detailed analysis of each solder joint, the system not only calculates the minimum bounding rectangle, but also further refines the process based on the rectangle. For example, a deep learning model (such as Mask R-CNN) is used to generate a target mask image of each solder joint while identifying the solder joint, where the solder joint area is white and the background is black. This target mask image not only identifies the solder joint location (i.e., the target segmentation area), but also provides detailed information on the shape and size of the solder joint, which facilitates subsequent algorithms or operators to evaluate the welding quality, such as calculating solder joint coverage and identifying defects such as bridging or voids.

[0142] Through such differentiated processing, the present embodiment can flexibly adapt to different levels of industrial inspection needs, achieving both efficient preliminary screening and ensuring in-depth analysis capabilities of specific solder joint quality issues.

[0143] Optionally, in the embodiment of the present application, after step 103, the method further includes:

[0144] 104. Determine the object category of the query object; and mark the category of the query object in the target segmentation area for manual identification and calibration.

[0145] In this step, determining the object category of the query object refers to using image processing and machine learning techniques to identify which category a specific object in the image belongs to. This usually involves matching within a predefined set of categories, such as distinguishing different types of product defects, parts or components. By marking the object category of the query object in the target segmented area, that is, assigning the target segmented area with a category consistent with the object category of the query object, the subsequent manual review or calibration process can quickly understand and confirm whether the judgment of the automated system is accurate.

[0146] In the embodiment of the present application, the quality control link applied to automobile parts manufacturing is considered:

[0147] Determine the object category: Assume that the target of detection is various potential defects on the engine block of a car, such as cracks, scratches, holes, etc. Through a well-trained deep learning model (such as a convolutional neural network, CNN), the system can analyze high-resolution component images and identify different types of defects. The model will compare and analyze the image features with the pre-learned features of various defects, and finally assign a most likely defect category label, such as "crack" or "scratch", to each abnormal area found.

[0148] Target segmentation area annotation: Once the defects are classified, the system will superimpose or attach text labels (such as "crack"), color codes (such as red for severe defects, yellow for minor defects) or other visual identifiers near or directly in the defect area previously highlighted by the binarization process, so that quality inspectors can intuitively see the classification of defects by the automated system. For example, for an identified suspected crack area, the system will not only binarize it to highlight it, but also annotate "crack" next to it, so that manual review can quickly locate and verify whether the classification is correct, and make adjustments or confirmations if necessary.

[0149] By taking such steps, not only can the efficiency of defect detection be improved, but also the accuracy of the detection results can be ensured, because manual calibration can serve as a final safety net to correct any possible misclassification, which is crucial to ensure product quality.

[0150] Optionally, in the embodiment of the present application, after step 103, the method further includes: adjusting the target mask image, wherein the adjustment at least includes: fine-tuning the target mask image or deleting an erroneous object category using a preset fine-tuning tool;

[0151] The preset fine-tuning tools include at least: a polygon tool, an area pen tool, an eraser tool, a circle tool, a polyline tool, a fill tool, a three-point drawing tool, an ellipse tool and a rectangle tool.

[0152] In this step, adjusting the target mask image means that after obtaining the target mask image, the user or system is allowed to optimize the recognition result through a series of graphic editing tools, including correcting incorrect target segmentation area division, removing irrelevant connected areas, or correcting object category labeling errors.

[0153] The preset fine-tuning tool combines a variety of graphic editing functions, allowing users to precisely modify automatically generated masks, ensuring that the final analysis results are more accurate and reliable.

[0154] In the embodiment of the present application, it is assumed that the application scenario is component defect detection on an electronic board:

[0155] Adjustment of the target detection task: After automatically identifying and boxing out the locations of all capacitors on the electronic board, the system generated their minimum bounding rectangles. However, some of the rectangles failed to accurately cover the capacitors, partly due to errors in edge detection. At this point, the operator manually adjusted several rectangles using the polygon tool to ensure that they more closely fit the actual contours of the capacitors, improving positioning accuracy.

[0156] Adjustment of instance segmentation task: For the detection of small cracks in integrated circuit (IC) chips, the system performed instance segmentation tasks, marking the minimum bounding rectangle and finely dividing the mask area of ​​the crack. During the review process, it was found that the mask area of ​​some cracks contained background noise. The operator selected the eraser tool and the region pen tool to carefully erase the wrong pixels and fill in the missing parts, so that the mask area completely matched the actual crack.

[0157] Deletion of incorrect target segmentation areas: During a PCB solder joint inspection, the algorithm mistakenly identified some solder residues as solder joints. Most of the misidentified areas have been eliminated through the set threshold screening, but a small amount of residues remain. Through manual review, these incorrect target segmentation areas are directly selected using the rectangle tool and deleted. At the same time, the associated solder joint categories are also removed to avoid interference in subsequent analysis.

[0158] Category display during manual adjustment: In view of the complexity of the inspection task, the system design allows the category label of the target to be displayed during the manual adjustment stage. For example, when inspecting a complex circuit board assembly, the operator turns on the category display function, and each automatically identified component is marked with its estimated category name (such as "capacitor", "resistor"). In this way, when the classification of a certain area needs to be adjusted, the operator can directly select a new area through the three-point drawing tool and manually correct the category label to the correct "inductor", which improves the efficiency and accuracy of manual verification.

[0159] Polygon tool application: When detecting the boundary of an irregularly shaped integrated chip, the initial automated processing may not fit its edge perfectly, causing part of the chip edge to be confused with the background. The operator uses the polygon tool to manually outline the precise outline of the chip by clicking point by point. This process is similar to digital drawing, and each click leaves a mark on the image until a closed polygon is formed. Finally, the system will update the target mask image based on this path to ensure that the boundary of the chip is accurate.

[0160] Optimization of the regional pen tool: When faced with continuous and large error detection areas, such as a piece of metal residue that is considered a short circuit but is actually part of a normal circuit, the operator will select the regional pen tool and directly drag on the image to draw the area range that is expected to be retained or excluded. The tool can intelligently fill or clear the pixels in the specified area to achieve rapid correction and maintain the correctness of the connectivity analysis of the circuit diagram.

[0161] Eraser tool for precise cleaning: The Eraser tool plays an important role in removing minor defects, such as removing misidentified small blocks of stray pixels around integrated circuit pins. It allows operators to erase unwanted mask parts with pixel-level accuracy without affecting the surrounding correctly identified areas, ensuring the accuracy of defect detection.

[0162] The importance of displaying and adjusting categories: When reviewing the classification of components on high-density printed circuit boards (PCBs), the categories automatically marked by the system occasionally deviate, such as mistaking a small inductor for a resistor. At this time, by turning on the category display function, the operator can intuitively see the estimated category label next to each inspection area. Use the rectangle tool or directly click on the wrong label to quickly locate it, then call the corresponding tool (such as the three-point drawing tool to re-outline the correct area) and change the category to "inductor". This interaction greatly improves the efficiency and quality of the review.

[0163] Through the above embodiments, it can be seen that in the field of industrial product inspection, the flexible use of preset fine-tuning tools can not only significantly improve the accuracy of the automatic inspection system, but also ensure that the inspection results can adapt to complex and changeable actual scenarios, reduce human errors, and improve overall inspection efficiency and reliability.

[0164] Figure 5 A schematic diagram of the structure of an image annotation device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device comprises:

[0165] A generating module 51 is used to generate an initial mask image according to an input query object and an image to be annotated in the acquired dataset to be annotated, wherein the initial mask image is annotated with a region of interest, and the region of interest refers to an initial segmentation region of the query object in the image to be annotated;

[0166] A filtering module 52, configured to filter out the initial segmented regions that do not meet the conditions by using a preset area filtering condition to obtain the remaining initial segmented regions, wherein the preset area filtering condition is constructed based on the masked region area of ​​the query object and the masked region area in the initial segmented regions;

[0167] A construction module 53, used to construct a circumscribed rectangle corresponding to the remaining initial segmented area;

[0168] The generating module 51 is further configured to generate a target mask image based on the circumscribed rectangle and the image to be annotated, and annotate the target segmentation region of the query object in the image to be annotated in the target mask image.

[0169] Optionally, in an embodiment of the present application, the generation module 51 is specifically used to determine an input query object, the query object is used to retrieve matching images to be annotated from the data set to be annotated, wherein the query object includes an image fragment; taking the query object as input to extract an object feature vector of the query object through a first feature extraction network; obtaining the data set to be annotated, the data set to be annotated including the image to be annotated; taking the image to be annotated in the data set to be annotated as input to extract an image feature vector of the image to be annotated through a second feature extraction network; for each image to be annotated, generating a corresponding initial mask image according to the object feature vector and the image feature vector of the image to be annotated.

[0170] Optionally, in the embodiment of the present application, the filtering module 52 is specifically used to calculate the mask area of ​​the query object and the mask area in the initial segmentation area, the mask area of ​​the query object is determined according to the number of pixels included in the query object, and the mask area in the initial segmentation area is determined according to the number of pixels included in the initial segmentation area; by presetting the area filtering condition: τ*S Q ≤S k ≤(1 / τ)S Q , filter less than or equal to τ*S Q The initial segmentation area of ​​the area, and greater than or equal to (1 / τ)S Q The initial segmentation area of ​​the area is obtained to obtain the remaining initial segmentation area; where τ represents the area filtering threshold coefficient, τ<1, S Q It is represented as the mask area of ​​the query object, S k It is represented as the area of ​​the mask region in the initial segmentation region.

[0171] Optionally, in an embodiment of the present application, the generation module 51 is specifically used to determine the coordinate values ​​of the circumscribed rectangle corresponding to the remaining initial segmented area, and input the coordinate values ​​of the circumscribed rectangle into a third feature extraction network to extract a rectangular feature vector of the circumscribed rectangle through the third feature extraction network; based on the rectangular feature vector and the image feature vector of the image to be annotated, a target mask image is generated.

[0172] Optionally, in the embodiment of the present application, the device further includes: a determination module 54 and a processing module 55;

[0173] The determination module 54 is used to determine the task type of the image annotation task, and the task type at least includes a target detection task and an instance segmentation task;

[0174] The processing module 55 is used to update the minimum bounding rectangle corresponding to the remaining initial segmentation area if the determination module 54 determines that the task type includes a target detection task; if the determination module 54 determines that the task type includes an instance segmentation task, update the minimum bounding rectangle corresponding to the remaining initial segmentation area, and update the target mask image based on the minimum bounding rectangle.

[0175] Optionally, in the embodiment of the present application, the determination module 54 is further used to determine the object category of the query object;

[0176] The generating module 51 is further used to mark the category of the query object in the target segmentation area for manual identification and calibration.

[0177] Optionally, in an embodiment of the present application, the processing module 55 is also used to adjust the target mask image, and the adjustment at least includes: using a preset fine-tuning tool to fine-tune the target mask image or delete erroneous object categories; wherein the preset fine-tuning tool at least includes: a polygon tool, an area pen tool, an eraser tool, a circle tool, a polyline tool, a fill tool, a three-point drawing tool, an ellipse tool and a rectangle tool.

[0178] Figure 5 The image annotation device can perform Figure 1 The implementation principle and technical effect of the image annotation method described in the embodiment are not described in detail. The specific way in which each module and unit performs operations in the image annotation device in the above embodiment has been described in detail in the embodiment of the method, and will not be described in detail here.

[0179] In one possible design, Figure 5 The image annotation apparatus of the embodiment shown can be implemented as a computing device, such as Figure 6As shown, the computing device may include a storage component 61 and a processing component 62;

[0180] The storage component 61 stores one or more computer instructions, wherein the one or more computer instructions are called and executed by the processing component 62 .

[0181] The processing component 62 is used to: generate an initial mask image according to an input query object and an image to be annotated in the acquired data set to be annotated, and the initial mask image is annotated with a region of interest, and the region of interest refers to the initial segmentation region of the query object in the image to be annotated; filter out the initial segmentation regions that do not meet the conditions through a preset area filtering condition to obtain the remaining initial segmentation regions, and the preset area filtering condition is constructed based on the mask region area of ​​the query object and the mask region area in the initial segmentation region; construct a bounding rectangle corresponding to the remaining initial segmentation region, generate a target mask image based on the bounding rectangle and the image to be annotated, and annotate the target segmentation region of the query object in the image to be annotated in the target mask image.

[0182] The processing component 62 may include one or more processors to execute computer instructions to complete all or part of the steps in the above method. Of course, the processing component may also be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0183] The storage component 61 is configured to store various types of data to support operations at the terminal. The storage component can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0184] The display component 63 can be an electroluminescent (EL) element, a liquid crystal display or a micro display having a similar structure, or a retinal direct display or a similar laser scanning display.

[0185] Of course, a computing device may also include other components, such as input / output interfaces, communication components, etc.

[0186] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc.

[0187] The communication component is configured to facilitate, among other things, wired or wireless communications between the computing device and other devices.

[0188] Among them, the computing device can be a physical device or an elastic computing host provided by a cloud computing platform, etc. In this case, the computing device can refer to a cloud server, and the above-mentioned processing components, storage components, etc. can be basic server resources rented or purchased from the cloud computing platform.

[0189] The present application also provides a computer storage medium storing a computer program, wherein the computer program can achieve the above-mentioned Figure 1 The image annotation method of the illustrated embodiment.

[0190] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0191] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0193] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image annotation method, characterized in that: include: Generate an initial mask image according to the input query object and the image to be annotated in the acquired dataset to be annotated, and the initial mask image is annotated with a region of interest, where the region of interest refers to an initial segmentation region of the query object in the image to be annotated; Filtering out the initial segmentation regions that do not meet the conditions by using a preset area filtering condition to obtain the remaining initial segmentation regions, wherein the preset area filtering condition is constructed based on the mask region area of ​​the query object and the mask region area in the initial segmentation region; A bounding rectangle corresponding to the remaining initial segmented area is constructed, a target mask image is generated based on the bounding rectangle and the image to be annotated, and a target segmented area of ​​the query object in the image to be annotated is annotated in the target mask image.

2. The method according to claim 1, characterized in that The step of generating an initial mask image according to the input query object and the image to be annotated in the acquired data set to be annotated includes: Determining an input query object, the query object is used to retrieve matching images to be annotated from the dataset to be annotated, wherein the query object includes an image segment; Taking the query object as input, extracting an object feature vector of the query object through a first feature extraction network; Acquire a dataset to be annotated, wherein the dataset to be annotated includes images to be annotated; Taking the image to be annotated in the to-be-annotated data set as input, and extracting an image feature vector of the image to be annotated through a second feature extraction network; For each of the images to be annotated, a corresponding initial mask image is generated according to the object feature vector and the image feature vector of the image to be annotated.

3. The method according to claim 1, characterized in that The method of filtering out the initial segmentation areas that do not meet the conditions by using the preset area filtering conditions to obtain the remaining initial segmentation areas includes: Calculating the mask area of ​​the query object and the mask area in the initial segmented area, wherein the mask area of ​​the query object is determined according to the number of pixels included in the query object, and the mask area in the initial segmented area is determined according to the number of pixels included in the initial segmented area; Filter by preset area condition: τ*S Q ≤S k ≤(1 / τ)S Q , filter less than or equal to τ*S Q The initial segmentation area of ​​the area, and greater than or equal to (1 / τ)S Q The initial segmentation region of the area is obtained to obtain the remaining initial segmentation region; Among them, τ represents the area filtering threshold coefficient, τ<1, S Q It is represented as the mask area of ​​the query object, S k It is represented as the area of ​​the mask region in the initial segmentation region.

4. The method according to claim 2, characterized in that: The generating of the target mask image based on the bounding rectangle corresponding to the remaining initial segmented area and the image to be labeled includes: Determine the coordinate values ​​of the circumscribed rectangle corresponding to the remaining initial segmented area, and input the coordinate values ​​of the circumscribed rectangle into a third feature extraction network, so as to extract a rectangular feature vector of the circumscribed rectangle through the third feature extraction network; A target mask image is generated based on the rectangular feature vector and the image feature vector of the image to be labeled.

5. The method according to claim 4, characterized in that Also includes: Determine a task type of the image annotation task, where the task type includes at least a target detection task and an instance segmentation task; If the task type includes a target detection task, updating the minimum bounding rectangle corresponding to the remaining initial segmented area; If the task type includes an instance segmentation task, the minimum bounding rectangle corresponding to the remaining initial segmented area is updated, and the target mask image is updated based on the minimum bounding rectangle.

6. The method according to claim 1, characterized in that After generating the target mask image and marking the target segmentation area of ​​the query object in the image to be marked in the target mask image, the method further includes: Determining an object category of the query object; The category of the query object is marked in the target segmentation area for manual identification and calibration.

7. The method according to claim 6, characterized in that Also includes: Adjusting the target mask image, wherein the adjustment at least includes: fine-tuning the target mask image or deleting an erroneous object category using a preset fine-tuning tool; The preset fine-tuning tools include at least: a polygon tool, an area pen tool, an eraser tool, a circle tool, a polyline tool, a fill tool, a three-point drawing tool, an ellipse tool and a rectangle tool.

8. An image annotation device, characterized in that: include: A generating module, configured to generate an initial mask image according to an input query object and an image to be annotated in the acquired dataset to be annotated, wherein the initial mask image is annotated with a region of interest, wherein the region of interest refers to an initial segmented region of the query object in the image to be annotated; A filtering module, used to filter out the initial segmented areas that do not meet the conditions through a preset area filtering condition to obtain the remaining initial segmented areas, wherein the preset area filtering condition is constructed based on the mask area area of ​​the query object and the mask area area in the initial segmented areas; A construction module, used to construct a circumscribed rectangle corresponding to the remaining initial segmented area; The generating module is further configured to generate a target mask image based on the circumscribed rectangle and the image to be annotated, and annotate the target segmentation region of the query object in the image to be annotated in the target mask image.

9. A computing device, characterized in that It comprises a processing component and a storage component; the storage component stores one or more computer instructions; the one or more computer instructions are used to be called and executed by the processing component to implement the image annotation method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that: A computer program is stored, and when the computer program is executed, the image annotation method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Object segmentation method and device

    CN113313720A

  • Image processing method, device and equipment, readable storage medium and program product

    CN116977685A

  • Joint target labeling method and device, equipment and storage medium

    CN117078927A

  • Target object labeling method and device, computer equipment and storage medium

    CN117636028A

Cited By

  • Remote sensing sample labeling method and device based on artificial intelligence assistance

    CN121438319A

  • Pseudo-label generation method, segmentation model training method, concentration detection method and related products

    CN121527757A