Image annotation automatic supplement method and system based on region annotation box matching degree search

Abstract

The application relates to the technical field of target detection, in particular to an image labeling automatic complementation method and system based on region labeling frame matching degree search, and core steps of the application include the following steps: labeling a template region frame and a to-be-matched region frame on an LCD product image, the width and height of the two frames are consistent or within an error range, and the centers are close to the center of a liquid crystal target module; labeling all the labeling frames of the liquid crystal targets in the template region frame, and labeling at least one labeling frame in the to-be-matched region frame; acquiring the coordinates of the region frames and the labeling frames, establishing a mapping relationship, calculating an initial offset, and generating a template labeling frame set after translation within a set range; determining an optimal offset through parallel calculation and parallel search strategies, screening out the labeling frames to be complemented in the to-be-matched region; and combining the complemented labeling frames with the manually labeled frames, eliminating auxiliary frames, and obtaining complete labeling data. The application can efficiently process high-resolution images, and significantly improves labeling efficiency and quality.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to an automatic image annotation missing method and system based on region bounding box matching degree search. Background Technology

[0002] In the field of industrial inspection, computer vision technology has developed rapidly in recent years. Deep learning-based object detection methods have been widely used in many scenarios due to their excellent performance. These methods can effectively identify and locate target objects by learning from large-scale labeled datasets. However, their detection accuracy and generalization ability are highly dependent on the support of high-quality, large-scale labeled datasets.

[0003] In the specific scenario of LCD liquid crystal inspection, data annotation faces numerous severe challenges. LCD images not only contain a vast number of liquid crystal targets—tens of thousands in a single image—but also exhibit significant diversity in liquid crystal morphology due to manufacturing processes. Simultaneously, the warping of the product often results in uneven background lighting in the captured images, further complicating the annotation process. Under these circumstances, traditional annotation methods reveal significant shortcomings:

[0004] Firstly, manual annotation is extremely inefficient: existing annotation tools require operators to annotate each liquid crystal target individually. The dense distribution and varied shapes of LCD liquid crystals greatly increase the time cost and labor intensity of manual annotation. For example, if a single image contains 10,000 liquid crystals, manual annotation would take 4-5 hours. This inefficient annotation method is far from meeting the timeliness requirements of industrial batch detection for building large-scale datasets.

[0005] Secondly, existing automated annotation methods suffer from severe limitations in adaptability: Currently widely used traditional template matching techniques based on image pixels, such as SSIM-based structural similarity matching, are extremely sensitive to local deformations and positional shifts of the target. In LCD detection scenarios, complex variations in lighting and the diverse morphologies of liquid crystals lead to numerous missed and false matches, severely impacting annotation accuracy. Furthermore, the resolution of a single LCD image is typically as high as 5000×20000 pixels. Traditional template matching techniques consume significant computational resources and are inefficient when processing such high-resolution images. For example, in some practical applications, due to computational resource constraints, this method cannot complete the annotation of a large number of LCD images within a reasonable timeframe.

[0006] In summary, existing annotation methods in LCD liquid crystal detection scenarios either fail to meet batch production requirements due to low efficiency, or suffer from poor adaptability and high resource consumption, making it difficult to guarantee annotation quality and efficiency. A new annotation method is urgently needed to overcome these bottlenecks. Summary of the Invention

[0007] Therefore, the technical problem to be solved by the present invention is to overcome the problems of extremely low efficiency of manual annotation and insufficient adaptability of automated methods in LCD liquid crystal detection.

[0008] To address the aforementioned technical problems, this invention provides an automatic image annotation missing completion method and system based on region bounding box matching degree search, which can efficiently and accurately perform large-scale LCD image annotation. Further, the method includes the following steps:

[0009] Step S1: For LCD products of the same type, mark the template area frame and the area frame to be matched on its image. The width and height of the template area frame and the area frame to be matched are the same or within the allowable error range, and the center point of the two is close to the center of the liquid crystal target module they cover.

[0010] Step S2: Mark all liquid crystal targets within the area defined by the template area frame, and mark at least one liquid crystal target within the area defined by each matching area frame.

[0011] Step S3: Obtain the coordinates of the template region box, the region box to be matched, and all annotation boxes; establish the mapping relationship between the region and the annotation boxes; calculate the initial offset between the region to be matched and the template region; and define the search offset within a set range to generate a translated set of template annotation boxes.

[0012] Step S4: Based on the translated template annotation box set, the overlap between its annotation box and the annotation box of the area to be matched and the area overlap are calculated in parallel. The search space is traversed by a batch search strategy to determine the optimal offset that maximizes the overlap. According to the optimal offset, the annotation boxes to be filled in the area to be matched are selected from the translated template annotation box set.

[0013] Step S5: Merge the obtained annotation boxes to be filled with the manually annotated annotation boxes, and remove the two types of auxiliary boxes, template area boxes and matching area boxes, to obtain the annotation data of all liquid crystal targets in the LCD product image.

[0014] In one embodiment of the present invention, step S3, the method for establishing the mapping relationship between the region and the annotation box includes:

[0015] The coordinates of the unique template region are represented as follows: ,in , , , These represent the coordinates of the left, top, right, and bottom boundaries of template region A, and N regions to be matched. The coordinates of each region to be matched are represented as follows: , ,in , , , These are the regions to be matched. The coordinates of the left, top, right, and bottom boundaries; M LCD target annotation boxes. The coordinates of each annotation box are , ,in , , , These are the coordinates of the left, top, right, and bottom boundaries of the annotation box, respectively.

[0016] Traverse all LCD target annotation boxes Based on its coordinates, determine whether it belongs to the template region or the region to be matched, and classify it into the corresponding set:

[0017] like satisfy and and and Then it is classified into the template area annotation box set. ;

[0018] like If it does not belong to the template area, then check whether it belongs to the area to be matched. :when satisfy and and and Then, the set of bounding boxes categorized into the corresponding regions to be matched. .

[0019] In one embodiment of the present invention, the method for calculating the initial offset between the region to be matched and the template region in step S3 is as follows:

[0020] For each region to be matched Calculate its initial position offset from template region A. , This represents the initial lateral offset based on the center of template region A. This represents the initial vertical offset based on the center of template region A, and its calculation method is as follows:

[0021] ,

[0022] .

[0023] In one embodiment of the present invention, the method for defining a search offset within a set range to generate a translated set of template annotation boxes is as follows:

[0024] Define a search space, and define a search offset within the search space. , This indicates the horizontal search offset. Indicates the vertical search offset;

[0025] Set of annotation boxes for template area Offset calculations are performed on the coordinates of all annotation boxes: the initial offset is added to the coordinates of the left, top, right, and bottom boundaries of each annotation box. and search offset The translated template annotation box set is obtained. ,in The coordinates of each translated annotation box are ;

[0026] in, Indicates the region to be matched For the initial lateral offset of the center of template region A Indicates the region to be matched For the initial longitudinal offset of the center of template region A; These are the coordinates of the left, top, right, and bottom boundaries of the translated annotation box.

[0027] In one embodiment of the present invention, the coordinates of the left, top, right, and bottom boundaries of the translated annotation box are... The calculation method is as follows:

[0028] , ,

[0029] , ,

[0030] in, , , , These are the template area annotation boxes. Middle annotation box The coordinates of the left, top, right, and bottom boundaries.

[0031] In one embodiment of the present invention, step S4, the method for selecting the annotation boxes to be filled in the matching area from the translated template annotation box set, includes:

[0032] The translated template annotation box set Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries; the set of bounding boxes for the region to be matched. Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries;

[0033] According to the above and stated The coordinates are used to calculate the overlap between the translated template annotation box and the annotation box of the area to be matched. :

[0034] ,

[0035] in, express and The intersection and union ratio, express and The distance between the center points, express and The length of the diagonal of the minimum bounding rectangle. It is a smoothing factor;

[0036] According to the degree of overlap ,calculate P annotation boxes and The sum of the overlap of the Q bounding boxes represents the overlap between the translated template area and the area to be matched. :

[0037] ,

[0038] The search offset combinations within the search space are divided into multiple data batches, each batch containing a subset of offset combinations. The GPU is used to compute in parallel the values ​​corresponding to all offset combinations within each batch. The value is used to obtain the overlap rate in each batch. The maximum combination of search offsets and the corresponding set of translated template annotation boxes ;

[0039] The optimal result obtained by comparing each batch Value, to determine the degree of overlap The largest combination of search offsets is taken as the optimal offset, and the corresponding optimal offset is... In the process, select those that are related to The annotation boxes that do not overlap are the annotation boxes to be filled in the area to be matched.

[0040] In one embodiment of the present invention, the and Distance between the center points The calculation method is as follows:

[0041] .

[0042] In one embodiment of the present invention, the and The minimum length of the diagonal of the bounding rectangle The calculation method is as follows:

[0043] ,

[0044] in, To add to the annotation box and annotation box The maximum value of the right boundary coordinates. To add to the annotation box and annotation box Minimum value of the coordinates of the middle left boundary; To add to the annotation box and annotation box The maximum value of the lower boundary coordinates; To add to the annotation box and annotation box The minimum value of the coordinates of the upper and middle boundaries.

[0045] Based on the same inventive concept, this invention also provides an automatic image annotation completion system based on region bounding box matching degree search. The system includes: a region bounding box calibration module, a preliminary bounding box generation module, a coordinate processing and offset calculation module, an optimal offset search and completion box filtering module, and an annotation data integration and output module.

[0046] The region frame calibration module is configured to: for the same type of LCD product, mark a template region frame and a region frame to be matched on its image, wherein the width and height of the template region frame and the region frame to be matched are the same or within the allowable error range, and the center point of the two is close to the center of the liquid crystal target module they cover.

[0047] The annotation box initial generation module is configured to: annotate all liquid crystal targets within the area defined by the template area box, and annotate at least one liquid crystal target within the area defined by each area box to be matched.

[0048] The coordinate processing and offset calculation module is configured to: obtain the coordinates of the template region box, the region box to be matched, and all annotation boxes; establish a mapping relationship between the region and the annotation boxes; calculate the initial offset between the region to be matched and the template region; and define a search offset within a set range to generate a translated set of template annotation boxes.

[0049] The optimal offset search and completion box filtering module is configured to: based on the translated template annotation box set, calculate the overlap between its annotation box and the annotation box of the region to be matched and the region overlap in parallel, traverse the search space using a batch search strategy, determine the optimal offset that maximizes the overlap, and filter out the annotation boxes to be completed in the region to be matched from the translated template annotation box set according to the optimal offset.

[0050] The annotation data integration and output module is configured to: merge the acquired annotation boxes to be filled with the manually annotated annotation boxes, remove the two types of auxiliary boxes, template area boxes and matching area boxes, and obtain the annotation data of all liquid crystal targets in the LCD product image.

[0051] The present invention also provides a computer storage medium storing a computer software product, the computer software product including several instructions for causing a computer device to execute the image annotation automatic missing part method based on region bounding box matching degree search.

[0052] The technical solution of the present invention has the following advantages compared with the prior art:

[0053] This invention significantly reduces the manual annotation workload of bounding boxes in LCD images by defining template regions and matching regions and combining a small amount of manual annotation. It generates a translated set of template bounding boxes by combining initial offsets and search offsets, and calculates the overlap ratio using multi-dimensional indicators such as intersection-union ratio and center point distance, improving the accuracy of automatic annotation completion. Employing GPU parallel computing and batch search strategies, it efficiently handles the large-scale computational needs of high-resolution images, significantly improving efficiency while ensuring annotation quality. It is particularly suitable for scenarios with high region alignment, such as LCD detection, providing an efficient and feasible solution for deep learning models to obtain large-scale labeled datasets. Attached Figure Description

[0054] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein...

[0055] Figure 1 This is a schematic flowchart of an image annotation automatic missing part method based on region annotation box matching degree search provided in an embodiment of the present invention;

[0056] Figure 2 These are schematic diagrams of liquid crystal target annotations in the template area and the area to be matched, where (a) represents a schematic diagram of liquid crystal target annotations in the template area and (b) represents a schematic diagram of liquid crystal target annotations in the area to be matched.

[0057] Figure 3 These are schematic diagrams of liquid crystal target annotations in the template area and the area to be matched, where (a) represents a schematic diagram of liquid crystal target annotations in the template area, and (b) represents a schematic diagram of liquid crystal target annotation completion in the area to be matched.

[0058] Figure 4 This is a schematic diagram of an image annotation automatic missing part system based on region bounding box matching degree search provided in an embodiment of the present invention;

[0059] Explanation of reference numerals in the accompanying drawings: 100, Region bounding box calibration module; 200, Initial generation of annotation boxes module; 300, Coordinate processing and offset calculation module; 400, Optimal offset search and box completion filtering module; 500, Annotation data integration and output module. Detailed Implementation

[0060] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0061] Example 1:

[0062] See Figure 1This invention provides an automatic image annotation missing completion method and system based on region bounding box matching degree search, which can efficiently and accurately perform large-scale LCD image annotation. Further, the method includes the following steps:

[0063] Step S1: For LCD products of the same type, mark the template area frame and the area frame to be matched on its image. The width and height of the template area frame and the area frame to be matched are the same or within the allowable error range, and the center point of the two is close to the center of the liquid crystal target module they cover.

[0064] Step S2: Mark all liquid crystal targets within the area defined by the template area frame, and mark at least one liquid crystal target within the area defined by each matching area frame.

[0065] Step S3: Obtain the coordinates of the template region box, the region box to be matched, and all annotation boxes; establish the mapping relationship between the region and the annotation boxes; calculate the initial offset between the region to be matched and the template region; and define the search offset within a set range to generate a translated set of template annotation boxes.

[0066] Step S4: Based on the translated template annotation box set, the overlap between its annotation box and the annotation box of the area to be matched and the area overlap are calculated in parallel. The search space is traversed by a batch search strategy to determine the optimal offset that maximizes the overlap. According to the optimal offset, the annotation boxes to be filled in the area to be matched are selected from the translated template annotation box set.

[0067] Step S5: Merge the obtained annotation boxes to be filled with the manually annotated annotation boxes, and remove the two types of auxiliary boxes, template area boxes and matching area boxes, to obtain the annotation data of all liquid crystal targets in the LCD product image.

[0068] As can be seen from the above technical solution, this invention, by matching the template region with the region to be matched, only requires labeling all liquid crystal targets in the template region and a small number of liquid crystal targets in the region to be matched. This allows for large-scale labeling through automatic gap filling, significantly reducing manual labeling workload, time costs, and labor intensity. It is suitable for scenarios with a large number of liquid crystal targets in LCD images. By combining the initial offset and the search offset to generate a translated template label box set, and using multi-dimensional overlap indicators (cross-over ratio, center point distance, etc.) and batch GPU parallel computation to determine the optimal offset, this invention ensures labeling accuracy while efficiently handling the complex calculations of high-resolution LCD images, balancing labeling quality and efficiency to meet the needs of industrial batch inspection.

[0069] like Figure 2As shown, the manual annotation tool Labelme is used to complete the calibration of the template region box and the region box to be matched in the images of LCD products of the same type: the template region box is used to define the liquid crystal target module as a reference, and all liquid crystal targets within its coverage area need to be fully annotated (e.g., ...). Figure 2 As shown in (a)); the region bounding box to be matched corresponds to a liquid crystal target module that has structural similarity to the template module and needs to be annotated and completed through a matching algorithm. Therefore, only sparse annotation of at least one liquid crystal target is required within its coverage area (e.g., Figure 2 (as shown in (b)).

[0070] Furthermore, in this embodiment, the method for establishing the mapping relationship between the region and the annotation box in step S3 includes:

[0071] The coordinates of the unique template region are represented as follows: ,in , , , These represent the coordinates of the left, top, right, and bottom boundaries of template region A, and N regions to be matched. The coordinates of each region to be matched are represented as follows: , ,in , , , These are the regions to be matched. The coordinates of the left, top, right, and bottom boundaries; M LCD target annotation boxes. The coordinates of each annotation box are , ,in , , , These are the coordinates of the left, top, right, and bottom boundaries of the annotation box, respectively.

[0072] Traverse all LCD target annotation boxes Based on its coordinates, determine whether it belongs to the template region or the region to be matched, and classify it into the corresponding set:

[0073] like satisfy and and and Then it is classified into the template area annotation box set. ;

[0074] like If it does not belong to the template area, then check whether it belongs to the area to be matched. :when satisfy and and and Then, the set of bounding boxes categorized into the corresponding regions to be matched. .

[0075] Furthermore, after classifying all the bounding boxes into their corresponding regions, the method for calculating the initial offset between the region to be matched and the template region is as follows:

[0076] For each region to be matched Calculate its initial position offset from template region A. , This represents the initial lateral offset based on the center of template region A. This represents the initial vertical offset based on the center of template region A, and its calculation method is as follows:

[0077] , .

[0078] Furthermore, a search offset is defined within a set range. Based on the search offset and the initial offset, a set of translated template annotation boxes is generated. The method is as follows:

[0079] Taking into account the inherent characteristics of LCD products, such as staggered rows and regional warping, a search space is set to cover potential positional deviations: lateral search offset. The value is an integer within the range [-10, 10] (step size 1 pixel), representing the vertical search offset. The value is an integer within the range [-30, 30] (step size 1 pixel), and a two-dimensional search offset is defined in this search space. ;

[0080] Set of annotation boxes for template area Offset calculations are performed on the coordinates of all annotation boxes: the initial offset is added to the coordinates of the left, top, right, and bottom boundaries of each annotation box. and search offset The translated template annotation box set is obtained. ,in The coordinates of each translated annotation box are ;

[0081] in, Indicates the region to be matched For the initial lateral offset of the center of template region A Indicates the region to be matched For the initial longitudinal offset of the center of template region A; These are the coordinates of the left, top, right, and bottom boundaries of the translated annotation box, respectively, and their calculation method is as follows:

[0082] , , , ,

[0083] in, , , , These are the template area annotation boxes. Middle annotation box The coordinates of the left, top, right, and bottom boundaries.

[0084] Further, in this embodiment, step S4, the method for selecting the annotation boxes to be filled in the matching area from the translated template annotation box set, includes:

[0085] The translated template annotation box set Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries; the set of bounding boxes for the region to be matched. Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries;

[0086] According to the above and stated The coordinates are used to calculate the overlap between the translated template annotation box and the annotation box of the area to be matched. :

[0087] ,

[0088] in, express and The intersection and union ratio, , Indicates the annotation box and annotation box Area of ​​the overlapping region , Indicates the annotation box and annotation box Total area covered ; express and The distance between the center points, ; express and The length of the diagonal of the minimum bounding rectangle. This is a smoothing factor, typically taken as 1e-16, used to avoid a denominator of 0. Specifically, the... and The minimum length of the diagonal of the bounding rectangle The calculation formula is:

[0089] ;in, To add to the annotation box and annotation box The maximum value of the right boundary coordinates. To add to the annotation box and annotation box Minimum value of the coordinates of the middle left boundary; To add to the annotation box and annotation box The maximum value of the lower boundary coordinates; To add to the annotation box and annotation box The minimum value of the coordinates of the upper and middle boundaries;

[0090] According to the degree of overlap ,calculate P annotation boxes and The sum of the overlap of the Q bounding boxes represents the overlap between the translated template area and the area to be matched. :

[0091] ;

[0092] Because each The operations are identical and in no particular order. Considering the limitations of GPU computing resources, all combinations of search offsets within the search space are divided into multiple data batches, with each batch containing a portion of the offsets. Combining and utilizing GPUs to compute all offsets within each batch in parallel. Combination corresponding The value is used to obtain the overlap rate in each batch. Maximum search offset combination and the corresponding set of translated template annotation boxes ;

[0093] The optimal result obtained by comparing each batch Value, to determine the degree of overlap Maximum search offset The combination is used as the optimal offset, and the optimal offset corresponds to In the process, select those that are related to All annotation boxes in the text do not overlap (i.e., the condition is met). The annotation boxes are the annotation boxes that need to be filled in the area to be matched.

[0094] Through the above steps, the entire process from offset search to box completion determination is realized. It not only utilizes GPU parallel computing to improve efficiency, but also solves memory limitations by batch processing, and finally accurately outputs the box completion annotation of the area to be matched.

[0095] To verify the effectiveness of the method provided by this invention, a single LCD image with a resolution of 6137×20802 was selected, containing 8784 LCD targets. All targets needed to be labeled. The hardware configuration of the experimental environment was an NVIDIA GeForce RTX 2080Ti graphics card (11GB video memory) and an Intel(R) Core(TM) i7-8700 CPU@3.20GHz processor. The software environment was Python 3.8, PyTorch 1.13, and CUDA 11.7.

[0096] In the experiment, the image was divided into 1 template region and 35 regions to be matched, for a total of 36 regions. The Labelme manual annotation tool was used to manually annotate all 224 liquid crystal targets within the template region, such as... Figure 3 As shown in (a), two liquid crystal targets are manually marked in each region to be matched as a matching reference.

[0097] Set the horizontal search offset range to The vertical search offset range is With a step size of 1 pixel, the total number of search combinations is 21 × 61 = 1281. GPU batch processing is used, with 200 offset combinations processed in each batch. Tensor transformation and broadcasting mechanisms are used to implement template region bounding boxes (…). ) and the label box of the region to be matched ( Matrix-based parallel computation.

[0098] Taking the region to be matched, B1, as an example, its initial offset from the template region is: After searching, the optimal search offset was determined to be... Corresponding regional overlap After offset calculation, 222 of the 224 annotation boxes in the template area were selected as fill boxes that did not overlap with the manually labeled boxes in area B1. The fill effect is as follows. Figure 3 As shown in (b).

[0099] Furthermore, the number of manually labeled bounding boxes per image was reduced from 8784 for all images to 8784 / 36+36+35×2=338, significantly reducing the annotation workload. The mean accuracy (mAP@0.50) between automatically completed bounding boxes and all manually labeled bounding boxes reached nearly 100%, validating the annotation accuracy of this method.

[0100] Experimental results show that the present invention, through region division and template matching strategies, can significantly reduce the workload of manual annotation while achieving accurate and automatic missing filling of liquid crystal targets in LCD images, thus meeting the requirements of industrial batch detection for annotation efficiency and accuracy.

[0101] Example 2:

[0102] Based on the same inventive concept as Embodiment 1, the present invention also provides an automatic image annotation missing system based on region bounding box matching degree search, for implementing the steps of the method described in Embodiment 1.

[0103] like Figure 4 As shown, the image annotation automatic missing-filling system based on region bounding box matching degree search includes the following modules: region bounding box calibration module 100, initial bounding box generation module 200, coordinate processing and offset calculation module 300, optimal offset search and missing-filling box filtering module 400, and annotation data integration and output module 500; wherein,

[0104] The region frame calibration module 100 is configured to: mark a template region frame and a region frame to be matched on the image of the same type of LCD product, wherein the width and height of the template region frame and the region frame to be matched are the same or within the allowable error range, and the center point of the two is close to the center of the liquid crystal target module they cover.

[0105] The annotation box initial generation module 200 is configured to: annotate all liquid crystal targets within the area defined by the template area box, and annotate at least one liquid crystal target within the area defined by each area box to be matched.

[0106] The coordinate processing and offset calculation module 300 is configured to: obtain the coordinates of the template region box, the region box to be matched and all annotation boxes, establish the mapping relationship between the region and the annotation boxes, calculate the initial offset between the region to be matched and the template region, and define the search offset within a set range to generate a translated set of template annotation boxes.

[0107] The optimal offset search and completion box filtering module 400 is configured to: based on the translated template annotation box set, calculate the overlap between its annotation box and the annotation box of the region to be matched and the region overlap in parallel, traverse the search space using a batch search strategy, determine the optimal offset that maximizes the overlap, and filter out the annotation boxes to be completed in the region to be matched from the translated template annotation box set according to the optimal offset.

[0108] The annotation data integration and output module 500 is configured to: merge the acquired annotation boxes to be filled with the manually annotated annotation boxes, remove the two types of auxiliary boxes, template area boxes and matching area boxes, and obtain the annotation data of all liquid crystal targets in the LCD product image.

[0109] This embodiment proposes an automatic image annotation completion system based on region bounding box matching degree search, which is used to implement the aforementioned automatic image annotation completion method based on region bounding box matching degree search. Therefore, the specific implementation of the system can be found in the embodiment section of the aforementioned method. For example, the region bounding box calibration module 100, the initial bounding box generation module 200, the coordinate processing and offset calculation module 300, the optimal offset search and completion box filtering module 400, and the annotation data integration and output module 500 are respectively used to implement steps S1, S2, S3, S4, and S5 in the method described in Embodiment 1. Therefore, its specific implementation can be referred to the description of the corresponding embodiments. To avoid redundancy, it will not be repeated here.

[0110] Example 3:

[0111] The present invention also provides a computer storage medium storing a computer software product, the computer software product including several instructions for causing a computer device to execute the image annotation automatic missing method based on region bounding box matching degree search as described in Embodiment 1.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An image annotation automatic filling method based on region annotation box matching degree search, characterized in that, Includes the following steps: Step S1: For LCD products of the same type, mark the template area frame and the area frame to be matched on its image. The width and height of the template area frame and the area frame to be matched are the same or within the allowable error range, and the center point of the two is close to the center of the liquid crystal target module they cover. Step S2: Mark all liquid crystal targets within the area defined by the template area frame, and mark at least one liquid crystal target within the area defined by each matching area frame. Step S3: Obtain the coordinates of the template region box, the region box to be matched, and all annotation boxes; establish the mapping relationship between the region and the annotation boxes; calculate the initial offset between the region to be matched and the template region; and define the search offset within a set range to generate a translated set of template annotation boxes. Step S4: Based on the translated template annotation box set, the overlap between its annotation box and the annotation box of the area to be matched and the area overlap are calculated in parallel. The search space is traversed by a batch search strategy to determine the optimal offset that maximizes the overlap. According to the optimal offset, the annotation boxes to be filled in the area to be matched are selected from the translated template annotation box set. In step S4, the method for selecting the annotation boxes to be filled in the matching area from the translated template annotation box set includes: The translated template annotation box set Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries; the set of bounding boxes for the region to be matched. Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries; According to the above and stated The coordinates are used to calculate the overlap between the translated template annotation box and the annotation box of the area to be matched. : ， in, express and The intersection and union ratio, express and The distance between the center points, express and The length of the diagonal of the minimum bounding rectangle. It is a smoothing factor; According to the degree of overlap ,calculate P annotation boxes and The sum of the overlap of the Q bounding boxes represents the overlap between the translated template area and the area to be matched. : ， The search offset combinations within the search space are divided into multiple data batches, each batch containing a subset of offset combinations. The GPU is used to compute in parallel the values ​​corresponding to all offset combinations within each batch. The value is used to obtain the overlap rate in each batch. The maximum combination of search offsets and the corresponding set of translated template annotation boxes ; The optimal result obtained by comparing each batch Value, to determine the degree of overlap The largest combination of search offsets is taken as the optimal offset, and the corresponding optimal offset is... In the process, select those that are related to The annotation boxes that do not overlap are the annotation boxes to be filled in the area to be matched; Step S5: Merge the obtained annotation boxes to be filled with the manually annotated annotation boxes, and remove the two types of auxiliary boxes, template area boxes and matching area boxes, to obtain the annotation data of all liquid crystal targets in the LCD product image.

2. The automatic image annotation completion method based on region bounding box matching degree search according to claim 1, characterized in that, In step S3, the methods for establishing the mapping relationship between the region and the annotation box include: The coordinates of the unique template region are represented as follows: ,in , , , These represent the coordinates of the left, top, right, and bottom boundaries of template region A, and N regions to be matched. The coordinates of each region to be matched are represented as follows: , ,in , , , These are the regions to be matched. The coordinates of the left, top, right, and bottom boundaries; M LCD target annotation boxes. The coordinates of each annotation box are , ,in , , , These are the coordinates of the left, top, right, and bottom boundaries of the annotation box, respectively. Traverse all LCD target annotation boxes Based on its coordinates, determine whether it belongs to the template region or the region to be matched, and classify it into the corresponding set: like satisfy and and and Then it is classified into the template area annotation box set. ; like If it does not belong to the template area, then check whether it belongs to the area to be matched. :when satisfy and and and Then, the set of bounding boxes categorized into the corresponding regions to be matched. .

3. The automatic image annotation completion method based on region bounding box matching degree search according to claim 1, characterized in that, In step S3, the method for calculating the initial offset between the region to be matched and the template region is as follows: For each region to be matched Calculate its initial position offset from template region A. , This represents the initial lateral offset based on the center of template region A. This represents the initial vertical offset based on the center of template region A, and its calculation method is as follows: ， 。 4. The automatic image annotation completion method based on region bounding box matching degree search according to claim 1, characterized in that, The method for defining the search offset within a set range to generate a translated set of template annotation boxes is as follows: Define a search space, and define a search offset within the search space. , This indicates the horizontal search offset. Indicates the vertical search offset; Set of annotation boxes for template area Offset calculations are performed on the coordinates of all annotation boxes: the initial offset is added to the coordinates of the left, top, right, and bottom boundaries of each annotation box. and search offset The translated template annotation box set is obtained. ,in The coordinates of each translated annotation box are ; in, Indicates the region to be matched For the initial lateral offset of the center of template region A Indicates the region to be matched For the initial longitudinal offset of the center of template region A; These are the coordinates of the left, top, right, and bottom boundaries of the translated annotation box.

5. The automatic image annotation completion method based on region bounding box matching degree search according to claim 4, characterized in that, The coordinates of the left, top, right, and bottom boundaries of the translated annotation box The calculation method is as follows: ， ， ， ， in, , , , These are the template area annotation boxes. Middle annotation box The coordinates of the left, top, right, and bottom boundaries.

6. The automatic image annotation completion method based on region bounding box matching degree search according to claim 1, characterized in that, The and Distance between the center points The calculation method is as follows: 。 7. The automatic image annotation completion method based on region bounding box matching degree search according to claim 1, characterized in that, The and The minimum length of the diagonal of the bounding rectangle The calculation method is as follows: ， in, To add to the annotation box and annotation box The maximum value of the right boundary coordinates. To add to the annotation box and annotation box Minimum value of the coordinates of the middle left boundary; To add to the annotation box and annotation box The maximum value of the lower boundary coordinates; To add to the annotation box and annotation box The minimum value of the coordinates of the upper and middle boundaries.

8. An automatic image annotation and missing completion system based on region bounding box matching degree search, characterized in that, The automatic image annotation and missing completion system based on region bounding box matching degree search includes the following modules: The region frame calibration module is configured to: mark a template region frame and a region frame to be matched on the image of the same type of LCD product, wherein the width and height of the template region frame and the region frame to be matched are the same or within the allowable error range, and the center point of the two is close to the center of the liquid crystal target module they cover. The annotation box initial generation module is configured to: annotate all liquid crystal targets within the area defined by the template area box, and annotate at least one liquid crystal target within the area defined by each area box to be matched; The coordinate processing and offset calculation module is configured to: obtain the coordinates of the template region box, the region box to be matched and all annotation boxes, establish the mapping relationship between the region and the annotation boxes, calculate the initial offset between the region to be matched and the template region, and define the search offset within a set range to generate a set of translated template annotation boxes. The optimal offset search and completion box filtering module is configured to: based on the translated template annotation box set, calculate the overlap between its annotation box and the annotation box of the region to be matched and the region overlap in parallel, traverse the search space using a batch search strategy, determine the optimal offset that maximizes the overlap, and filter the annotation boxes to be filled in the region to be matched from the translated template annotation box set according to the optimal offset. In the optimal offset search and completion box filtering module, the method for filtering the annotation boxes to be filled in the matching area from the translated template annotation box set includes: The translated template annotation box set Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries; the set of bounding boxes for the region to be matched. Any of the annotation boxes in the middle is denoted as Its coordinates are , , , , These are the annotation boxes. The coordinates of the left, top, right, and bottom boundaries; According to the above and stated The coordinates are used to calculate the overlap between the translated template annotation box and the annotation box of the area to be matched. : ， in, express and The intersection and union ratio, express and The distance between the center points, express and The length of the diagonal of the minimum bounding rectangle. It is a smoothing factor; According to the degree of overlap ,calculate P annotation boxes and The sum of the overlap of the Q bounding boxes represents the overlap between the translated template area and the area to be matched. : ， The search offset combinations within the search space are divided into multiple data batches, each batch containing a subset of offset combinations. The GPU is used to compute in parallel the values ​​corresponding to all offset combinations within each batch. The value is used to obtain the overlap rate in each batch. The maximum combination of search offsets and the corresponding set of translated template annotation boxes ; The optimal result obtained by comparing each batch Value, to determine the degree of overlap The largest combination of search offsets is taken as the optimal offset, and the corresponding optimal offset is... In the process, select those that are related to The annotation boxes that do not overlap are the annotation boxes to be filled in the area to be matched; The annotation data integration and output module is configured to merge the acquired annotation boxes to be filled with the manually annotated annotation boxes, remove the two types of auxiliary boxes, template area boxes and matching area boxes, and obtain the annotation data of all liquid crystal targets in the LCD product image.

9. A computer storage medium, characterized in that, The computer storage medium stores a computer software product, which includes several instructions for causing a computer device to execute the automatic image annotation missing method based on region bounding box matching degree search as described in any one of claims 1 to 7.

Images