Image annotation devices, annotation methods, and training sample construction methods

By using an image annotation device and method, and by selecting magnified areas using thumbnails and performing coordinate transformation, the problem of low efficiency in ultra-high resolution image annotation is solved, enabling efficient annotation and training sample preparation for planar products such as LCD panels, silicon wafers, and substrates.

CN115937348BActive Publication Date: 2026-11-10INTERFACE TECH (CHENGDU) CO LTD +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211705070.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-29
Publication Date
2026-11-10
Estimated Expiration
2042-12-29

AI Technical Summary

Technical Problem

In existing technologies, the annotation process for ultra-high resolution images is inefficient, especially in the inspection of defects in planar products such as LCD panels, silicon wafers, and substrates. The image loading and rendering process is time-consuming, resulting in extremely low efficiency in preparing training sample data.

Method used

An image annotation device is used, including a storage unit, a cache unit, a display unit, a calibration unit, an extraction unit, a coordinate transformation unit, and a synchronization unit. It selects the magnified area through thumbnails and extracts local data from the original image for annotation based on coordinate transformation, avoiding repeated data loading and rendering processes.

Benefits of technology

It improves the annotation efficiency of ultra-high resolution images, reduces the dependence on the performance of annotation devices, and is suitable for ultra-high resolution image annotation and training sample preparation of planar products such as LCD panels, silicon wafers, and substrates. It also reduces rendering latency and improves operational efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115937348B_ABST
    Figure CN115937348B_ABST
Patent Text Reader

Abstract

This invention belongs to the field of image processing technology, and provides an image annotation device, annotation method, and training sample construction method. The device includes a storage unit, a cache unit, a display unit, a calibration unit, an extraction unit, a coordinate transformation unit, and a synchronization unit. The cache unit ensures the read / write speed of the device throughout the annotation process, avoiding latency caused by reading and writing large amounts of data. The calibration unit, extraction unit, transformation unit, and synchronization unit complete the switching between thumbnails and the original image, avoiding latency caused by real-time processing of large amounts of data. Therefore, it is highly efficient for annotating ultra-high resolution images and reduces the performance dependence on the annotation device, making it suitable for annotating ultra-high resolution images of planar products such as LCD panels, silicon wafers, and substrates, and for preparing training samples.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and particularly relates to an image annotation device, annotation method, and training sample construction method. Background Technology

[0002] Semantic segmentation involves dividing an image into several visually meaningful or interesting regions, identifying the semantic category of each region, and performing a semantic reasoning process from low to high levels to ultimately obtain a segmented image with pixel-by-pixel semantic annotations. This facilitates subsequent image analysis and visual understanding. Since the emergence of semantic segmentation models represented by fully convolutional neural networks (FCNs), this method has been widely applied to various image classification and recognition tasks, including object recognition and detection, intelligent security, autonomous driving technology, human-computer interaction, and virtual reality, achieving excellent results and experiencing significant development. Common semantic segmentation models include U-NET, SegNet, DeepLab, DilatedConvolutions, RefineNet, and PSPNet. Therefore, existing technologies also include extensive research and applications using semantic segmentation to identify and judge product appearance defects.

[0003] However, it should be noted that the utilization of semantic segmentation models depends on model training. Therefore, preparing training sample data for the corresponding scenarios is a prerequisite for using semantic segmentation models. Accordingly, existing technologies have developed software tools for preparing training sample data for semantic segmentation models. However, these tools are typically applicable to fields such as medical imaging, intelligent security, autonomous driving technology, human-computer interaction, and virtual reality. When used in industrial automation fields where semantic segmentation is used to identify and judge product appearance defects, the user experience is poor due to the significant differences in application scenarios across industrial sectors.

[0004] Typically, in the inspection of appearance defects in planar products such as LCD panels, silicon wafers, and substrates, defects are extremely small relative to the overall product size, such as dead pixels in LCDs. Therefore, when using semantic segmentation models to identify product appearance defects, ultra-high resolution images are required. Taking the defect inspection of a certain LCD panel as an example, the original image used for defect inspection has an ultra-high resolution of 16384x22500, and the original image file size is 242MB. Therefore, in the preparation of training samples, i.e., the image annotation process, due to the limitation of the physical resolution of the display unit, it is necessary to continuously zoom in, zoom out, and drag the image to complete the annotation of the entire image. During the zooming in, zooming out, and moving of the image, different display areas need to be rendered in real time. Therefore, during the operation, there is a data loading and rendering process from blurry to clear, making the annotation process extremely time-consuming. Taking the defect inspection of the aforementioned LCD panel as an example, each loading and rendering process takes about 10 seconds. Therefore, the efficiency of training sample data preparation is extremely low. Summary of the Invention

[0005] The technical problem to be solved by this invention is to propose an image annotation device, annotation method and training sample construction method, which is highly efficient for the annotation of ultra-high resolution images and is applicable to the annotation of ultra-high resolution images of planar products such as liquid crystal panels, silicon wafers, and substrates and the preparation of training samples.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] The image annotation device includes a storage unit, a cache unit, a display unit, a calibration unit, an extraction unit, a coordinate transformation unit, and a synchronization unit;

[0008] The storage unit is used to store the original image, as well as a thumbnail of the original image, interest point type annotations, and interest point boundary coordinates;

[0009] The cache unit is used to load all the data of the original image and the thumbnail of the original image;

[0010] The calibration unit is used to specify a magnified area on the display image of the thumbnail, and to mark the type and boundary of the point of interest on the display image of the local original image corresponding to the magnified area;

[0011] The extraction unit is used to extract the coordinates of the boundary of the magnified region in the thumbnail during the image annotation process, and to extract the coordinates of the boundary of the point of interest in the local original image corresponding to the magnified region.

[0012] The conversion unit is used for coordinate transformation of the boundary of the magnified area between the thumbnail and the original image, and coordinate transformation of the boundary of the point of interest between the original image and the local original image;

[0013] The synchronization unit is used to extract image data of the local original image corresponding to the magnified area based on the coordinates of the magnified area boundary in the original image obtained by coordinate transformation of the magnified area boundary in the thumbnail;

[0014] The display unit is used to display the thumbnail and the local original image corresponding to the magnified area.

[0015] The present invention also provides an image annotation method, which uses the image annotation device described above for annotation, and includes the following steps:

[0016] S1. Load all data of the original image and the thumbnail of the original image into the cache unit;

[0017] S2. Display the thumbnail in the thumbnail window of the display unit, and specify the magnified area of ​​the thumbnail on the displayed image of the thumbnail through the calibration unit; then, extract the coordinates of the boundary of the magnified area in the thumbnail through the extraction unit.

[0018] S3. The coordinates of the boundary of the magnified area in the thumbnail are converted into the coordinates of the boundary of the magnified area in the original image by the conversion unit.

[0019] S4. Based on the coordinates of the boundary of the magnified region in the original image obtained by the conversion, the synchronization unit extracts the image data of the local original image within the boundary range of the magnified region from the original image;

[0020] S5. Based on the extracted image data of the local original image, display the local original image in the detail window of the display unit; and identify points of interest in the displayed image of the local original image;

[0021] S6. Based on the identified points of interest, the type and boundary of the points of interest are marked on the display image of the local original image using a calibration unit;

[0022] S7. The extraction unit extracts the coordinates of the boundary of the interest point in the local original image, and the conversion unit converts the coordinates of the boundary of the interest point in the local original image into the coordinates of the boundary of the interest point in the original image; the type label of the interest point and the coordinates of the boundary in the original image are stored in the storage unit.

[0023] S8. Repeat steps S2 to S7 until all interest points in the original image are labeled.

[0024] Furthermore, in step S2, specifying the magnified area of ​​the thumbnail through the calibration unit includes the following steps:

[0025] First, based on the pixel values ​​of the magnified area of ​​the specified thumbnail, and based on the magnification factor of the original image relative to the thumbnail, the pixel values ​​of the corresponding local original image are calculated.

[0026] Then, based on a preset threshold, it is determined whether the pixel values ​​of the calculated local original image match the physical pixel values ​​of the detail window of the display unit. If they match, the specified magnified area is completed; otherwise, a prompt is made to specify it again.

[0027] As a preferred embodiment, the pixels of the local original image correspond one-to-one with the physical pixels of the detail window of the display unit; the preset threshold is: the pixel width of the local original image is less than or equal to the physical pixel width of the detail window of the display unit; at the same time, the pixel height of the local original image is less than or equal to the physical pixel height of the detail window of the display unit.

[0028] As a preferred embodiment, in step S2, the calibration unit uses a rectangular frame, and selects the magnified area of ​​the thumbnail by specifying any two diagonally distributed vertices of the rectangular frame. Alternatively, in step S2, the calibration unit uses a rectangular frame with a preset pixel specification to select the magnified area of ​​the thumbnail.

[0029] As a preferred embodiment, in step S6, the calibration unit delineates and marks the boundaries of the points of interest based on the cursor movement trajectory.

[0030] As a preferred embodiment, the thumbnail of the original image is a thumbnail of the region to be labeled in the original image;

[0031] In step S2, the extraction unit extracts the coordinates of the magnified area of ​​the thumbnail in the thumbnail based on the origin of the thumbnail.

[0032] In step S3, the conversion unit converts the coordinates of the boundary of the magnified area in the thumbnail into the coordinates of the boundary of the magnified area in the original image, based on the magnification factor between the region to be labeled in the original image and the coordinates of the origin of the thumbnail in the original image.

[0033] Furthermore, step S7 includes the following steps:

[0034] First, the extraction unit extracts the coordinates of the boundary of the point of interest in the local original image based on the origin of the local original image.

[0035] Then, through the conversion unit, based on the magnification factor of the magnified area of ​​the local original image relative to the thumbnail, and the coordinates of the origin of the local original image in the thumbnail, the coordinates of the boundary of the point of interest in the local original image are converted into the coordinates of the boundary of the point of interest in the thumbnail.

[0036] Subsequently, through the conversion unit, based on the magnification factor between the region to be labeled in the original image and the thumbnail, and the coordinates of the origin of the thumbnail in the original image, the coordinates of the boundary of the point of interest in the thumbnail are converted into the coordinates of the boundary of the point of interest in the original image.

[0037] The present invention also provides a method for constructing training samples, comprising the following steps:

[0038] First, all the image samples to be labeled are labeled according to the image labeling method described above, so as to obtain the type label and boundary coordinates of all interest points of each image sample;

[0039] Then, according to the format requirements of the semantic segmentation model training sample data, each image sample is cropped based on the coordinates of the boundaries of each interest point in each image sample to obtain the training sample composed of each interest point in each image sample.

[0040] Furthermore, the training sample data for the semantic segmentation model is required to be in the format of images with the same pixel width and height.

[0041] Based on the coordinates of the boundaries of each interest point in each image sample, the image samples are cropped, including the following steps:

[0042] First, determine the minimum bounding square of the current point of interest based on its boundary, and then crop the image corresponding to the point of interest based on the minimum bounding square.

[0043] Then, based on the pixel values ​​of the cropped interest point image and the format requirements of the input data of the semantic segmentation model, a judgment is made, and the results are processed as follows:

[0044] If the pixel value of the cropped point of interest image is less than the required pixel value, the cropped point of interest image is upsampled until the pixel value of the point of interest image matches the required pixel value, and the training sample image corresponding to the point of interest is obtained.

[0045] If the pixel value of the cropped interest point image is equal to the pixel value required by the format, then the training sample image corresponding to that interest point is obtained.

[0046] If the pixel value of the cropped point of interest image is greater than the required pixel width and height, the cropped point of interest image is downsampled until the pixel value of the point of interest image matches the required pixel value, thus obtaining the training sample image corresponding to that point of interest.

[0047] The beneficial effects of this invention are:

[0048] This invention avoids latency caused by repeatedly loading data by fully loading all the data of the original image. Secondly, it loads and displays a thumbnail of the original image, allowing selection of a magnified area based on the thumbnail. Then, based on coordinate transformation, it extracts a local portion of the original image data corresponding to the magnified area from the entire image for display. This local original image data originates from the original image; therefore, its clarity meets the requirements for detail display, facilitating the identification and annotation of defects. Simultaneously, the local original image has fewer pixels and a smaller data volume, resulting in faster rendering speed, better matching the physical resolution of the display unit, and less computational load. Therefore, it can complete rendering and display more quickly, avoiding latency caused by rendering.

[0049] Therefore, the annotation method of the present invention is highly efficient for annotating ultra-high resolution images and reduces the performance dependence on the annotation device. It is suitable for annotating ultra-high resolution images of planar products such as liquid crystal panels, silicon wafers, and substrates and for preparing training samples, which facilitates the application of semantic segmentation models in defect identification of planar products such as liquid crystal panels, circuit boards, and substrates. Attached Figure Description

[0050] Figure 1 This is a diagram of the user interface of Embodiment 1 of the present invention;

[0051] Figure 2 This is a flowchart illustrating the annotation method of the present invention.

[0052] Figure reference numerals: 1. Thumbnail; 2. Rectangle; 3. Partial original image; 4. Boundary of point of interest; 5. Pixel. Detailed Implementation

[0053] To enable those skilled in the art to better understand the technical solutions of this invention, the invention will be clearly and completely described below with reference to embodiments. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort should fall within the scope of protection of this invention.

[0054] Example 1

[0055] The image annotation device includes a storage unit, a cache unit, a display unit, a calibration unit, an extraction unit, a coordinate transformation unit, and a synchronization unit.

[0056] The storage unit is used to store the original image, as well as a thumbnail of the original image, interest point type labels, and interest point boundary coordinates.

[0057] The cache unit is used to load all the data of the original image and the thumbnail of the original image.

[0058] The calibration unit is used to specify a magnified area on the displayed image of the thumbnail, and to mark the type and boundary of the point of interest on the displayed image of the local original image corresponding to the magnified area.

[0059] The extraction unit is used during the image annotation process to extract the coordinates of the boundary of the magnified region in the thumbnail, and to extract the coordinates of the boundary of the point of interest in the local original image corresponding to the magnified region.

[0060] The conversion unit is used for coordinate transformation of the boundary of the magnified region between the thumbnail and the original image, and coordinate transformation of the boundary of the point of interest between the original image and the local original image.

[0061] The synchronization unit is used to extract image data of the local original image corresponding to the magnified area based on the coordinates of the boundary of the magnified area in the original image obtained by coordinate transformation of the boundary of the magnified area in the thumbnail.

[0062] The display unit is used to display the thumbnail and the local original image corresponding to the magnified area.

[0063] This device ensures high read / write speed throughout the annotation process through a high-speed cache unit, avoiding latency caused by reading and writing large amounts of data. Through calibration, extraction, transformation, and synchronization units, it completes the switching between thumbnails and original images, avoiding latency caused by real-time processing of large amounts of data. Therefore, the annotation efficiency is high. Its user interface is as follows... Figure 1 As shown, the thumbnail and a portion of the original image are displayed in separate sections on the same screen. Figure 1 The partial original image 3, displayed in the upper left corner of the interface, is displayed on the right side of the interface.

[0064] Example 2

[0065] This embodiment describes an image annotation method using the image annotation device of Embodiment 1. The process is as follows: Figure 2 As shown, it includes the following steps:

[0066] S1, Data Loading

[0067] Thumbnails can be generated based on all or part of the data from the original image. For example, when only a certain area of ​​a product is labeled, a thumbnail can be generated based on only part of the data corresponding to that area.

[0068] Thumbnail generation can occur before this step and be loaded during this step, for example, by generating the thumbnail synchronously after shooting. However, this method is mainly suitable for thumbnails generated based on all data from the original image. Therefore, to further reduce the data volume and improve flexibility and adaptability to different product models and annotation requirements, optimally, the area to be annotated is defined on the original image, and the thumbnail of the original image is the thumbnail of the area to be annotated in the original image. In this step, the thumbnail is compressed and generated in real time and loaded into the cache unit.

[0069] Therefore, in this embodiment, the first step is to load all the data of the original image into the cache unit. The original image is an image of the liquid crystal panel with a resolution of 22500x16384, a file size of 242MB, and an aspect ratio of 19:10.

[0070] Then, thumbnail generation and loading are performed. Specifically, in this embodiment, the product is photographed in the center, and the area outside the product region in the original image is the background area. Therefore, the product region in the original image is defined as the region to be labeled. Meanwhile, since the original image in this embodiment is taken from a standardized production line, the product shooting position is fixed, and the position of the product region in the original image is also relatively fixed. Therefore, the data extraction of the region to be labeled in the original image is performed by a synchronization unit, directly cropping the area according to the set pixel coordinates of the product region in the original image. The resolution of the cropped region to be labeled is 21600×14400, and the aspect ratio is 15:10. Afterwards, based on the data of the cropped region to be labeled, a thumbnail with a resolution of 675×450 is compressed and generated, and loaded into the cache unit.

[0071] Of course, the area outside the product area in the original image is the background color, which contrasts strongly with the product. Therefore, in addition to the direct cropping method mentioned above, edge extraction algorithms can also be used to obtain the edge information of the product area, thereby completing the data extraction of the product area in the original image.

[0072] S2, Specify the zoom area

[0073] In this step, the loaded thumbnail is rendered and displayed in the thumbnail window of the display unit. The magnified area of ​​the thumbnail is specified on the displayed image of the thumbnail by the calibration unit. Then, the coordinates of the boundary of the magnified area in the thumbnail are extracted by the extraction unit.

[0074] Furthermore, in this step, specifying the magnified area of ​​the thumbnail through the calibration unit includes the following steps:

[0075] First, based on the pixel values ​​of the magnified area of ​​the specified thumbnail, and based on the magnification factor of the original image relative to the thumbnail, the pixel values ​​of the corresponding local original image are calculated.

[0076] Then, based on a preset threshold, it is determined whether the pixel values ​​of the calculated local original image match the physical pixel values ​​of the detail window of the display unit. If they match, the specified magnified area is completed; otherwise, a prompt is made to specify it again.

[0077] The above-mentioned determination of whether the pixel values ​​of the local original image obtained by calculation match the physical pixel values ​​of the display unit's detail window is only a calculation and matching of pixel values, not an actual extraction of the local original image. Its purpose is to determine the compatibility between the pixels of the local original image and the material pixels of the display unit's detail window, that is, the correspondence between the two.

[0078] Specifically, the pixel values ​​of the original local image and the material pixels in the detail view window of the display unit can be categorized into three cases: greater than, equal to, and less than. For cases where the pixel value is greater than, the original local image needs to be compressed to meet the display requirements of the detail view window; alternatively, after display, the original local image needs to be dragged within the detail view window to meet browsing requirements. For cases where the pixel value is equal to, it can be displayed directly. For cases where the pixel value is less than, the original local image needs to be interpolated to meet the display requirements of the detail view window; alternatively, the display requirements can be met by filling the area around the original local image with a background color. The specific settings can be configured according to the needs of the application scenario.

[0079] In the above processing methods, image compression and scaling may lead to loss of detail and increased latency; image interpolation and scaling may lead to mosaic effects and increased latency. Therefore, in this embodiment, the pixels of the local original image correspond one-to-one with the physical pixels of the detail window of the display unit; the preset threshold is: the pixel width of the local original image is less than or equal to the physical pixel width of the detail window of the display unit; simultaneously, the pixel height of the local original image is less than or equal to the physical pixel height of the detail window of the display unit. Therefore, by defining the area to be annotated to reduce the amount of data, and by reducing or avoiding the rendering computation of interpolation scaling and compression / scaling, fast display is further guaranteed.

[0080] In this embodiment, the display is a 4K screen with a resolution of 3840×2160, and the user interface is as follows. Figure 1 As shown, the thumbnail and a portion of the original image are displayed in separate sections on the same screen. Figure 1 Displayed in the upper left corner of the interface, used to show the thumbnail. Figure 1 The physical pixel size of the thumbnail window is 675×450; the local original image 3 is displayed on the right side of the operation interface. The physical pixel size of the detail window used to display the local original image 3 is 1440×1440, that is, the maximum pixel value of the local original image 3 is 1440×1440, and the pixel value of the specified magnified area does not exceed 45×45.

[0081] The magnified area can be specified using any selection method. In this embodiment, the calibration unit uses a rectangle, and the magnified area of ​​the thumbnail is selected by specifying any two diagonally distributed vertices of the rectangle. Figure 1 In the image, the pixel value of rectangle 2 is 45×24.

[0082] After selecting the enlarged area of ​​the thumbnail, the coordinates of the boundary of the enlarged area in the thumbnail can be extracted using the extraction unit.

[0083] S3, Zoom-in area coordinate transformation

[0084] In this step, the coordinates of the boundary of the magnified area in the thumbnail are converted by the conversion unit into the coordinates of the boundary of the magnified area in the original image.

[0085] Coordinate transformation could be achieved by directly generating a coordinate lookup table between pixels in the original image and pixels in the thumbnail during thumbnail generation, and then performing the coordinate transformation by looking up the table during the conversion process. However, this method involves a large amount of data and processing. Therefore, in this embodiment, a real-time calculation method is used for coordinate transformation.

[0086] Specifically, in this embodiment, the thumbnail of the original image is a thumbnail of the region to be labeled in the original image. Therefore, in the preceding step S2, the extraction unit extracts the coordinates of the magnified region in the thumbnail based on the origin (0,0), i.e., the top-left vertex of the thumbnail. In this step, the conversion unit converts the coordinates of the boundary of the magnified region in the thumbnail to the coordinates of the region to be labeled in the original image and the magnification factor between the region to be labeled in the original image and the thumbnail, as well as the coordinates of the origin of the thumbnail in the original image. The calculation formula is as follows:

[0087] a = x' + a' * (w / w')

[0088] b = y' + b' * (h / h')

[0089] Where 'a' represents the x-coordinate of the boundary pixel in the original image, 'b' represents the y-coordinate of the boundary pixel in the original image, 'w' is the pixel width of the area to be labeled in the original image, and 'h' is the pixel height of the area to be labeled in the original image; 'a' represents the x-coordinate of the boundary pixel in the thumbnail, 'b' represents the y-coordinate of the boundary pixel in the thumbnail, 'w' is the pixel width of the thumbnail, and 'h' is the pixel height of the thumbnail; 'x' represents the x-coordinate of the origin of the thumbnail in the original image, and 'y' represents the y-coordinate of the origin of the thumbnail in the original image. 'w / w' represents the magnification factor in the width direction between the area to be labeled in the original image and the thumbnail, and 'h / h' represents the magnification factor in the height direction between the area to be labeled in the original image and the thumbnail.

[0090] In this embodiment, the pixel value of the area to be labeled in the original image is 21600×14400, the pixel value of the thumbnail is 675×450, and the magnification factor in both the width and height directions is 32 times. Figure 1 In the image, for ease of layout, the vertical axis represents the width, corresponding to the x-axis; the horizontal axis represents the height, corresponding to the y-axis. Taking the top-left corner of rectangle 2 in the image as an example, its position in the thumbnail... Figure 1 The coordinates in the image are (40, 80), abbreviated. Figure 1 The coordinates of the origin in the original image are (450, 992). Therefore, its coordinates in the original image are:

[0091] a = 450 + 40 * 32 = 1730

[0092] b = 768 + 80 * 32 = 3552

[0093] S4. Extract local original image

[0094] Based on the coordinates of the boundary of the magnified region in the original image obtained from the conversion, the synchronization unit extracts the image data of the local original image within the boundary range of the magnified region from the original image.

[0095] S5. Identify Points of Interest

[0096] Based on the extracted image data of the local original image, the local original image is displayed in the detail window of the display unit; and points of interest are identified in the displayed image of the local original image. In this example, the points of interest are the locations of defects in the product.

[0097] S6, Point of Interest (POI) Labeling

[0098] Based on the identified points of interest, the type and boundary of the points of interest are labeled on the displayed image of the local original image using a calibration unit.

[0099] To accommodate the uncertainty of the shape of the points of interest and facilitate image cropping in the subsequent sample production process, in this embodiment, the calibration unit delineates and marks the boundaries of the points of interest based on the cursor movement trajectory.

[0100] S7. Extract and store point-of-interest information.

[0101] The extraction unit extracts the coordinates of the boundary of the interest point in the local original image, and the conversion unit converts the coordinates of the boundary of the interest point in the local original image into the coordinates of the boundary of the interest point in the original image; the type label of the interest point and the coordinates of the boundary in the original image are stored in the storage unit.

[0102] Since the boundaries of the points of interest are marked on the displayed image of the local original image, and in this embodiment, the pixels of the local original image correspond one-to-one with the physical pixels of the detail window of the display unit, the relationship between the local original image and the original image is that of a local to a global relationship, and their coordinate systems are the same. Therefore, the coordinates can be transformed using an offset function. That is, the coordinates of the boundary of the point of interest in the original image are equal to the coordinates of the boundary of the point of interest in the local original image plus the coordinates of the origin of the local original image in the original image. The coordinates of the origin of the local original image can be obtained in step S3 above. However, this method is only applicable when the pixels of the local original image correspond one-to-one with the physical pixels of the detail window of the display unit.

[0103] Therefore, to ensure the universality of coordinate transformation and facilitate adaptation to more scenarios by modifying the configuration, this embodiment uses thumbnails as an intermediary for coordinate transformation. Specifically, it includes the following steps:

[0104] First, the extraction unit extracts the coordinates of the boundary of the point of interest in the local original image based on the origin (0,0) of the local original image, which is the top left corner vertex of the local original image.

[0105] Then, through the conversion unit, based on the magnification factor of the magnified area of ​​the local original image relative to the thumbnail, and the coordinates of the origin of the local original image in the thumbnail, the coordinates of the boundary of the point of interest in the local original image are converted into the coordinates of the boundary of the point of interest in the thumbnail. The calculation formula is as follows:

[0106] a' = x” + a” * (w” / w”')

[0107] b' = y” + b” * (h” / h”')

[0108] Where a' represents the x-coordinate of the boundary pixel in the thumbnail, b' represents the y-coordinate of the boundary pixel in the thumbnail, w” is the pixel width of the magnified area of ​​the thumbnail, and h” is the pixel height of the magnified area of ​​the thumbnail; a” represents the x-coordinate of the boundary pixel in the local original image, b” represents the y-coordinate of the boundary pixel in the local original image, w”' is the pixel width of the local original image, and h”' is the pixel height of the local original image; x” represents the x-coordinate of the origin of the local original image in the thumbnail, and y” represents the y-coordinate of the origin of the local original image in the thumbnail. w” / w”' is the magnification factor in the width direction between the magnified area of ​​the thumbnail and the local original image, and h” / h”' is the magnification factor in the height direction between the magnified area of ​​the thumbnail and the local original image.

[0109] Then, through the conversion unit, based on the magnification factor between the region to be labeled in the original image and the thumbnail, and the coordinates of the origin of the thumbnail in the original image, the coordinates of the boundary of the point of interest in the thumbnail are converted into the coordinates of the boundary of the point of interest in the original image. The calculation formula is as follows:

[0110] a = x' + a' * (w / w')

[0111] b = y' + b' * (h / h')

[0112] Where, 'a' represents the x-coordinate of the boundary pixel in the original image, 'b' represents the y-coordinate of the boundary pixel in the original image, 'w' represents the pixel width of the area to be labeled in the original image, and 'h' represents the pixel height of the area to be labeled in the original image; 'a' represents the x-coordinate of the boundary pixel in the thumbnail, 'b' represents the y-coordinate of the boundary pixel in the thumbnail, 'w' represents the pixel width of the thumbnail, and 'h' represents the pixel height of the thumbnail; 'x' represents the x-coordinate of the origin of the thumbnail in the original image, and 'y' represents the y-coordinate of the origin of the thumbnail in the original image; 'w / w' represents the magnification factor in the width direction between the area to be labeled in the original image and the thumbnail, and 'h / h' represents the magnification factor in the height direction between the area to be labeled in the original image and the thumbnail.

[0113] In this embodiment, the pixel value of the area to be labeled in the original image is 21600×14400, the pixel value of the thumbnail is 675×450, and the magnification factor in both the width and height directions is 32 times; the pixel value of the selected magnified area does not exceed 45×45, and the pixel value of the local original image does not exceed 1440×1440, with a magnification factor in both the width and height directions of 1 / 32 times. Figure 1 Taking pixel 5 on the boundary 4 of the point of interest as an example, its coordinates in the local original image 3 are (160, 448), abbreviated. Figure 1 The coordinates of the origin in the original image are (450, 992). The coordinates of the origin in the local original image 3 are at the scaled-down location. Figure 1 The coordinates of the point are (40, 80). Therefore, the coordinates of pixel 5 on the boundary 4 of the point of interest in the original image are:

[0114] a' = 40 + 160 * 1 / 32 = 45

[0115] b' = 80 + 448 * 1 / 32 = 94

[0116] a = 450 + 45 * 32 = 1890

[0117] b = 992 + 94 * 32 = 4000

[0118] If a decimal occurs during the calculation, it should be rounded down to the nearest integer.

[0119] S8. Repeat steps S2 to S7 until all interest points in the original image are labeled.

[0120] Using a hardware configuration of i7 (CPU) / 16g (RAM) / 512G (HD), the method of this embodiment was tested in practice. The extraction and display of the local original image data corresponding to the selected magnified area took less than 1 second. In contrast, in the prior art, the image reading time is about 3 seconds, the rendering and display time is about 7 seconds, and the overall latency reaches 10 seconds when switching the display area.

[0121] Example 3

[0122] This embodiment is also an image annotation method. The only difference between it and Embodiment 2 is that in step S2, the calibration unit uses a rectangular frame with a preset pixel specification to select the magnified area of ​​the specified thumbnail.

[0123] The preset pixel specifications of the rectangle are set according to the data format of the final training samples. In this embodiment, the data format of the final training samples is 512×512. Therefore, the pixel value of the local original image is set to 2048×2048, and the preset pixel specifications of the rectangle are set to 64×64.

[0124] Besides the rectangular boxes used in Embodiments 2 and 3, the magnified area can also be specified by other shapes of selection boxes. However, the input data format requirements for semantic segmentation models are usually square images, so rectangular boxes are more convenient.

[0125] Example 4

[0126] This embodiment is also an image annotation method, and its difference from Embodiment 2 is only that: the thumbnail is generated based on all the data of the original image, and the thumbnail is generated synchronously after the image is captured. In this step S1, the original image and its thumbnail are directly loaded. At this time, the area to be annotated in the original image is its entire area. Therefore, in the subsequent coordinate transformation process, the coordinates of the origin of the thumbnail coordinates in the original image are (0,0).

[0127] Example 5

[0128] This embodiment describes a method for constructing training samples for a semantic segmentation model, including the following steps:

[0129] First, all the image samples to be labeled are labeled according to the image labeling method described in Example 2 to obtain the type labeling and boundary coordinates of all interest points of each image sample;

[0130] Then, according to the format requirements of the semantic segmentation model training sample data, each image sample is cropped based on the coordinates of the boundaries of each interest point in each image sample to obtain the training sample composed of each interest point in each image sample.

[0131] Typically, training sample data for semantic segmentation models requires images with the same pixel width and height. In this embodiment, the final training sample data format is 512×512. Since defects are usually small, i.e., fewer pixels, further cropping of each image sample is performed based on the coordinates of the boundaries of each interest point. This includes the following steps:

[0132] First, determine the minimum bounding square of the current point of interest based on its boundary, and then crop the image corresponding to the point of interest based on the minimum bounding square.

[0133] Then, based on the pixel values ​​of the cropped interest point image and the format requirements of the input data of the semantic segmentation model, a judgment is made, and the results are processed as follows:

[0134] If the pixel value of the cropped point of interest image is less than the required pixel value, the cropped point of interest image is upsampled until the pixel value of the point of interest image matches the required pixel value, and the training sample image corresponding to the point of interest is obtained.

[0135] If the pixel value of the cropped interest point image is equal to the pixel value required by the format, then the training sample image corresponding to that interest point is obtained.

[0136] If the pixel value of the cropped point of interest image is greater than the required pixel width and height, the cropped point of interest image is downsampled until the pixel value of the point of interest image matches the required pixel value, thus obtaining the training sample image corresponding to that point of interest.

[0137] It should be noted that the above embodiments are merely preferred embodiments and are not intended to limit the present invention. It should be pointed out that those skilled in the art can make various modifications, equivalent substitutions, and improvements without departing from the spirit and scope of the claims, and all such modifications, substitutions, and improvements should be included within the scope of protection of the present invention.

Claims

1. An image annotation device, characterized in that: It includes a storage unit, a cache unit, a display unit, a calibration unit, an extraction unit, a coordinate transformation unit, and a synchronization unit; The storage unit is used to store the original image, as well as a thumbnail of the original image, interest point type annotations, and interest point boundary coordinates; The cache unit is used to load all the data of the original image and the thumbnail of the original image; the thumbnail of the original image is a thumbnail of the area to be labeled in the original image; The calibration unit is used to specify a magnified area on the display image of the thumbnail, and to mark the type and boundary of the point of interest on the display image of the local original image corresponding to the magnified area; The extraction unit is used in the image annotation process to extract the coordinates of the boundary of the magnified area in the thumbnail based on the origin of the thumbnail, and to extract the coordinates of the boundary of the point of interest in the local original image corresponding to the magnified area based on the origin of the local original image. The conversion unit is configured to: convert the coordinates of the boundary of the magnified region in the thumbnail to the coordinates of the magnified region in the original image based on the magnification factor between the region to be labeled in the original image and the coordinates of the origin of the thumbnail in the original image; convert the coordinates of the boundary of the point of interest in the local original image to the coordinates of the boundary of the point of interest in the thumbnail based on the magnification factor between the magnified region of the local original image and the coordinates of the origin of the local original image in the thumbnail; and convert the coordinates of the boundary of the point of interest in the thumbnail to the coordinates of the boundary of the point of interest in the original image based on the magnification factor between the region to be labeled in the original image and the thumbnail and the coordinates of the origin of the thumbnail in the original image. The synchronization unit is used to extract the image data of the local original image corresponding to the magnified area from all the data of the original image loaded by the cache unit, based on the coordinates of the boundary of the magnified area in the original image obtained by coordinate transformation of the boundary of the magnified area in the thumbnail; The display unit is used to display the thumbnail and the local original image corresponding to the magnified area.

2. An image annotation method, characterized in that, The image annotation device as described in claim 1 is used for annotation, comprising the following steps: S1. Load all data of the original image and the thumbnail of the original image into the cache unit; the thumbnail of the original image is a thumbnail of the region to be labeled in the original image. S2. Display the thumbnail in the thumbnail window of the display unit, and specify the magnified area of ​​the thumbnail on the displayed image of the thumbnail through the calibration unit; then, extract the coordinates of the magnified area of ​​the thumbnail in the thumbnail based on the coordinate origin of the thumbnail through the extraction unit. S3. Through the conversion unit, based on the magnification factor between the region to be annotated in the original image and the thumbnail, and the coordinates of the origin of the thumbnail in the original image, the coordinates of the boundary of the magnified region in the thumbnail are converted into the coordinates of the boundary of the magnified region in the original image. S4. Based on the coordinates of the boundary of the magnified region in the original image obtained by the conversion, the synchronization unit extracts the image data of the local original image within the boundary range of the magnified region from the original image; S5. Based on the extracted image data of the local original image, display the local original image in the detail window of the display unit; and identify points of interest in the displayed image of the local original image; S6. Based on the identified points of interest, the type and boundary of the points of interest are marked on the display image of the local original image using a calibration unit; S7. Through the extraction unit, based on the coordinate origin of the local original image, the coordinates of the boundary of the point of interest in the local original image are extracted. Through the conversion unit, based on the magnification factor of the magnified area of ​​the local original image relative to the thumbnail, and the coordinates of the coordinate origin of the local original image in the thumbnail, the coordinates of the boundary of the point of interest in the local original image are converted into the coordinates of the boundary of the point of interest in the thumbnail. Then, based on the magnification factor between the region to be labeled in the original image and the thumbnail, and the coordinates of the origin of the thumbnail in the original image, the coordinates of the boundary of the point of interest in the thumbnail are converted into the coordinates of the boundary of the point of interest in the original image; the type label of the point of interest and the coordinates of the boundary in the original image are stored in the storage unit. S8. Repeat steps S2 to S7 until all interest points in the original image are labeled.

3. The image annotation method according to claim 2, characterized in that, In step S2, specifying the magnified area of ​​the thumbnail through the calibration unit includes the following steps: First, based on the pixel values ​​of the magnified area of ​​the specified thumbnail, and based on the magnification factor of the original image relative to the thumbnail, the pixel values ​​of the corresponding local original image are calculated. Then, based on a preset threshold, it is determined whether the pixel values ​​of the calculated local original image match the physical pixel values ​​of the detail window of the display unit. If they match, the specified magnified area is completed; otherwise, a prompt is made to specify it again.

4. The image annotation method according to claim 3, characterized in that, The pixels of the local original image correspond one-to-one with the physical pixels of the detail window of the display unit; the preset threshold is: the pixel width of the local original image is less than or equal to the physical pixel width of the detail window of the display unit; at the same time, the pixel height of the local original image is less than or equal to the physical pixel height of the detail window of the display unit.

5. The image annotation method according to claim 2, 3 or 4, characterized in that, In step S2, the calibration unit uses a rectangular frame, and selects the magnified area of ​​the thumbnail by specifying any two diagonally distributed vertices of the rectangular frame.

6. The image annotation method according to claim 2, 3 or 4, characterized in that, In step S2, the calibration unit uses a rectangular frame with a preset pixel specification to select the magnified area of ​​the specified thumbnail.

7. The image annotation method according to claim 2, 3 or 4, characterized in that, In step S6, the calibration unit delineates and marks the boundaries of the points of interest based on the cursor movement trajectory.

8. A method for constructing training samples, characterized in that, Includes the following steps: First, all the image samples to be labeled are labeled according to the image labeling method described in any one of claims 2 to 7, so as to obtain the type labeling and boundary coordinates of all interest points of each image sample; Then, according to the format requirements of the training sample data of the semantic segmentation model, each image sample is cropped according to the coordinates of the boundary of each interest point in each image sample to obtain the training sample composed of each interest point in each image sample. The training sample data for the semantic segmentation model must be in the format of images with the same pixel width and height. Based on the coordinates of the boundaries of each interest point in each image sample, the image samples are cropped, including the following steps: First, determine the minimum bounding square of the current point of interest based on its boundary, and then crop the image corresponding to the point of interest based on the minimum bounding square. Then, based on the pixel values ​​of the cropped interest point image and the format requirements of the input data of the semantic segmentation model, a judgment is made, and the results are processed as follows: If the pixel value of the cropped point of interest image is less than the required pixel value, the cropped point of interest image is upsampled until the pixel value of the point of interest image matches the required pixel value, and the training sample image corresponding to the point of interest is obtained. If the pixel value of the cropped interest point image is equal to the pixel value required by the format, then the training sample image corresponding to that interest point is obtained. If the pixel value of the cropped point of interest image is greater than the required pixel width and height, the cropped point of interest image is downsampled until the pixel value of the point of interest image matches the required pixel value, thus obtaining the training sample image corresponding to that point of interest.