Rectangular frame marking method, device, equipment and storage medium

Abstract

The present application discloses a rectangular frame annotation method, device, equipment and storage medium, which relates to the field of image recognition technology, including: obtaining the positioning coordinates of multiple objects in an image to be annotated, wherein the multiple objects include multiple targets; judging whether the multiple targets are partially blocked based on the positioning coordinates of the multiple targets; expanding the positioning coordinate coverage of the partially blocked target based on the positioning coordinates of the partially blocked target and the positioning coordinates of the object blocking the partially blocked target, so that the expanded positioning coordinate coverage of the partially blocked target includes the blocked area of ​​the partially blocked target; based on the expanded positioning coordinate coverage of the partially blocked target, drawing and inserting a rectangular frame corresponding to the partially blocked target. The present application reduces the gap between the positioning information of the partially blocked target and its actual positioning information, improves the accuracy of the image annotation data, and thereby improves the training effect of the deep neural network.

Description

Technical Field

[0001] The present application relates to the technical field of image recognition, and in particular to a rectangular frame labeling method, device, equipment and storage medium. Background Art

[0002] In the training process of deep neural networks, a large amount of image annotation data is required. At present, the method of drawing a rectangular frame for the target object in the image to be annotated and saving the category label of the target object is usually used for image annotation. In the process of drawing the rectangular frame, it is necessary to preprocess the image first, then perform edge detection on the preprocessed image to extract features, then identify the features to obtain the contour of the target object, fit the circumscribed rectangle to each target object contour, determine the positioning information of the target object, and finally draw and insert the rectangular frame according to the positioning information of the target object. However, since the rectangular frame is currently drawn and inserted based on the positioning information corresponding to the part of the target object exposed outside, when the target object is partially obscured, the positioning information of the target object is quite different from the actual positioning information, resulting in inaccurate image annotation, thereby affecting the training effect of the deep neural network. Summary of the invention

[0003] The main purpose of the present application is to provide a rectangular frame annotation method, device, equipment and storage medium, aiming to solve the technical problem of inaccurate image annotation data in the existing image.

[0004] To achieve the above purpose, the present application proposes a rectangular frame marking method, comprising: Acquire positioning coordinates of a plurality of objects in the image to be annotated, wherein the plurality of objects include a plurality of targets; Determining whether the multiple targets are partially blocked based on the positioning coordinates of the multiple targets; Based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target, expanding the positioning coordinate coverage range of the partially obscured target so that the expanded positioning coordinate coverage range of the partially obscured target includes the obscured area of ​​the partially obscured target; Based on the expanded positioning coordinate coverage of the partially obscured target, a rectangular frame corresponding to the partially obscured target is drawn and inserted.

[0005] In some embodiments, the step of obtaining the location coordinates of multiple objects in the image to be annotated includes: Preprocessing the image to be labeled to obtain a preprocessed image, wherein the preprocessing includes grayscale processing, denoising processing and binarization processing; Performing edge detection on the preprocessed image to obtain contours corresponding to multiple objects in the preprocessed image; Fit the circumscribed rectangle to the contour corresponding to each object to obtain the location coordinates of each object.

[0006] In some embodiments, the step of determining whether the multiple targets are partially blocked based on the positioning coordinates of the multiple targets includes: Based on the positioning coordinates corresponding to each target, the length and width of the circumscribed rectangle corresponding to each target are obtained; Based on the length and width of the circumscribed rectangle corresponding to each target, an average length and an average width of the circumscribed rectangles corresponding to all targets are obtained; Based on the length and width of the circumscribed rectangle corresponding to each target and the average length and average width of the circumscribed rectangles corresponding to all targets, it is determined whether each target is partially blocked.

[0007] In some embodiments, the step of determining whether each target is partially blocked based on the length and width of the bounding rectangle corresponding to each target and the average length and average width of the bounding rectangles corresponding to all targets includes: Setting a first threshold; Compare the length of the circumscribed rectangle corresponding to each target with the average length of the circumscribed rectangles corresponding to all targets to obtain a first ratio; Compare the width of the bounding rectangle corresponding to each target with the average width of the bounding rectangles corresponding to all targets to obtain a second ratio; Compare the first ratio and the first threshold, and the second ratio and the first threshold, and when the first ratio is not greater than the first threshold or the second ratio is not greater than the first threshold, determine that the target is partially obscured.

[0008] In some embodiments, the step of expanding the positioning coordinate coverage range of the partially obscured target based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target so that the expanded positioning coordinate coverage range of the partially obscured target includes the obscured area of ​​the partially obscured target comprises: Determining whether the positioning coordinates of the partially obscured target are included in the positioning coordinate coverage range of the object that obscures the partially obscured target; The positioning coordinates of the partially obscured target within the positioning coordinate coverage range of the object that obscures the partially obscured target are updated to expand the positioning coordinate coverage range of the partially obscured target so that the expanded positioning coordinate coverage range of the partially obscured target includes the obscured area of ​​the partially obscured target.

[0009] In some embodiments, the step of updating the positioning coordinates of the partially obscured target within the coverage range of the positioning coordinates of the object that obscures the partially obscured target comprises: Based on the positioning coordinates of the partially occluded target that are not included in the positioning coordinate coverage range of the object that occludes the partially occluded target and the average length and average width of the circumscribed rectangles corresponding to all targets, the positioning coordinates of the partially occluded target that are included in the positioning coordinate coverage range of the object that occludes the partially occluded target are updated.

[0010] In some embodiments, the method further comprises: Based on the coverage of the positioning coordinates of the target that is not partially blocked, a rectangular frame corresponding to the target that is not partially blocked is drawn and inserted.

[0011] In addition, to achieve the above purpose, the present application also proposes a rectangular frame marking device, including: An acquisition module, used to acquire the location coordinates of multiple objects in the image to be annotated, wherein the multiple objects include multiple targets; A judging module, configured to judge whether the plurality of targets are partially blocked based on the positioning coordinates of the plurality of targets; An expansion module, configured to expand the coverage range of the positioning coordinates of the partially obscured target based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target, so that the expanded coverage range of the positioning coordinates of the partially obscured target includes the obscured area of ​​the partially obscured target; A drawing module is used to draw and insert a rectangular frame corresponding to the partially blocked target based on the expanded positioning coordinate coverage of the partially blocked target.

[0012] In addition, to achieve the above-mentioned purpose, the present application also proposes a rectangular frame marking device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the rectangular frame marking method described above.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the rectangular frame marking method described above are implemented.

[0014] One or more technical solutions proposed in this application have at least the following technical effects: In the process of image annotation, the present application uses the positioning coordinates of all targets to be annotated to determine the partially occluded target, and according to the positioning coordinates of the partially occluded target and the object that occludes the partially occluded target, the positioning coordinate coverage range of the partially occluded target is expanded, and a rectangular frame of the partially occluded target is drawn and inserted based on the expanded positioning coordinate coverage range of the partially occluded target, so that the gap between the positioning information of the partially occluded target and its actual positioning information is reduced, thereby improving the accuracy of the image annotation data, and further improving the training effect of the deep neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0016] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A schematic diagram of a flow chart provided for an embodiment of a rectangular frame marking method of the present application; Figure 2 A brief flowchart diagram of the rectangular frame marking method embodiment of the present application; Figure 3 A schematic diagram of a partially obscured target provided by an embodiment of the rectangular frame marking method of the present application; Figure 4 This is a schematic diagram of the module structure of the rectangular frame marking device according to an embodiment of the present application; Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the rectangular frame marking method in the embodiment of the present application.

[0018] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0019] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0020] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0022] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0023] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0024] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0025] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0026] In the description of the embodiments of the present application, the technical terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the embodiments of the present application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the embodiments of the present application.

[0027] In the description of the embodiments of the present application, unless otherwise clearly specified and limited, technical terms such as "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal connection of two elements or the interaction relationship between two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the embodiments of the present application can be understood according to the specific circumstances.

[0028] In the training process of deep neural networks, a large amount of image annotation data is required. At present, the method of drawing a rectangular frame for the target object in the image to be annotated and saving the category label of the target object is usually used for image annotation. In the process of drawing the rectangular frame, it is necessary to preprocess the image first, then perform edge detection on the preprocessed image to extract features, then identify the features to obtain the contour of the target object, fit the circumscribed rectangle to each target object contour, determine the positioning information of the target object, and finally draw and insert the rectangular frame according to the positioning information of the target object. However, since the rectangular frame is currently drawn and inserted based on the positioning information corresponding to the part of the target object exposed outside, when the target object is partially obscured, the positioning information of the target object is quite different from the actual positioning information, resulting in inaccurate image annotation, thereby affecting the training effect of the deep neural network.

[0029] In order to solve the above-mentioned defects, the inventive concept of the present application is as follows: The present application proposes a rectangular frame annotation method. In the process of image annotation, the present application uses the positioning coordinates of all targets to be annotated to determine the partially occluded target, and according to the positioning coordinates of the partially occluded target and the object that occludes the partially occluded target, the positioning coordinate coverage range of the partially occluded target is expanded, and a rectangular frame of the partially occluded target is drawn and inserted based on the expanded positioning coordinate coverage range of the partially occluded target, so that the gap between the positioning information of the partially occluded target and its actual positioning information is reduced, thereby improving the accuracy of the image annotation data, and further improving the training effect of the deep neural network.

[0030] Based on the above considerations, in order to solve the problem of inaccurate existing image annotation data, after in-depth research, a rectangular frame annotation method, device, equipment and storage medium are proposed, which are suitable for image annotation scenarios.

[0031] In order to illustrate the technical solution of the present application, a specific embodiment is provided below for illustration.

[0032] The present application embodiment provides a rectangular frame marking method, referring to Figure 1 , Figure 1A schematic diagram of the flow chart provided for an embodiment of the rectangular frame marking method of the present application.

[0033] In this embodiment, the rectangular frame marking method includes steps S10 to S40: Step S10, obtaining the location coordinates of multiple objects in the image to be annotated.

[0034] It should be noted that the image to be annotated refers to the image data that has not yet been labeled or annotated in the image annotation task. The images to be annotated may come from various sources, such as surveillance cameras, mobile phone cameras, professional cameras, satellite images, etc. They may contain various types of content, such as people, animals, vehicles, buildings, natural landscapes, etc. The annotation process may involve manual annotation, semi-automatic annotation, or automatic annotation. In the training process of deep neural networks, in order to obtain a large amount of annotated data, automatic annotation methods are usually used to improve efficiency. After the annotation is completed, these labeled image data can be used to train machine learning or deep learning models. These models can learn how to recognize and classify different objects from the annotated data and make accurate predictions in future tasks. The quality of the images to be annotated is crucial to the training effect of the model. Therefore, when performing image annotation tasks, it is necessary to ensure the accuracy of the annotation of image data.

[0035] Multiple objects represent all types of content contained in the image to be annotated, such as people, animals, vehicles, buildings, natural landscapes, etc. Multiple targets represent the content that needs to be annotated in the image to be annotated.

[0036] Image annotation usually involves annotating a certain type of content in an image to be annotated. Therefore, multiple objects include not only content that needs to be annotated, but also content that does not need to be annotated, that is, multiple objects include multiple targets.

[0037] The positioning coordinates are used to indicate the location of each object in the image to be annotated.

[0038] Exemplarily, the coordinates of the four vertices of the circumscribed rectangle corresponding to each object in the image to be annotated are used as the positioning coordinates of the object.

[0039] In a feasible implementation, step S10 may include steps S11 to S13: Step S11, preprocessing the image to be annotated to obtain a preprocessed image.

[0040] It should be noted that before image annotation, it is usually necessary to preprocess the images to ensure their quality and consistency. These preprocessing processes help reduce irrelevant information in the images and improve the accuracy and efficiency of annotation.

[0041] In a feasible implementation manner, the preprocessing includes grayscale processing, denoising processing and binarization processing.

[0042] Image grayscale processing is an important part of digital image processing, which involves the process of converting color images into grayscale images. Grayscale images, also known as grayscale images, are images represented by grayscale. In this kind of image, each pixel has only one sampled color, and its brightness value varies from 0% (pure white) to 100% (pure black), usually quantized into 256 gray levels (0~255). Common grayscale processing methods include weighted average method, maximum method and average method. Weighted average method: Based on the different sensitivity of the human visual system to different color components, red is usually the least sensitive, green is the highest, and blue is the second. Therefore, the weighted average method can be used to calculate the grayscale value. Maximum method: The maximum value of the three components in the color image is used as the grayscale value of the grayscale image. Average method: The three components in the color image are averaged to obtain the grayscale value.

[0043] Exemplarily, the image to be labeled is processed by using a weighted average method to obtain a grayscale image of the image to be labeled.

[0044] De-noising is a key step in digital image processing, which aims to reduce or eliminate noise in the image to improve the quality and clarity of the image. In digital images, noise may come from many factors, such as sensor noise, transmission errors, environmental interference, etc. Common noise types include Gaussian noise, salt and pepper noise, etc. Common denoising methods include mean filtering algorithm, median filtering algorithm and bilateral filtering algorithm. Mean filtering algorithm: Filter out noise by calculating the mean of pixels in the window area, but it may cause image blur, especially the edges and details. Median filtering algorithm: Based on the theory of sorting statistics, the median of each pixel value in the neighborhood is used to replace the central pixel value. It is particularly effective for salt and pepper noise, and can better preserve the edges and details of the image. Bilateral filtering algorithm: Combining the spatial proximity of the image and the similarity of pixel values, it can not only remove noise but also protect the edges of the image. It is a nonlinear filtering method.

[0045] Exemplarily, a median filtering algorithm is used to process the grayscale image of the image to be labeled to obtain a denoised image.

[0046] Binarization is an image processing technique that simplifies image data and highlights key features by converting the pixel values ​​in an image to only two possible values. In binarization, an image is converted to an image containing only black and white. This is usually achieved by setting a threshold value, setting all pixels in the image with grayscale values ​​greater than the threshold to white (255) and pixels less than or equal to the threshold to black (0). In this way, the image is converted to a binary image, that is, each pixel has only two possible values. Common binarization methods include global thresholding, adaptive thresholding, and local thresholding. Global thresholding: This is one of the simplest binarization methods. It sets a global threshold and applies the same threshold to the entire image for binarization. Adaptive thresholding: This method automatically adjusts the threshold based on the local characteristics of the image. It divides the image into several small blocks, calculates the threshold for each block separately, and performs binarization. Local thresholding: Similar to the adaptive thresholding method, but instead of dividing the image into fixed-size blocks, a sliding window is used to determine the threshold for each pixel.

[0047] Exemplarily, the denoised image is processed using a global threshold method to obtain a preprocessed image.

[0048] Step S12: performing edge detection on the preprocessed image to obtain contours corresponding to multiple objects in the preprocessed image.

[0049] It should be noted that edge detection refers to the process of identifying and extracting the edge of an object from an image. The basic principle of edge detection is based on the grayscale or brightness change between the local area of ​​the image and the surrounding area. When this change reaches a certain level, it is regarded as an edge. In order to accurately detect the edge, it is necessary to calculate the gradient of each pixel in the image (that is, the direction and rate of brightness change). The magnitude of the gradient reflects the intensity of the brightness change, so the edge point usually corresponds to the pixel with a larger gradient. Common edge detection methods include the Sobel operator, the Canny operator, and the Scharr operator. Sobel operator: an edge detection operator of the first-order derivative, which locates the edge by calculating the maximum value of the first-order gradient of the image grayscale. Canny operator: a multi-stage algorithm that can produce relatively ideal edge detection results and has a strong ability to suppress noise. It first uses a Gaussian filter to smooth the image, then calculates the amplitude and direction of the image gradient, then performs non-maximum suppression, and finally uses a double threshold method to detect and connect edges. Scharr operator: similar to the Sobel operator.

[0050] Exemplarily, the Sobel operator is used to perform edge detection on the preprocessed image to obtain the contour corresponding to each object in the preprocessed image.

[0051] Step S13, fitting a circumscribed rectangle to the outline corresponding to each object to obtain the location coordinates of each object.

[0052] It should be noted that fitting the circumscribed rectangle is to fit the outline of the object in the image to be annotated into a rectangle in order to better describe or represent the object. Fitting the circumscribed rectangle has a wide range of applications in many fields, especially in the application of rectangular frame annotation. The process of fitting the circumscribed rectangle includes: using the contour search algorithm to detect the contour in the image, screening the contour of interest according to the area, perimeter or other features of the contour, and using the polygon approximation algorithm to approximate the screened contour to a polygon. If the approximation result is directly a rectangle, it can be used directly. If the approximation result is not a rectangle, further processing is required. For example, the four closest vertices can be selected to form a rectangle, or the vertices can be adjusted to fit a rectangle through other algorithms (such as the least squares method).

[0053] Exemplarily, a circumscribed rectangle is fitted for each object to obtain a circumscribed rectangle corresponding to each object, thereby obtaining four vertex coordinates of each circumscribed rectangle, and the four vertex coordinates are used as the positioning coordinates of the object.

[0054] Step S20: determining whether the multiple targets are partially blocked based on the positioning coordinates of the multiple targets.

[0055] It should be noted that in the image annotation process, targets of the same type are usually annotated. When there are a large number of targets of the same type that are not partially occluded in the image, the sizes of the bounding rectangles corresponding to the targets of the same type are not much different. However, if the target to be annotated is partially occluded, the size of its corresponding bounding rectangle is very different from the size of the average bounding rectangle. Therefore, when there are a large number of targets of the same type to be annotated in the image, an averaging method can be used to determine whether the target to be annotated is partially occluded. By comparing the size of the bounding rectangle of each target to be annotated with the size of the average bounding rectangle of all targets to be annotated, it is determined whether each target to be annotated is partially occluded.

[0056] In a feasible implementation, step S20 may include steps S21 to S23: Step S21, based on the positioning coordinates corresponding to each target, obtain the length and width of the circumscribed rectangle corresponding to each target.

[0057] It should be noted that each target corresponds to a fitted circumscribed rectangle, and each rectangle has four vertices, so the positioning coordinates corresponding to each target are the coordinates of the four vertices of the circumscribed rectangle corresponding to each target. According to the coordinates of the four vertices of the circumscribed rectangle corresponding to each target, the length and width of the circumscribed rectangle can be calculated.

[0058] Step S22, based on the length and width of the circumscribed rectangle corresponding to each target, obtain the average length and average width of the circumscribed rectangles corresponding to all targets.

[0059] It should be noted that the lengths of the bounding rectangles corresponding to all the objects are averaged to obtain the average length of the bounding rectangles corresponding to all the objects. The widths of the bounding rectangles corresponding to all the objects are averaged to obtain the average width of the bounding rectangles corresponding to all the objects.

[0060] Step S23, based on the length and width of the bounding rectangle corresponding to each target and the average length and average width of the bounding rectangles corresponding to all targets, it is determined whether each target is partially blocked.

[0061] It should be noted that when the length (or width) of the circumscribed rectangle corresponding to the target to be labeled is significantly different from its corresponding average value, it is determined that the target to be labeled is partially blocked.

[0062] In a feasible implementation, step S23 may include steps S231 to S234: Step S231, setting a first threshold.

[0063] It should be noted that the first threshold is used to determine whether the target to be marked is partially blocked. The first threshold is set according to specific circumstances and is not specifically limited here.

[0064] Exemplarily, the first threshold is 0.8.

[0065] Step S232: Compare the length of the bounding rectangle corresponding to each target with the average length of the bounding rectangles corresponding to all targets to obtain a first ratio.

[0066] Step S233: Compare the width of the bounding rectangle corresponding to each target with the average width of the bounding rectangles corresponding to all targets to obtain a second ratio.

[0067] Step S234, comparing the first ratio and the first threshold, and the second ratio and the first threshold, and determining that the target is partially blocked when the first ratio is not greater than the first threshold or the second ratio is not greater than the first threshold.

[0068] It should be noted that when the following situations exist, it is judged that the target is partially obscured: the first ratio is less than or equal to the first threshold, and the second ratio is greater than the first threshold; the first ratio is greater than the first threshold, and the second ratio is less than or equal to the first threshold; the first ratio is less than or equal to the first threshold, and the second ratio is less than or equal to the first threshold.

[0069] Step S30, based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target, the positioning coordinate coverage range of the partially obscured target is expanded so that the expanded positioning coordinate coverage range of the partially obscured target includes the obscured area of ​​the partially obscured target.

[0070] It should be noted that the position of the occluded area of ​​the partially occluded target is determined by judging whether the four vertex coordinates of the circumscribed rectangle of the partially occluded target fall within the range covered by the four vertex coordinates of the circumscribed rectangle of the object that occludes the target, and then the range covered by the positioning coordinates of the partially occluded target is expanded according to the position of the occluded area, so that the positioning coordinates of the partially occluded target are closer to its actual positioning coordinates.

[0071] In a feasible implementation, step S30 may include steps S31-S32: Step S31 , determining whether the positioning coordinates of the partially obscured target are included in the positioning coordinate coverage range of the object that obscures the partially obscured target.

[0072] It should be noted that judging whether the positioning coordinates of the partially obscured target are included in the positioning coordinate coverage of the object that obscures the partially obscured target is to determine the location of the obscured area of ​​the partially obscured target, that is, in which direction the positioning coordinates of the partially obscured target should be expanded. The object that obscures the partially obscured target is the obstructing object corresponding to the partially obscured target.

[0073] Exemplarily, the positioning coordinates of the partially occluded target are the coordinates of the four vertices of the circumscribed rectangle of the partially occluded target, and the positioning coordinates of the object that occludes the partially occluded target are the coordinates of the four vertices of the circumscribed rectangle of the occluding object corresponding to the partially occluded target.

[0074] Step S32, updating the positioning coordinates of the partially occluded target within the positioning coordinate coverage range of the object that occludes the partially occluded target to expand the positioning coordinate coverage range of the partially occluded target so that the expanded positioning coordinate coverage range of the partially occluded target includes the occluded area of ​​the partially occluded target.

[0075] It should be noted that after determining the direction in which the positioning coordinates of the partially occluded target should be expanded, the range covered by the positioning coordinates of the partially occluded target is expanded to the range covered by the coordinates of the four vertices of the average circumscribed rectangle corresponding to all targets.

[0076] In a feasible implementation manner, the step of updating the positioning coordinates of the partially obscured target within the coverage range of the positioning coordinates of the object that obscures the partially obscured target includes: Based on the positioning coordinates of the partially occluded target that are not included in the positioning coordinate coverage range of the object that occludes the partially occluded target and the average length and average width of the circumscribed rectangles corresponding to all targets, the positioning coordinates of the partially occluded target that are included in the positioning coordinate coverage range of the object that occludes the partially occluded target are updated.

[0077] It should be noted that in the process of updating the positioning coordinates of the partially occluded target, it is necessary to first determine which coordinates of the positioning coordinates of the partially occluded target need to be retained and which coordinates need to be discarded. Then, taking the retained coordinates as the starting point, and taking the average length and average width of the bounding rectangles corresponding to all targets as the length and width of the bounding rectangle of the partially occluded target, the updated coordinates of the discarded coordinates are obtained.

[0078] The positioning coordinates of the partially occluded target that are included in the coverage range of the positioning coordinates of the object that occludes the partially occluded target indicate that the positioning coordinates of the partially occluded target need to be discarded; the positioning coordinates of the partially occluded target that are not included in the coverage range of the positioning coordinates of the object that occludes the partially occluded target indicate that the positioning coordinates of the partially occluded target need to be retained.

[0079] Step S40 , based on the expanded positioning coordinate coverage of the partially blocked target, draw and insert a rectangular frame corresponding to the partially blocked target.

[0080] It should be noted that after the positioning coordinate coverage range of the partially obscured target is expanded, the expanded coverage range includes the obscured area of ​​the partially obscured target, and may cover the entire obscured area or may only cover a part of the obscured area. However, compared with the original positioning coordinate coverage range of the obscured target, the expanded positioning coordinate coverage range of the partially obscured target in this embodiment is closer to its actual positioning coordinate coverage range, which greatly improves the accuracy of image annotation.

[0081] According to the coverage range of the positioning coordinates after the partial occluded target is enlarged, the transmission projection matrix is ​​obtained, and the matrix is ​​inserted into the image to be annotated. The image with the inserted area as 255 and the uninserted area as 0 is obtained as the mask image. Finally, the mask image and the image to be annotated are summed to obtain the annotated image.

[0082] In a feasible implementation, the rectangular frame marking method further includes: Based on the positioning coordinate coverage of the target that is not partially obscured, a rectangular box corresponding to the target that is not partially obscured is drawn and inserted.

[0083] It should be noted that the target in the image to be annotated may be partially occluded or not partially occluded. For partially occluded targets, it is necessary to expand the coverage range of their positioning coordinates. For targets that are not partially occluded, a rectangular frame is directly drawn and inserted according to the coverage range of its positioning coordinates.

[0084] In the process of image annotation, the present embodiment uses the positioning coordinates of all targets to be annotated to determine the partially occluded target, and according to the positioning coordinates of the partially occluded target and the object that occludes the partially occluded target, the positioning coordinate coverage range of the partially occluded target is expanded, and a rectangular frame of the partially occluded target is drawn and inserted based on the expanded positioning coordinate coverage range of the partially occluded target, so that the gap between the positioning information of the partially occluded target and its actual positioning information is reduced, thereby improving the accuracy of the image annotation data and further improving the training effect of the deep neural network.

[0085] For example, to help understand the implementation process of the rectangular frame marking method obtained in the above embodiment, please refer to Figure 2 , Figure 2 A brief flowchart of the rectangular frame marking method embodiment of the present application is provided, specifically: Step S101 , grayscale processing is performed on the image to be annotated to obtain a grayscale image of the image to be annotated.

[0086] It should be noted that this embodiment adopts the grayscale processing method of the weighted average method.

[0087] Step S102, denoising the grayscale image of the image to be annotated to obtain a denoised image.

[0088] It should be noted that this embodiment adopts a denoising method using a median filtering algorithm.

[0089] Step S103, binarization processing is performed on the denoised image to obtain a pre-processed image.

[0090] It should be noted that this embodiment adopts a binarization processing method using a global threshold method.

[0091] Step S104: performing edge detection on the preprocessed image to obtain contours corresponding to multiple objects in the preprocessed image.

[0092] It should be noted that this embodiment adopts the edge detection method of the Sobel operator.

[0093] Step S105: fitting a circumscribed rectangle to the outline corresponding to each object to obtain the location coordinates of each object.

[0094] It should be noted that, after obtaining the four vertices of the circumscribed rectangle, this embodiment sorts the four vertices to obtain the positioning coordinates of each object.

[0095] Step S106, based on the positioning coordinates corresponding to each target, obtain the length and width of the circumscribed rectangle corresponding to each target.

[0096] It should be noted that this embodiment calculates the coordinate change values ​​of the four vertices of the circumscribed rectangle corresponding to the target in the x-axis direction to obtain the length of the circumscribed rectangle corresponding to the target; calculates the coordinate change values ​​of the four vertices of the circumscribed rectangle corresponding to the target in the y-axis direction to obtain the width of the circumscribed rectangle corresponding to the target.

[0097] For example, the positioning coordinates of target B are (10, 50), (20, 50), (20, 40), and (10, 40). Then the length of the circumscribed rectangle corresponding to target B is 10, and the width is 10.

[0098] Step S107, based on the length and width of the circumscribed rectangle corresponding to each target, obtain the average length and average width of the circumscribed rectangles corresponding to all targets.

[0099] For example, the positioning coordinates of target B are (10, 50), (20, 50), (20, 40), (10, 40), and the positioning coordinates of target A are (40, 60), (60, 60), (60, 40), (40, 40). Then the average length of the circumscribed rectangles corresponding to all targets is 15, and the average width is 15.

[0100] Step S108 , judging whether each target is partially blocked based on the length and width of the bounding rectangle corresponding to each target and the average length and average width of the bounding rectangles corresponding to all targets.

[0101] Exemplarily, the positioning coordinates of target B are (10, 50), (20, 50), (20, 40), (10, 40), and the positioning coordinates of target A are (40, 60), (60, 60), (60, 40), (40, 40). Based on the calculation, it is determined that target B is partially obscured and target A is not partially obscured.

[0102] Step S109 , for the target that is not partially blocked, based on the positioning coordinate coverage range of the target that is not partially blocked, a rectangular frame corresponding to the target is drawn and inserted.

[0103] Step S110 , for the partially obscured target, determining whether the positioning coordinates of the partially obscured target are included in the positioning coordinate coverage range of the object that obscures the partially obscured target.

[0104] Step S111, updating the positioning coordinates of the partially occluded target within the positioning coordinate coverage range of the object that occludes the partially occluded target to expand the positioning coordinate coverage range of the partially occluded target so that the expanded positioning coordinate coverage range of the partially occluded target includes the occluded area of ​​the partially occluded target.

[0105] Step S112: based on the expanded positioning coordinate coverage of the partially blocked target, draw and insert a rectangular frame corresponding to the partially blocked target.

[0106] For example, refer to Figure 3 , Figure 3 A schematic diagram of a partially occluded target provided in an embodiment of the rectangular frame labeling method of the present application, in which the solid line portion is the circumscribed rectangle of the object that occludes the partially occluded target, the dashed line portion is the circumscribed rectangle of the partially occluded target after expansion, and the dash-dot line portion is the circumscribed rectangle of the partially occluded target before expansion. The vertices of the circumscribed rectangle of the partially occluded target before expansion are 1, 2, 3, and 4, and the vertices of the circumscribed rectangle of the partially occluded target after expansion are 1, 5, 6, and 4. Vertices 2 and 3 are discarded coordinates.

[0107] The partially obscured target in the figure is B, and target A is the object that obscures target B. Among the four vertices (vertices 1, 2, 3, and 4) of the circumscribed rectangle of target B, the coordinates of two vertices (vertices 2 and 3) are included in the coverage range of the four vertices of the circumscribed rectangle of target A. In this case, the coordinates of these two vertices (vertices 2 and 3) need to be discarded and updated, and the coordinates of the remaining two vertices (vertices 1 and 4) are retained. Then select any of the retained vertices as the starting point, and extend in the direction of the obscured area of ​​target B with the average length of the circumscribed rectangles corresponding to all targets as the length and the average width as the width, and find the updated vertices (vertices 5 and 6). Finally, the transmission projection matrix is ​​calculated based on the vertices of the circumscribed rectangle that have been determined (vertices 1, 5, 6, and 4), and the matrix is ​​inserted into the image to be labeled. The image with the inserted area as 255 and the non-inserted area as 0 is obtained as a mask to obtain the image to be labeled with the inserted area as 255, and then summed with the image to be labeled to complete the rectangular box labeling of target B.

[0108] In the process of image annotation, the present embodiment uses the positioning coordinates of all targets to be annotated to determine the partially occluded target, and according to the positioning coordinates of the partially occluded target and the object that occludes the partially occluded target, the positioning coordinate coverage range of the partially occluded target is expanded, and a rectangular frame of the partially occluded target is drawn and inserted based on the expanded positioning coordinate coverage range of the partially occluded target, so that the gap between the positioning information of the partially occluded target and its actual positioning information is reduced, thereby improving the accuracy of the image annotation data and further improving the training effect of the deep neural network.

[0109] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the rectangular frame marking method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0110] This application also provides a rectangular frame marking device, please refer to Figure 4 , Figure 4 This is a schematic diagram of the module structure of the rectangular frame marking device of the embodiment of the present application. The rectangular frame marking device includes: The acquisition module 410 is used to acquire the location coordinates of multiple objects in the image to be annotated, wherein the multiple objects include multiple targets.

[0111] The determination module 420 is used to determine whether the multiple targets are partially blocked based on the positioning coordinates of the multiple targets.

[0112] The expansion module 430 is used to expand the positioning coordinate coverage range of the partially occluded target based on the positioning coordinates of the partially occluded target and the positioning coordinates of the object that occludes the partially occluded target, so that the expanded positioning coordinate coverage range of the partially occluded target includes the occluded area of ​​the partially occluded target.

[0113] The drawing module 440 is used to draw and insert a rectangular frame corresponding to the partially blocked target based on the expanded positioning coordinate coverage of the partially blocked target.

[0114] The rectangular frame annotation device provided by the present application adopts the rectangular frame annotation method in the above embodiment, which can solve the technical problem of inaccurate image annotation data in the prior art. Compared with the prior art, the beneficial effects of the rectangular frame annotation device provided by the present application are the same as the beneficial effects of the rectangular frame annotation method provided by the above embodiment, and other technical features in the rectangular frame annotation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0115] The present application provides a rectangular frame marking device, which includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the rectangular frame marking method in the above embodiment.

[0116] Reference below Figure 5 , Figure 5The device structure diagram of the hardware operating environment involved in the rectangular frame marking method in the embodiment of the present application is shown in FIG. The rectangular frame marking device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The rectangular frame marking device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0117] like Figure 5 As shown, the rectangular frame marking device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the rectangular frame marking device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the rectangular frame marking device to communicate with other devices wirelessly or by wire to exchange data. Although the rectangular frame marking device with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0118] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0119] The rectangular frame annotation device provided by the present application adopts the rectangular frame annotation method in the above embodiment, which can solve the technical problem of inaccurate image annotation data in the prior art. Compared with the prior art, the beneficial effects of the rectangular frame annotation device provided by the present application are the same as the beneficial effects of the rectangular frame annotation method provided by the above embodiment, and the other technical features in the rectangular frame annotation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0120] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0121] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0122] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the rectangular frame labeling method in the above-mentioned embodiment.

[0123] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.

[0124] The computer-readable storage medium may be included in the rectangular frame marking device; or may exist independently without being assembled into the rectangular frame marking device.

[0125] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the rectangular frame annotation device, the rectangular frame annotation device: obtains the positioning coordinates of multiple objects in the image to be annotated, wherein the multiple objects include multiple targets; determines whether the multiple targets are partially obscured based on the positioning coordinates of the multiple targets; based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target, expands the positioning coordinate coverage range of the partially obscured target, so that the expanded positioning coordinate coverage range of the partially obscured target includes the obscured area of ​​the partially obscured target; based on the expanded positioning coordinate coverage range of the partially obscured target, draws and inserts a rectangular frame corresponding to the partially obscured target.

[0126] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0128] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0129] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned rectangular frame annotation method, and can solve the technical problem of inaccurate image annotation data in the prior art. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the rectangular frame annotation method provided by the above-mentioned embodiment, and will not be described in detail here.

[0130] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A rectangular frame marking method, characterized in that: The rectangular frame marking method comprises: Acquire positioning coordinates of a plurality of objects in the image to be annotated, wherein the plurality of objects include a plurality of targets; Determining whether the multiple targets are partially blocked based on the positioning coordinates of the multiple targets; Based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target, expanding the positioning coordinate coverage range of the partially obscured target so that the expanded positioning coordinate coverage range of the partially obscured target includes the obscured area of ​​the partially obscured target; Based on the expanded positioning coordinate coverage of the partially obscured target, a rectangular frame corresponding to the partially obscured target is drawn and inserted.

2. The rectangular frame marking method according to claim 1, characterized in that: The step of obtaining the positioning coordinates of multiple objects in the image to be annotated comprises: Preprocessing the image to be labeled to obtain a preprocessed image, wherein the preprocessing includes grayscale processing, denoising processing and binarization processing; Performing edge detection on the preprocessed image to obtain contours corresponding to multiple objects in the preprocessed image; Fit the circumscribed rectangle to the contour corresponding to each object to obtain the location coordinates of each object.

3. The rectangular frame marking method according to claim 1, characterized in that: The step of determining whether the multiple targets are partially blocked based on the positioning coordinates of the multiple targets comprises: Based on the positioning coordinates corresponding to each target, the length and width of the circumscribed rectangle corresponding to each target are obtained; Based on the length and width of the circumscribed rectangle corresponding to each target, an average length and an average width of the circumscribed rectangles corresponding to all targets are obtained; Based on the length and width of the circumscribed rectangle corresponding to each target and the average length and average width of the circumscribed rectangles corresponding to all targets, it is determined whether each target is partially blocked.

4. The rectangular frame marking method according to claim 3, characterized in that: The step of judging whether each target is partially blocked based on the length and width of the bounding rectangle corresponding to each target and the average length and average width of the bounding rectangles corresponding to all targets comprises: Setting a first threshold; Compare the length of the circumscribed rectangle corresponding to each target with the average length of the circumscribed rectangles corresponding to all targets to obtain a first ratio; Compare the width of the bounding rectangle corresponding to each target with the average width of the bounding rectangles corresponding to all targets to obtain a second ratio; Compare the first ratio and the first threshold, and the second ratio and the first threshold, and when the first ratio is not greater than the first threshold or the second ratio is not greater than the first threshold, determine that the target is partially obscured.

5. The rectangular frame marking method according to claim 1, characterized in that: The step of expanding the coverage range of the positioning coordinates of the partially obscured target based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target so that the expanded coverage range of the positioning coordinates of the partially obscured target includes the obscured area of ​​the partially obscured target comprises: Determining whether the positioning coordinates of the partially obscured target are included in the positioning coordinate coverage range of the object that obscures the partially obscured target; The positioning coordinates of the partially obscured target within the positioning coordinate coverage range of the object that obscures the partially obscured target are updated to expand the positioning coordinate coverage range of the partially obscured target so that the expanded positioning coordinate coverage range of the partially obscured target includes the obscured area of ​​the partially obscured target.

6. The rectangular frame marking method according to claim 5, characterized in that: The step of updating the positioning coordinates of the partially obscured target within the coverage range of the positioning coordinates of the object that obscures the partially obscured target comprises: Based on the positioning coordinates of the partially occluded target that are not included in the positioning coordinate coverage range of the object that occludes the partially occluded target and the average length and average width of the circumscribed rectangles corresponding to all targets, the positioning coordinates of the partially occluded target that are included in the positioning coordinate coverage range of the object that occludes the partially occluded target are updated.

7. The rectangular frame marking method according to claim 1, characterized in that: The method further comprises: Based on the coverage of the positioning coordinates of the target that is not partially blocked, a rectangular frame corresponding to the target that is not partially blocked is drawn and inserted.

8. A rectangular frame marking device, characterized in that: The rectangular frame marking device comprises: An acquisition module, used to acquire the location coordinates of multiple objects in the image to be annotated, wherein the multiple objects include multiple targets; A judging module, configured to judge whether the plurality of targets are partially blocked based on the positioning coordinates of the plurality of targets; An expansion module, configured to expand the coverage range of the positioning coordinates of the partially obscured target based on the positioning coordinates of the partially obscured target and the positioning coordinates of the object that obscures the partially obscured target, so that the expanded coverage range of the positioning coordinates of the partially obscured target includes the obscured area of ​​the partially obscured target; A drawing module is used to draw and insert a rectangular frame corresponding to the partially blocked target based on the expanded positioning coordinate coverage of the partially blocked target.

9. A rectangular frame marking device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the rectangular frame labeling method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the rectangular frame marking method according to any one of claims 1 to 7 are implemented.

Images