Image annotation method and device, equipment and medium

Abstract

The embodiment of the invention provides an image annotation method and device, equipment and a medium, which are used for solving the problem that the OD learning effect is influenced due to annotation failure or overlarge deviation caused by the fact that an image enhancement technology in the related technology cannot enable annotation boxes to synchronously and accurately transform. In the embodiment of the invention, the electronic equipment obtains the mask image for distinguishing the target and the background corresponding to the to-be-enhanced image, when affine transformation, perspective transformation and even some elastic deformation are performed on the image, corresponding synchronous transformation is performed on the mask image, and based on the external rectangular frame of each pixel point for identifying the target in the mask image, the target and the background of the to-be-enhanced image are identified. According to the image enhancement method, the corresponding annotation box in the image after enhancement processing is determined, and because the mask image can accurately and effectively distinguish the target from the background, the corresponding annotation box can be accurately determined, so that the determination of the annotation box is realized during image enhancement, and the OD learning effect is improved.

Description

Technical Field

[0001] The present application relates to the fields of target detection, image enhancement, distorted images, security and machine vision technology, and in particular to an image annotation method, device, equipment and medium. Background Art

[0002] In object detection (OD) models, in general scenarios, objects appear in the form of a fixed inductive bias, such as upright people and undistorted vehicle license plates. Therefore, data collection or data set construction uses such good scenes as the main data source. However, in actual usage scenarios, due to camera angles, environmental factors, technical limitations, target morphology, etc., there are often phenomena such as image rotation, image distortion, and multiple poses and angles of the target. Since these scenes, here called singular scenes, are different, it is difficult to fully cover them during the data collection stage, and the learned OD model is also difficult to maintain a high generalization ability for these scenes. Figure 1 A schematic diagram of a good scenario provided in an embodiment of the present application, Figure 2 A schematic diagram of a strange scene provided in an embodiment of the present application, comprising Figure 1 and Figure 2 By comparison, the characters at the edge of the strange scene are tilted nearly 45° and deformed to a certain extent, while this phenomenon does not occur in the good scene. In actual tests, targets in the corners of the strange scene are often not detected or the detection frame fits poorly.

[0003] In order to improve the performance in strange scenes, data simulation solutions are usually used. The essence of data simulation solutions is image enhancement technology in the field of deep learning. The main difference between strange scenes and good scenes is that there is a certain degree of "distortion" in the image, and the target itself does not have any difference. Therefore, as long as the transformation from good scenes to strange scenes is simulated, images that approximate strange scenes can be created, so that learning can be performed based on these approximate images to maintain the generalization ability of the OD model in strange scenes.

[0004] However, current image enhancement technology cannot accurately transform the annotation box synchronously, resulting in invalid annotation or excessive deviation, which in turn affects the OD learning effect. Summary of the invention

[0005] The embodiments of the present application provide an image annotation method, apparatus, device and medium, which are used to solve the problem in the related art that the image enhancement technology cannot accurately transform the annotation box synchronously, resulting in annotation failure or excessive deviation, thereby affecting the OD learning effect.

[0006] In a first aspect, an embodiment of the present application provides an image annotation method, the method comprising: Receiving an image to be enhanced, and obtaining a mask image corresponding to the image for distinguishing a target from a background; Performing image enhancement processing on the image and the mask image based on any preset transformation; Based on the circumscribed upright rectangle method, a circumscribed rectangular frame containing each pixel point identifying the target in the enhanced mask image is determined; and a rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image is determined as a labeling frame of the target.

[0007] Furthermore, the method of determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the Minimum Bounding Upright Rectangle (MBUR) method, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0008] Furthermore, the method of determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the Ellipse-Minimum Bounding UprightRectangle (E-MBUR) method, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0009] Furthermore, the obtaining of a mask image corresponding to the image for distinguishing the target from the background includes: Based on the pre-trained segmentation model, identifying the connected domain of the target contained in the image; The pixel points in the connected domain are adjusted to the pre-saved pixel values ​​identifying the target, and the pixel points outside the connected domain are adjusted to the pre-saved pixel values ​​identifying the background; and a corresponding mask image for distinguishing the target type from the background is obtained.

[0010] Furthermore, each preset transformation includes an affine transformation and a perspective transformation and an elastic deformation that do not affect the connected domain.

[0011] Furthermore, the step of determining the rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image as the corresponding annotation frame includes: For each pixel point corresponding to the circumscribed rectangular frame, a pixel point in the enhanced image having the same position as the pixel point is determined as a target pixel point; The rectangular frame formed by each acquired target pixel point is determined as the corresponding annotation frame.

[0012] Furthermore, the method further comprises: The enhanced image and the corresponding determined annotation box are added to a training set, so as to train the OD model based on the training set.

[0013] In a second aspect, an embodiment of the present application further provides an image annotation device, the device comprising: A receiving and acquiring module, used for receiving an image to be enhanced, and acquiring a mask image corresponding to the image for distinguishing a target from a background; A processing module is used to perform image enhancement processing on the image and the mask image based on any preset transformation; based on the circumscribed upright rectangle method, determine the circumscribed rectangular frame containing each pixel point identifying the target in the mask image after the enhancement processing; and determine the rectangular frame corresponding to the circumscribed rectangular frame in the image after the enhancement processing as the labeling frame of the target.

[0014] Furthermore, the processing module is specifically configured to determine, based on MBUR, a bounding rectangular frame containing each pixel point identifying the target in the enhanced mask image.

[0015] Furthermore, the processing module is specifically configured to determine, based on the E-MBUR, a bounding rectangular frame containing each pixel point identifying the target in the enhanced mask image.

[0016] Furthermore, the receiving and acquiring module is specifically used to identify the connected domain of the target contained in the image based on a pre-trained segmentation model; adjust the pixel points in the connected domain to the pre-saved pixel values ​​identifying the target, and adjust the pixel points outside the connected domain to the pre-saved pixel values ​​identifying the background; and obtain the corresponding mask image that distinguishes the target and the background of the target type.

[0017] Furthermore, the processing module is specifically configured to determine, for each pixel corresponding to the circumscribed rectangular frame, a pixel in the enhanced image having the same position as the pixel as the target pixel; The rectangular frame formed by each acquired target pixel point is determined as the corresponding annotation frame.

[0018] Furthermore, the processing module is also used to add the enhanced image and the corresponding determined annotation box to a training set, so as to train the OD model based on the training set.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device, the electronic device comprising a processor, the processor being configured to implement the steps of any of the above-described image annotation methods when executing a computer program stored in a memory.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described image annotation methods.

[0021] In the embodiment of the present application, the electronic device obtains a mask image corresponding to the image to be enhanced for distinguishing the target and the background, and when performing an affine transformation on the image, performs a corresponding radial transformation on the mask image, and determines the corresponding annotation box in the enhanced image based on the circumscribed rectangular box of each pixel point identifying the target in the mask image. Since the mask image can accurately and effectively distinguish the target and the background, the corresponding annotation box can be accurately determined, and then when enhancing the image, the determination of the annotation box is realized, thereby improving the OD learning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of a good scenario provided in an embodiment of the present application; Figure 2 A schematic diagram of a strange scene provided in an embodiment of the present application; Figure 3 A schematic diagram of an image annotation process provided in an embodiment of the present application; Figure 4 A schematic diagram of an image to be enhanced provided in an embodiment of the present application; Figure 5 A schematic diagram of an msk image provided in an embodiment of the present application; Figure 6 A schematic diagram of an enhanced image provided in an embodiment of the present application; Figure 7 A schematic diagram of a mask image after enhancement processing provided in an embodiment of the present application; Figure 8 A schematic diagram of a process of performing affine transformation on an image and a mask image provided in an embodiment of the present application; Fig. 9A schematic diagram of determining a mask image containing an external rectangular frame after enhancement provided in an embodiment of the present application; Fig.10 A schematic diagram of an enhanced image containing a rectangular frame provided in an embodiment of the present application; Fig.11 A schematic diagram of an image including a marked frame provided in an embodiment of the present application; Fig.12 A schematic diagram of a mask image including a marked box provided in an embodiment of the present application; Fig.13 A schematic diagram of an image after enhancement processing provided in an embodiment of the present application; Fig.14 A schematic diagram of a mask image after enhancement processing provided in an embodiment of the present application; Fig.15 A schematic diagram of an image including a rectangular frame provided in an embodiment of the present application; Fig.16 A schematic diagram of an enhanced image containing a rectangular frame provided in an embodiment of the present application; Fig.17 A schematic diagram of an image containing a rectangular frame determined using a related technology; Fig.18 A schematic diagram of an image containing a rectangular frame determined using a related technology; Fig.19 A detailed process diagram of image annotation provided in an embodiment of the present application; Fig. 20 A structural diagram of an image annotation device provided in an embodiment of the present application; Fig.21 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0026] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0027] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0028] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0029] In order to accurately and effectively determine the annotation box during image enhancement and improve the OD learning effect, the embodiments of the present application provide an image annotation method, device, equipment and medium.

[0030] The image annotation method comprises: an electronic device receives an image to be enhanced, and obtains a mask image corresponding to the image for distinguishing a target from a background; based on any preset transformation, the image and the mask image are subjected to image enhancement processing; based on a circumscribed upright rectangle method, an circumscribed rectangular frame of each pixel point containing a target identified in the enhanced mask image is determined; and a rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image is determined as a labeling frame of the target.

[0031] Embodiment 1:

[0032] Figure 3 A schematic diagram of an image annotation process provided in an embodiment of the present application, the process includes the following steps: S301: receiving an image to be enhanced, and obtaining a mask image corresponding to the image for distinguishing a target from a background.

[0033] The image annotation method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.

[0034] In order to accurately and effectively perform image annotation, the electronic device may first receive the image to be enhanced. In a possible implementation, when a user needs to enhance an image, he or she may select the image to be enhanced on the preset page of the device used by the user and click a preset button, such as an "Enhance" button. At this time, the electronic device may receive the image to be enhanced.

[0035] After receiving the image to be enhanced, the electronic device can obtain a mask image corresponding to the image that distinguishes the target from the background, wherein the target can be a pedestrian, a car, etc., and the mask image has the same size as the image. It should be clear that when the image contains multiple different targets, the electronic device can generate a corresponding mask image for each target. These mask images can accurately distinguish different targets from the background, thereby achieving accurate segmentation and recognition of multiple targets in the image.

[0036] Figure 4 A schematic diagram of an image to be enhanced provided in an embodiment of the present application.

[0037] Figure 4 Center left (left and right described here are Figure 4 The pedestrians (on the left and right) in the figure are the corresponding targets in the embodiments of the present application.

[0038] Figure 5 A schematic diagram of an msk image provided in an embodiment of the present application.

[0039] Figure 5 for Figure 4The image shown corresponds to the mask image that distinguishes the target from the background, which is determined by Figure 5 It can be seen that the outline of the target can be clearly shown in the mask image.

[0040] S302: Perform image enhancement processing on the image and the mask image based on any preset transformation.

[0041] In order to accurately and effectively perform image enhancement, the electronic device may perform image enhancement processing on the image based on any preset transformation, and in order to accurately and effectively annotate the image, the electronic device also performs the same image enhancement processing on the mask image. The transformation may be an affine transformation.

[0042] For example, the electronic device may rotate both the image and the mask image 45 degrees clockwise.

[0043] Figure 6 A schematic diagram of an enhanced image provided in an embodiment of the present application.

[0044] Figure 7 A schematic diagram of a mask image after enhancement processing provided in an embodiment of the present application.

[0045] in, Figure 6 and Figure 7 The corresponding enhancement processing in refers to a 45 degree clockwise rotation.

[0046] Figure 8 A schematic diagram of a process for performing affine transformation on an image and a mask image provided in an embodiment of the present application.

[0047] Depend on Figure 8 It can be seen that the same image enhancement processing can be performed on the image and the mask image based on the affine transformer (Affine Transformer), the same image enhancement processing can be performed on the image and the mask image based on the perspective transformer (Pective Transformer), elastic deformation that does not affect the connected domain can be used, such as water ripples, center scaling, etc., and the same image enhancement processing can be performed on the image and the mask image based on rotation (rotate).

[0048] S303: Based on the circumscribed upright rectangle method, determine the circumscribed rectangular frame containing each pixel point identifying the target in the enhanced mask image; and determine the rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image as the labeling frame of the target.

[0049] In order to process the image accurately and effectively, the electronic device can determine the bounding rectangle frame of each pixel point of the identified target in the enhanced mask image based on the bounding rectangle method, wherein the bounding rectangle frame is a bounding upright rectangle.

[0050] Fig. 9 A schematic diagram of determining a mask image containing an external rectangular frame after enhancement provided in an embodiment of the present application.

[0051] Depend on Fig. 9 It can be seen that the determined circumscribed rectangular frame includes each pixel of the marked target and is a circumscribed upright rectangle, which complies with the principle of labeling by the labeling frame.

[0052] In determining the enhanced mask image, the electronic device may determine the rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image as the annotation frame of the target in the enhanced image.

[0053] Fig.10 A schematic diagram of an enhanced image containing a rectangular frame provided in an embodiment of the present application.

[0054] Depend on Fig.10 It can be seen that the determined rectangular box completely contains the target and is a circumscribed upright rectangle, which conforms to the principle of annotation box annotation.

[0055] Fig.11 A schematic diagram of an image including a marked frame provided in an embodiment of the present application.

[0056] Depend on Fig.11 It can be seen that the annotation box completely contains the target.

[0057] Fig.12 A schematic diagram of a mask image including a marked box provided in an embodiment of the present application.

[0058] Depend on Fig.12 It can be seen that the annotation box completely contains every pixel of the identified target.

[0059] Fig.13 A schematic diagram of an enhanced image provided in an embodiment of the present application.

[0060] Fig.13 The oblique rectangle in the image is obtained by directly rotating the rectangle in the image. Fig.13 The upright rectangular frame in the figure is a labeling frame determined by the method provided in the embodiment of the present application. Since the labeling frame is usually required to be an upright rectangular frame, the method provided in the embodiment of the present application can accurately label the image.

[0061] Fig.14A schematic diagram of a mask image after enhancement processing provided in an embodiment of the present application.

[0062] Fig.14 The oblique rectangle in the figure is obtained by directly rotating the rectangle in the mask image. Fig.14 The upright rectangular frame in the middle is a rectangular frame determined by the method provided in the embodiment of the present application.

[0063] The embodiment of the present application is equivalent to providing a method for image enhancement based on instance segmentation, which can easily expand the rotation image enhancement to other image enhancement techniques, such as affine transformation or perspective transformation. In fact, affine transformations such as rotation, translation, and scaling can all be regarded as a special type of perspective transformation. Therefore, the accuracy of the annotation box depends entirely on the instance segmentation algorithm used, and the loss of box accuracy caused by the transformation itself is almost negligible.

[0064] The embodiment of the present application proposes an image enhancement technology for target detection based on an instance segmentation algorithm. For each target in the image, a mask of the same size as the image is constructed for it. When the image is transformed, the mask is transformed synchronously, and then a circumscribed upright rectangle is taken on the transformed mask. This method does not lose frame accuracy during spatial transformation; it effectively improves the problem of general detection algorithm failure due to image distortion in panoramic or wide-angle cameras and scenes with large depth of field; and applies affine transformation and slight perspective transformation to the OD algorithm without losing frame accuracy.

[0065] The embodiment of the present application realizes low-error rotation, affine transformation, and perspective transformation image enhancement technology; frame accuracy will not be lost during image enhancement and repeated enhancement processes; the entire image enhancement is encapsulated and has a modular nature, which can be used by almost all target detection training algorithms; it is easy to use. It is proposed to use a high-precision instance segmentation algorithm to perform image enhancement on the target detection algorithm; the translation, rotation, and aspect ratio jitter used in the OD training stage are expanded to affine transformation and perspective transformation, and a high-precision frame transformation solution is provided for them; the perspective transformation image enhancement technology is encapsulated and modularized for online use by various training frameworks; rotation, affine transformation, and perspective transformation image enhancement methods are used for target detection training, which effectively improves the problem of failure of the target detection algorithm in wide-angle cameras and distorted scenes.

[0066] It should be noted that in order to improve the performance in strange scenes, there are usually three solutions for improvement. One is to optimize the actual scene through targeted data collection. This solution has serious problems: long cycle and high cost. When data collection cannot be carried out in some confidential scenes, the targeted collection solution is completely unfeasible. The second solution, the data simulation solution, is essentially the image enhancement technology in the field of deep learning. The main difference between strange scenes and good scenes is that due to a certain degree of "distortion" in the image, there is no difference in the target itself (that is, there is no significant difference in the target type, but it is "distorted" with the image). Therefore, as long as the transformation from good scenes to strange scenes is simulated, data approximating strange scenes can be created, so as to learn based on these approximate data and maintain the generalization ability in strange scenes. Another common improvement solution is image distortion correction, that is, through the distortion correction algorithm, the image is "restored" to normal, so that the target detection algorithm does not fail. The correction algorithm often needs to be corrected according to the characteristics of the image and scene itself. Even if the correction is performed, it often only improves the degree of "distortion" and it is impossible to completely eliminate the distortion effect. In a dynamic camera where the lens moves and the focal length changes, the correction algorithm must also maintain dynamics, otherwise the correction will fail. We can understand that the correction method requires professionals to first have a deep understanding of the principles that cause distortion, and then offset the effects through inverse transformation, while the image enhancement method is to transfer the task of stable detection in strange scenes to the learning algorithm. Therefore, a low-cost, short-cycle solution is image enhancement technology. In the embodiment of the present application, image enhancement technology is used to expand the sample to improve the performance in strange scenes. This method can improve the performance in strange scenes at low cost and in a short cycle.

[0067] In the embodiment of the present application, the electronic device obtains a mask image corresponding to the image to be enhanced for distinguishing the target and the background, and when the image is transformed, the mask image is enhanced synchronously, and based on the circumscribed rectangular box of each pixel point identifying the target in the mask image, the corresponding annotation box in the enhanced image is determined. Since the mask image can accurately and effectively distinguish the target and the background, the corresponding annotation box can be accurately determined, and then when the image is enhanced, the determination of the annotation box is realized, thereby improving the OD learning effect.

[0068] Embodiment 2:

[0069] In order to accurately and effectively determine the annotation box, based on the above embodiment, in the embodiment of the present application, the method of determining the circumscribed rectangular box of each pixel point identifying the target in the enhanced mask image based on the circumscribed upright rectangle method includes: Based on the MBUR, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0070] In order to accurately and effectively determine the annotation box, the electronic device can process the enhanced mask image based on MBUR to determine the circumscribed rectangular box of each pixel point of the identified target in the enhanced mask image, which is the minimum circumscribed upright rectangular box.

[0071] The MBUR algorithm traverses the pixel points of the identified target in the mask image, and automatically determines a minimum upright rectangular box by calculating their position and distribution, which can completely contain all the pixel points of the identified target.

[0072] In order to accurately and effectively determine the annotation box, based on the above embodiment, in the embodiment of the present application, the method of determining the circumscribed rectangular box of each pixel point identifying the target in the enhanced mask image based on the circumscribed upright rectangle method includes: Based on the E-MBUR, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0073] In order to accurately and effectively determine the annotation box, the electronic device can process the enhanced mask image based on E-MBUR to determine the circumscribed rectangular box of each pixel point of the identified target in the enhanced mask image, and the circumscribed rectangular box is an ellipse-minimum circumscribed upright rectangular box.

[0074] The E-MBUR algorithm traverses the pixel points of the identified target in the mask image, and automatically determines a minimum ellipse that completely contains each pixel point of the identified target by calculating their position and distribution, and determines a rectangular box containing the ellipse, which can completely contain all the pixel points of the identified target.

[0075] It should be noted that the main difference between MBUR and E-MBUR is that MBUR uses an external rectangle to cover the target frame in order to ensure the integrity of the target, while E-MBUR is based on the probability distribution of the target valid pixels according to the ellipse center, and tries to maintain a high intersection over union (IOU). Obviously, according to the law of large numbers, E-MBUR is more similar to a planar Gaussian distribution, while the rectangular area distribution is obviously far from this smoothness. Therefore, to a certain extent, E-MBUR is an improvement on MBUR, and it does have a certain improvement effect in actual verification.

[0076] Here, we take simple rotation image enhancement as an example to illustrate the defects of the mainstream image enhancement methods currently used in OD algorithms. For rotation image enhancement, the main methods of label box transformation are MBUR and E-MBUR. Generally speaking, the rectangular box determined based on E-MBUR is better.

[0077] Fig.15 A schematic diagram of an image including a rectangular frame provided in an embodiment of the present application.

[0078] Fig.15 The rectangular frame shown in is a rectangular frame determined based on E-MBUR.

[0079] Fig.16 A schematic diagram of an enhanced image containing a rectangular frame provided in an embodiment of the present application.

[0080] Fig.16 For Fig.15 The image shown will be rotated 45 degrees clockwise and enhanced. The rectangular box marked as 4 is Fig.15 The rectangular frame is a rectangular frame after the rotation of the rectangular frame determined based on MBUR, the rectangular frame marked as 3 is the standard rectangular frame corresponding to the re-marking, that is, the expected rectangular frame, and the rectangular frame marked as 1 is a rectangular frame determined by the circumscribed rectangle method in the related art. Fig.16 It can be seen that the rectangular frame completely includes the rectangular frame marked as 3. It should be noted that when the rectangular frame is determined by the method in the related art, the smallest circumscribed upright rectangular frame that completely includes the rectangular frame marked as 4 is usually determined as the corresponding rectangular frame. The rectangular frame marked as 2 is a rectangular frame determined by the method provided in the embodiment of the present application.

[0081] Depend on Fig.16 It can be seen that the rectangular frame marked as 2 is closer to the rectangular frame marked as 4 than the rectangular frame marked as 1, that is, the rectangular frame determined by the method provided in the embodiment of the present application is closer to the standard rectangular frame than the rectangular frame determined by the method of the related technology.

[0082] It can be seen that for such an object that is originally upright and has an aspect ratio significantly greater than 1 or less than 1, E-MBUR does improve the accuracy of the target frame to a certain extent. In the embodiment of the present application, E-MBUR maintains 90% of the IOU, while MBUR only maintains 70% of the IOU.

[0083] If the inverse transformation or superposition transformation is continued, the method in the related art will obtain the rectangular box corresponding to the determined rectangular box after rotation, and the upright rectangle that completely contains the rotated rectangular box will be determined as the rectangular box corresponding to the annotation box. The method in the related art will amplify its error.

[0084] Fig.17 A schematic diagram of an image containing a rectangular frame determined using the relevant technology.

[0085] Fig.17 To enhance the transformed image again, the enhancement method is to rotate it 45° counterclockwise, and the image returns to its original state. Based on MBUR, the enhanced annotation box will be further expanded, and finally Fig.17 The deviation between the rectangular box marked as 1 and the actual rectangular box marked as 2 is further increased. Fig.17 The rectangular frame marked as 3 in the figure is the rectangular frame obtained after the initial enhancement and rotated.

[0086] Similarly, when the method in the related art is used to determine the rectangular frame based on E-MNUR, the inscribed circle of the rotated rectangular frame will be determined, and the upright rectangle that completely contains the inscribed circle will be determined. This method will further amplify the error.

[0087] Fig.18 A schematic diagram of an image containing a rectangular frame determined using the relevant technology.

[0088] Fig.18 To enhance the transformed image again, the enhancement method is to rotate it 45° counterclockwise, and the image returns to its original state. Based on E-MBUR, the enhanced annotation box will be further expanded, and finally Fig.18 The deviation between the rectangular box marked as 1 and the actual rectangular box marked as 2 is further increased. Fig.18 The rectangular frame marked as 3 in the figure is the rectangular frame obtained after the initial enhancement and rotated.

[0089] From the perspective of probability statistics, the E-MBUR and MBUR methods use a rough circumscribed rectangle to continuously expand the frame in order not to lose any valid pixels of the target. This is effective for all frames, but it also continuously introduces more background pixels, so the proportion of valid pixels of the target continues to decrease. The E-MBUR method, based on statistical methods, concludes that the distribution of targets is concentrated in the upright ellipse. Therefore, on the scale of the entire data set, it is concluded that the elliptical distribution can maintain more high IOUs, which to a certain extent suppresses the reduction in the proportion of valid pixels. Both are essentially considered on the entire data set, and in a specific image, affine transformation can be achieved without losing frame accuracy.

[0090] The rotation image enhancement method in the related art has the following disadvantages: the label is obtained by rough estimation, so the accuracy of the target box coordinate box is poor, resulting in inaccurate box after model learning; it is not reversible, that is, after the image is transferred from img_a to img_b through transformation T (synchronously, box_a is transformed to box_b through op), and then img_b is inversely transformed T' to obtain img_a, box_b cannot be inversely transformed back to box_a through op, or it cannot be inversely transformed with box_a. The errors are superimposed; it has error superposition. Taking the rotation image enhancement as an example, the image is rotated 90° four times, and the final image returns to the original state, while the label errors are constantly superimposed and introduce too much background; it has strong prior assumptions and is not applicable to all categories. It is mainly applicable to targets with effective pixels close to elliptical distribution or Gaussian distribution. For stretched human bodies or targets with key features concentrated at the edge, key features will be continuously lost; it is not universal, and only makes strong improvements on rotation data. It cannot be expanded in actual scenarios such as general affine transformations and perspective transformations that do not change the connected domain.

[0091] However, in the enhancement stage, the method provided in the embodiment of the present application does not use a vertical rectangular frame or an ellipse to approximate the contour of the target, does not cause the root cause of label deviation, and the mask image has an accurate target contour. Therefore, the labeling box can be determined accurately and effectively, and the model can be trained accurately and effectively.

[0092] Embodiment 3:

[0093] In order to accurately and effectively perform image annotation, based on the above embodiments, in the embodiment of the present application, the step of obtaining a mask image for distinguishing a target from a background corresponding to the image includes: Based on the pre-trained segmentation model, identifying the connected domain of the target contained in the image; The pixel points in the connected domain are adjusted to the pre-saved pixel values ​​identifying the target, and the pixel points outside the connected domain are adjusted to the pre-saved pixel values ​​identifying the background; and a corresponding mask image for distinguishing the target type from the background is obtained.

[0094] In order to accurately and effectively perform image annotation, the electronic device locally stores a pre-trained segmentation model. The electronic device inputs the image into the segmentation model to obtain the output of the recognition model. The output of the segmentation model is an image of a connected domain that identifies the target contained in the image. The electronic device can determine the connected domain that contains the target in the image based on the output, and adjust the pixel points in the connected domain to the pre-saved pixel values ​​that identify the target, and adjust the pixel points outside the connected domain to the pre-saved pixel values ​​that identify the background; thus, the corresponding mask image of the target and background that distinguishes the target type can be obtained.

[0095] In order to accurately and effectively perform image annotation, on the basis of the above embodiments, in the embodiments of the present application, each transformation includes an affine transformation and a perspective transformation and an elastic deformation that do not affect the connected domain; wherein the affine transformation includes rotation, scaling, cropping, and aspect ratio jitter.

[0096] In order to accurately and effectively perform image annotation, the transformations used in the embodiments of the present application need not affect the connected domain. The specific transformations used include affine transformations, perspective transformations that do not affect the connected domain, and elastic deformations that do not affect the connected domain. Among them, affine transformations include: translation, scaling, aspect ratio jitter, cropping, slight rotation, large-angle rotation, etc., which do not affect the elastic deformation of the connected domain, such as water ripples, center scaling, etc.

[0097] Embodiment 4:

[0098] In order to accurately and effectively perform image annotation, based on the above embodiments, in the embodiment of the present application, the rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image is determined as the corresponding annotation frame, including: For each pixel point corresponding to the circumscribed rectangular frame, a pixel point in the enhanced image having the same position as the pixel point is determined as a target pixel point; The rectangular frame formed by each acquired target pixel point is determined as the corresponding annotation frame.

[0099] In order to determine the annotation box, the electronic device can determine, for each pixel corresponding to the circumscribed rectangular box, the pixel in the enhanced image that has the same position as the pixel as the target pixel. After determining each target pixel, the electronic device can determine the rectangular box formed by each target pixel as the corresponding annotation box.

[0100] In order to accurately and effectively perform image annotation, based on the above embodiments, in the embodiment of the present application, the method further includes: The enhanced image and the corresponding determined annotation box are added to a training set, so as to train the OD model based on the training set.

[0101] The electronic device can add the enhanced image and the corresponding determined annotation box to the training set, so as to train the OD model based on the training set.

[0102] The embodiment of the present application is referred to as the instance segmentation-minimum bounding upright rectangle (IS-MBUR) method. Segment Anything Model (SAM) refers to various open source large models for target segmentation, which generally support the use of rectangular boxes, texts, and masks as prompts. It has very good context understanding and migration generalization capabilities, and has good segmentation capabilities for most target types with clear contours. Ground Truth (GT) means that the dataset also has mask images for instance segmentation, but such data annotation is relatively rare due to the expensive annotation cost. Custom image segmentation models (CustomIS model) are generally segmentation models for various industries or specific categories, and can also be used to determine mask images. In the above-mentioned 45° rotation transformation and inverse transformation, the IOU of the IS-MBUR method relative to the annotation box is 98.8% and 98.6%. The first loss of accuracy mainly comes from the error of the IS algorithm in boundary processing in the first transformation, and the secondary transformation error mainly comes from the rounding error caused by bilinear interpolation in the mask synchronization transformation. The overall error can be ignored. Compared with MBUR and E-MBUR, IS-MBUR was significantly improved.

[0103] Fig.19 A schematic diagram of a detailed process of image annotation provided in an embodiment of the present application.

[0104] S1901: Receive an image to be enhanced.

[0105] S1902: Obtain a mask image for distinguishing a target from a background corresponding to the image to be enhanced.

[0106] Among them, the mask image can be obtained based on SAM, GT or Custom IS model.

[0107] S1903: Perform image enhancement processing on the image and the mask image based on any preset transformation.

[0108] S1904: Based on the circumscribed upright rectangle method, determine the circumscribed rectangular frame of each pixel point of the identified target in the enhanced mask image.

[0109] S1905: Determine a rectangular box corresponding to the circumscribed rectangular box in the enhanced image as a labeling box of the target.

[0110] For SAM or large industry segmentation models, the detection can be directly provided to SAM as a prompt, and then the instance segmentation result can be directly obtained. YOLOv10n and the public COCO can be used to verify pedestrian targets in distorted scenes to confirm the effectiveness of the method provided in the embodiment of the present application. For customized instance segmentation models, generally only a few categories can be supported. IS-MBUR can be used for supported categories, and the E-MBUR method can be used for the remaining categories.

[0111] Table 1 is a schematic diagram of an enhancement effect verification provided by an embodiment of the present application:

[0112] The first column of Table 1 is the adopted method, the second column of Table 1 is the training data set, the third column of Table 1 is the mean average precision (mAP) determined based on the good-test set, the third column of Table 1 is the mAP determined based on the distorted-test set, the second line of Table 1 is the relevant information when no image enhancement is performed, the third line of Table 1 is the relevant information when the annotation box is determined based on MBUR using the method in the related art, the fourth line of Table 1 is the relevant information when the annotation box is determined based on E-MBUR using the method in the related art, and the fifth line of Table 1 is the relevant information when the annotation box is determined based on IS-MBUR provided in the embodiments of the present application.

[0113] From this verification, it can be known that the method in the related art slightly improves the detection effect in the distorted scene, but seriously damages the performance in the original good scene. The method used in the embodiment of the present application greatly improves the test effect in the distorted scene while having almost no negative impact on the effect of the original scene.

[0114] Embodiment 5: Based on the same inventive concept, the present application embodiment provides an image annotation device. Fig. 20 For a schematic diagram of the structure of an image annotation device provided in an embodiment of the present application, please refer to Fig. 20 , the device comprises: A receiving and obtaining module 2001 is used to receive an image to be enhanced and obtain a mask image corresponding to the image for distinguishing a target from a background; The processing module 2002 is used to perform image enhancement processing on the image and the mask image based on any preset transformation; based on the circumscribed upright rectangle method, determine the circumscribed rectangular frame containing each pixel point identifying the target in the mask image after the enhancement processing; and determine the rectangular frame corresponding to the circumscribed rectangular frame in the image after the enhancement processing as the labeling frame of the target.

[0115] In a possible implementation manner, the processing module 2002 is specifically configured to determine, based on the MBUR, a bounding rectangular frame that includes each pixel point identifying the target in the enhanced mask image.

[0116] In a possible implementation manner, the processing module 2002 is specifically configured to determine, based on the E-MBUR, a bounding rectangular frame that includes each pixel point identifying the target in the enhanced mask image.

[0117] In one possible implementation, the receiving and acquiring module 2001 is specifically used to identify the connected domain of the target contained in the image based on a pre-trained segmentation model; adjust the pixel points in the connected domain to the pre-saved pixel values ​​identifying the target, and adjust the pixel points outside the connected domain to the pre-saved pixel values ​​identifying the background; and obtain a corresponding mask image that distinguishes the target and the background of the target type.

[0118] In a possible implementation, the processing module 2002 is specifically configured to determine, for each pixel corresponding to the circumscribed rectangular frame, a pixel in the enhanced image having the same position as the pixel as the target pixel; The rectangular frame formed by each acquired target pixel point is determined as the corresponding annotation frame.

[0119] In a possible implementation, the processing module 2002 is further configured to add the enhanced image and the corresponding determined annotation box to a training set, so as to train the OD model based on the training set.

[0120] Embodiment 6: Based on the same inventive concept, an embodiment of the present application provides an electronic device, which can implement the steps of the image annotation method discussed above. Fig.21 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Fig.21 As shown, it includes: a processor 2101, a communication interface 2102, a memory 2103 and a communication bus 2104, wherein the processor 2101, the communication interface 2102, and the memory 2103 communicate with each other through the communication bus 2104; The memory 2103 stores a computer program. When the program is executed by the processor 2101, the processor 2101 performs the following steps: Receiving an image to be enhanced, and obtaining a mask image corresponding to the image for distinguishing a target from a background; Performing image enhancement processing on the image and the mask image based on any preset transformation; Based on the circumscribed upright rectangle method, a circumscribed rectangular frame containing each pixel point identifying the target in the enhanced mask image is determined; and a rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image is determined as a labeling frame of the target.

[0121] In a possible implementation, determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the MBUR, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0122] In a possible implementation, determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the E-MBUR, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0123] In a possible implementation, the acquiring a mask image corresponding to the image for distinguishing a target from a background includes: Based on the pre-trained segmentation model, identifying the connected domain of the target contained in the image; The pixel points in the connected domain are adjusted to the pre-saved pixel values ​​identifying the target, and the pixel points outside the connected domain are adjusted to the pre-saved pixel values ​​identifying the background; and a corresponding mask image for distinguishing the target type from the background is obtained.

[0124] In a possible implementation, each preset transformation includes an affine transformation and a perspective transformation and an elastic deformation that do not affect the connected domain.

[0125] In a possible implementation, determining the rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image as the corresponding annotation frame includes: For each pixel point corresponding to the circumscribed rectangular frame, a pixel point in the enhanced image having the same position as the pixel point is determined as a target pixel point; The rectangular frame formed by each acquired target pixel point is determined as the corresponding annotation frame.

[0126] In a possible implementation, the method further includes: The enhanced image and the corresponding determined annotation box are added to a training set, so as to train the OD model based on the training set.

[0127] Since the principle of solving the problem by the above electronic device is similar to that of the image annotation method, the implementation of the above electronic device can refer to the embodiment of the method, and the repeated parts will not be repeated.

[0128] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0129] The communication interface 2102 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the above processor.

[0130] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processing processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0131] Embodiment 7: Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing: Receiving an image to be enhanced, and obtaining a mask image corresponding to the image for distinguishing a target from a background; Performing image enhancement processing on the image and the mask image based on any preset transformation; Based on the circumscribed upright rectangle method, a circumscribed rectangular frame containing each pixel point identifying the target in the enhanced mask image is determined; and a rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image is determined as a labeling frame of the target.

[0132] In a possible implementation, determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the MBUR, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0133] In a possible implementation, determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the E-MBUR, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

[0134] In a possible implementation, the acquiring a mask image corresponding to the image for distinguishing a target from a background includes: Based on the pre-trained segmentation model, identifying the connected domain of the target contained in the image; The pixel points in the connected domain are adjusted to the pre-saved pixel values ​​identifying the target, and the pixel points outside the connected domain are adjusted to the pre-saved pixel values ​​identifying the background; and a corresponding mask image for distinguishing the target type from the background is obtained.

[0135] In a possible implementation, each preset transformation includes an affine transformation and a perspective transformation and an elastic deformation that do not affect the connected domain.

[0136] In a possible implementation, determining the rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image as the corresponding annotation frame includes: For each pixel point corresponding to the circumscribed rectangular frame, a pixel point in the enhanced image having the same position as the pixel point is determined as a target pixel point; The rectangular frame formed by each acquired target pixel point is determined as the corresponding annotation frame.

[0137] In a possible implementation, the method further includes: The enhanced image and the corresponding determined annotation box are added to a training set, so as to train the OD model based on the training set.

[0138] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0139] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0140] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0142] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image annotation method, characterized in that: The method comprises: Receiving an image to be enhanced, and obtaining a mask image corresponding to the image for distinguishing a target from a background; Performing image enhancement processing on the image and the mask image based on any preset transformation; Based on the circumscribed upright rectangle method, a circumscribed rectangular frame containing each pixel point identifying the target in the enhanced mask image is determined; and a rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image is determined as a labeling frame of the target.

2. The method according to claim 1, characterized in that The method of determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the minimum bounding upright rectangle method MBUR, a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

3. The method according to claim 1, characterized in that The method of determining the bounding rectangular frame of each pixel point identifying the target in the enhanced mask image based on the bounding upright rectangle method includes: Based on the Ellipse-Minimum Circumscribed Right Rectangle Method (E-MBUR), a bounding rectangular frame including each pixel point identifying the target in the enhanced mask image is determined.

4. The method according to claim 1, characterized in that: The step of obtaining a mask image corresponding to the image for distinguishing a target from a background includes: Based on a pre-trained segmentation model, identifying a connected domain of an object contained in the image; The pixel points in the connected domain are adjusted to the pre-saved pixel values ​​identifying the target, and the pixel points outside the connected domain are adjusted to the pre-saved pixel values ​​identifying the background; and a corresponding mask image for distinguishing the target type from the background is obtained.

5. The method according to claim 1, characterized in that Each preset transformation includes an affine transformation as well as a perspective transformation and an elastic deformation that do not affect the connected domain.

6. The method according to claim 1, characterized in that The step of determining a rectangular frame corresponding to the circumscribed rectangular frame in the enhanced image as a corresponding annotation frame includes: For each pixel point corresponding to the circumscribed rectangular frame, a pixel point in the enhanced image having the same position as the pixel point is determined as a target pixel point; The rectangular frame formed by each acquired target pixel point is determined as the corresponding annotation frame.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The enhanced image and the corresponding determined annotation box are added to a training set to train the object detection OD model based on the training set.

8. An image annotation device, characterized in that: The device comprises: A receiving and acquiring module, used for receiving an image to be enhanced, and acquiring a mask image corresponding to the image for distinguishing a target from a background; A processing module is used to perform image enhancement processing on the image and the mask image based on any preset transformation; based on the circumscribed upright rectangle method, determine the circumscribed rectangular frame containing each pixel point identifying the target in the mask image after the enhancement processing; and determine the rectangular frame corresponding to the circumscribed rectangular frame in the image after the enhancement processing as the labeling frame of the target.

9. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the image annotation method according to any one of claims 1 to 7 when executing a computer program stored in a memory.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the image annotation method according to any one of claims 1 to 7.

Images