Apparatus, method and storage medium for generating image with abnormal target

Abstract

The invention relates to an apparatus and method for generating an image with an abnormal target, and a storage medium. The device comprises a generation unit which randomly generates a bounding box of a first mask in a to-be-generated image area of a first image, the bounding box of the first mask does not overlap with an existing detection target in the first image and is intended to add an abnormal target to the bounding box of the first mask; the first acquisition unit is used for acquiring a second image and a corresponding second mask based on a partial region, comprising an external frame, of the first image, and the size of the second image meets the input requirement of a model required by the image adding unit; the image adding unit is used for adding the abnormal target into an external frame in a second mask in a second image; and a second acquisition unit that obtains the first image to which the abnormal target is added as a generated image based on the second image to which the abnormal target is added. The device can generate the image with the abnormal target on the premise of not influencing the quality of the original detection target.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an apparatus, method and machine-readable storage medium for generating an image with an abnormal target. Background Art

[0002] This section provides background information related to the present disclosure which is not necessarily prior art.

[0003] Object detection is an important image task. However, in actual use, the model may encounter abnormal interference targets that have never appeared before, and thus misidentify them. Since these abnormal interference targets do not exist in the original training data, in order to improve the stability of the model, a batch of data with abnormal targets needs to be created. Through this batch of newly created data, the model can learn how to combat these abnormal interferences and enhance robustness. Summary of the invention

[0004] This section provides a general summary of the disclosure, and is not a comprehensive disclosure of its full scope or all of its features.

[0005] The object of the present disclosure is to provide an image processing device, an image processing method and a machine-readable storage medium for automatically adding abnormal samples to an image for target detection.

[0006] According to one aspect of the present disclosure, there is provided a device for generating an image with an abnormal target, comprising: a generating unit, which randomly generates a bounding box of a first mask in an image area to be generated of a first image, wherein the bounding box of the first mask does not overlap with an existing detected target in the first image, and is intended to add the abnormal target to the bounding box of the first mask; a first acquiring unit, which obtains a second image and a corresponding second mask based on a partial area of ​​the first image including the bounding box, and a size of the second image meets the input requirements of a model required by an image adding unit; the image adding unit adds the abnormal target to the bounding box in the second mask in the second image; and a second acquiring unit, which obtains the first image with the abnormal target added as a generated image based on the second image with the abnormal target added.

[0007] According to another aspect of the present disclosure, a method for generating an image with an abnormal target is provided, comprising: randomly generating an outer frame of a first mask in an image area to be generated in a first image, wherein the outer frame of the first mask does not overlap with an existing detection target in the first image, and is intended to add the abnormal target to the outer frame of the first mask; obtaining a second image and a corresponding second mask based on a partial area of ​​the first image including the outer frame, wherein a size of the second image meets the input requirements of a model required by an image adding unit; adding the abnormal target to the outer frame in the second mask in the second image by the image adding unit; and obtaining the first image with the abnormal target added as a generated image based on the second image with the abnormal target added.

[0008] According to another aspect of the present disclosure, a machine-readable storage medium is provided, which carries a program product including machine-readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, can enable the computer to perform the method according to the present disclosure.

[0009] By using the apparatus, method and machine-readable storage medium for generating images with abnormal targets according to the present disclosure, images with abnormal targets can be generated without affecting the quality of the original detected targets. Using these newly generated image data for training can greatly improve the model's ability to handle abnormal situations.

[0010] Further areas of applicability will become apparent from the description provided herein.The description and specific examples in this summary are intended for purposes of illustration only and are not intended to limit the scope of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure. In the drawings:

[0012] Figure 1 is a block diagram illustrating a structure of an image processing apparatus according to an embodiment of the present disclosure;

[0013] Figure 2 is a block diagram illustrating a structure of a generation unit in an image processing apparatus according to an embodiment of the present disclosure;

[0014] Figure 3 is a block diagram illustrating a structure of an image processing apparatus according to another embodiment of the present disclosure;

[0015] Figure 4 A schematic diagram illustrating the principle of an image processing device according to another embodiment of the present disclosure;

[0016] Figure 5is a block diagram illustrating a structure of an image processing apparatus according to yet another embodiment of the present disclosure;

[0017] Figure 6 A schematic diagram illustrating the principle of an image processing device according to another embodiment of the present disclosure;

[0018] Figure 7 A flowchart for illustrating a method of generating an external bounding box according to an embodiment of the present disclosure is provided;

[0019] Figure 8 A flowchart illustrating an image processing method according to an embodiment of the present disclosure; and

[0020] Fig. 9 is a block diagram of an exemplary structure of a general-purpose personal computer in which the image processing apparatus and method according to the embodiments of the present disclosure can be implemented.

[0021] Although the present disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown as examples in the drawings and are described in detail herein. It should be understood, however, that the description of specific embodiments herein is not intended to limit the present disclosure to the specific forms disclosed, but rather, the present disclosure is intended to cover all modifications, equivalents, and alternatives that fall within the spirit and scope of the present disclosure. It should be noted that throughout the several drawings, corresponding reference numerals indicate corresponding parts. DETAILED DESCRIPTION

[0022] Examples of the present disclosure will now be described more fully with reference to the accompanying drawings.The following description is merely exemplary in nature and is not intended to limit the present disclosure, application, or uses.

[0023] Example embodiments are provided so that the disclosure will be exhaustive and its scope will be fully conveyed to those skilled in the art. Numerous specific details such as examples of specific components, devices and methods are set forth to provide a detailed understanding of the embodiments of the present disclosure. It will be apparent to those skilled in the art that the example embodiments may be implemented in many different forms without the use of specific details, none of which should be construed as limiting the scope of the present disclosure. In certain example embodiments, well-known processes, well-known structures and well-known techniques are not described in detail.

[0024] Figure 1 1 is a block diagram illustrating the structure of an image processing apparatus according to an embodiment of the present disclosure. Figure 1 As shown, the image processing apparatus 100 according to an embodiment of the present disclosure may include a generating unit 110, a first acquiring unit 120, an image adding unit 130 and a second acquiring unit 140. The image processing apparatus 100 is used to generate an image with an abnormal target.

[0025] The generating unit 110 can randomly generate a bounding box of the first mask in the image area to be generated of the first image, wherein the bounding box of the first mask does not overlap with the existing detection target in the first image, and is intended to add abnormal targets to the bounding box of the first mask. The bounding box of the first mask is, for example, a rectangular bounding box, but its shape is certainly not limited thereto. Since the abnormal target is to be added to the bounding box, the value inside the bounding box of the first mask is 1, and the value outside the bounding box is 0.

[0026] In addition, the image area to be generated may be the entire first image, or may be an area on the first image segmented according to the target detection task. For example, for a road target detection task, the image area to be generated may be an area segmented from the first image and covered by a road.

[0027] Further, the first acquisition unit 120 can obtain a second image and a corresponding second mask based on a partial area of ​​the first image including a bounding box. In the present application, an image with an abnormal target added is obtained based on a partial area of ​​the first image (including a bounding box). The significance of this is that general image inpainting models (such as Paint-by-example) have requirements for the size of the input image, so before adding the object, the input image needs to be resized. However, this size change sometimes destroys the original image quality and affects the training of subsequent detection models. Therefore, the present application is not based on the entire first image, but on a partial area of ​​the first image including a bounding box, to obtain an image with an abnormal target added through an image inpainting model. Therefore, the size of the second image meets the input requirements of the model required by the image adding unit, that is, the size of the second image matches the image size required by the image inpainting model.

[0028] Furthermore, the image adding unit 130 may add the abnormal target to the bounding box in the second mask in the second image. The image adding unit 130 may be implemented by an image repair model (eg, Paint-by-example).

[0029] Further, the second acquisition unit 140 may obtain the first image with the abnormal object added thereto as the generated image based on the second image with the abnormal object added thereto.

[0030] Thus, the image processing device 100 according to the embodiment of the present disclosure can obtain an image with abnormal targets added, while minimizing the damage to the image quality. Using these newly generated image data for training can greatly improve the model's ability to handle abnormal situations.

[0031] Combine the following Figure 2 The structure of the generating unit 110 is described in detail. Figure 21 is a block diagram illustrating the structure of a generation unit in an image processing apparatus according to an embodiment of the present disclosure. Figure 2 As shown, the generating unit 110 may include a selecting unit 1101 , a size acquiring unit 1102 and a determining unit 1103 .

[0032] In order to add an abnormal target to the first image, you first need to obtain the target mask for adding the abnormal target, that is, the position of the image where the abnormal target is generated. In addition, the image area to be generated is a large range, and many masks can be generated within this range. This is set according to the specific task. For example, if in the task of road target detection, the targets to be generated must be on the road, then the image is first segmented and the road is extracted as the image area to be generated. Of course, this step is optional, and the whole image can be used directly. Therefore, in the case where the image area to be generated is the area on the first image segmented according to the target detection task, the generation unit 110 can also include an area determination unit (not shown in the figure), which can determine part of the target area on the first image as the image area to be generated.

[0033] Specifically, the selection unit 1101 may randomly select a point in the image region to be generated as the center point of the circumscribed frame. If the image region to be generated is not specifically set, the entire image may be used to generate the center point.

[0034] The size acquisition unit 1102 may generate the size of the circumscribed frame based on the relationship between the position and size of the circumscribed frame, the relationship between the size of the abnormal target and the size of the reference object or the reference frame, the size of the first image, and a random number factor related to the size of the circumscribed frame.

[0035] Specifically, after the selection unit 1101 determines the center point of the bounding box of the first mask, the size acquisition unit 1102 first imagines the first mask as a square, and then adjusts it on this basis. The definition of size here is the side length s of the square mask, s = b*w*d*r. Among them, b is the basic size ratio, which is a hyperparameter that can be set, such as 0.1. For different abnormal targets such as bears, deer, flying birds, etc. in road target detection tasks, b can vary according to the ratio of the usual size of different abnormal targets to the size of the reference object or reference frame. Specifically, b reflects the relationship between the size of the abnormal target and the size of the reference object or reference frame. w is the image height, which reflects the size of the first image. r is a random number factor, which takes a value in [0.8,1.2], reflecting the random number factor related to the size of the bounding box. d is the distance ratio obtained based on "near big and far small", reflecting the relationship between the position and size of the bounding box. It can be understood by those skilled in the art that the random value range is not limited to the above range, and the above range is only an example.

[0036] For some flying targets, such as crows, it is also possible not to use the "closer is larger, farther is smaller" method, and d = 1 can be used directly. Alternatively, the value of d can be set considering the position of the flying target. For example, for a flying target in a standing posture, the "closer is larger, farther is smaller" method can be used, and for a flying target in a flying posture, the "closer is larger, farther is smaller" method can be not used.

[0037] The size acquisition unit 1102 establishes the relationship between the position and size of the bounding box by the following method: solving the relationship between the position and size of the bounding box through a linear model based on the size of the bounding box of the reference object in the image and the position of the reference object along the axis in the image that reflects the distance relationship; or manually setting two reference frames that reflect the distance relationship of the detection target, and solving the relationship between the position and size of the bounding box through a linear model based on the position and size of the two reference frames.

[0038] Specifically, the two methods are as follows:

[0039] The solution based on the reference object is suitable for the case where the camera is fixed, such as road video surveillance. There is a large amount of data from the same perspective. Select a common category with a fixed size as a reference object, such as a car. Select two cars with a large longitudinal distance and a complete outer frame, obtain the position (the vertical coordinate of the center of the outer frame) and size (the height of the outer frame), and use the linear model to solve the relationship between the position and size. Based on this, set the distance ratio according to the center vertical coordinate of the mask.

[0040] In the absence of a reference object, manually set two reference frames, draw the position and size and solve them using a linear model. You only need to set it once and use it on all images.

[0041] Further, after the size acquisition unit 1102 obtains the side length of the square mask by the above method, the mask is deformed in aspect ratio: the current aspect ratio of the mask is 1, and under the premise of keeping the center point coordinates and width unchanged, a random value is taken between, for example, [0.8, 1.2], and the height of the mask is multiplied by the random value. The random value is also an example of a random number factor related to the size of the bounding box. It can be understood by those skilled in the art that the random value range is not limited to the above range, and the above range is only an example.

[0042] Thus, the size acquisition unit 1102 can obtain the size of the circumscribed frame, that is, the width and height of the circumscribed frame.

[0043] Further, the determination unit 1103 determines that the bounding box is a valid bounding box only when the bounding box does not overlap with all existing detection targets in the first image and the bounding box does not exceed the area of ​​the image to be generated.

[0044] Specifically, for the current mask, the determination unit 1103 calculates the intersection-over-union (IoU) of the current mask and all existing road targets. If IoU>f (predetermined threshold), that is, the current mask may block the road target, the mask is determined to be invalid. Wherein, f is greater than or equal to 0. When f=0, the mask is not allowed to overlap with any road target. Since the surrounding parts of the mask's bounding box are not necessarily filled with abnormal targets, f can also be set to a number greater than 0, such as 0.1. In this case, the added abnormal targets will not cause disruptive occlusion of the road targets.

[0045] In addition, if the current mask exceeds the image area to be generated, the mask is also determined to be invalid. If the mask is determined to be invalid, the generation process needs to be re-executed.

[0046] By using the generation unit 110 according to the present application, an external frame with diversity (position diversity, size diversity, morphology diversity), rationality (conforming to natural laws), and effectiveness (without blocking the original target and without reducing image quality) can be generated to fill in abnormal targets.

[0047] Thus, the image processing device 110 can generate new targets with diversity (position diversity, size diversity, morphology diversity), rationality (in accordance with natural laws), and effectiveness (not blocking the original target and not reducing image quality) in the original image without affecting the quality of the original detection target. Using these newly generated image data for training can greatly improve the model's ability to handle abnormal situations.

[0048] Figure 3 is a block diagram illustrating a structure of an image processing apparatus according to another embodiment of the present disclosure, and Figure 4 Schematic diagram illustrating the principle of an image processing device according to another embodiment of the present disclosure. Figure 3 As shown, the image processing apparatus 200 may include a generating unit 210, a first acquiring unit 220, an image adding unit 230, and a second acquiring unit 240. The first acquiring unit 220 includes a first expanding unit 2201, a first intercepting unit 2202, and a first transforming unit 2203, and the second acquiring unit 240 includes a second transforming unit 2401, a second intercepting unit 2402, and a first pasting unit 2403. The generating unit 210 and the image adding unit 230 are similar to Figure 1 and Figure 2 The generation unit 110 and the image adding unit 130 are described herein, and therefore their description will be appropriately omitted.

[0049] The generating unit 210 may randomly generate a bounding box of the first mask in the image region to be generated of the first image, wherein the bounding box of the first mask does not overlap with the existing detected targets in the first image and is intended to add abnormal targets to the bounding box of the first mask. Figure 4 As shown, assume that there are image I (first image) and binary image I m (The first mask, with Figure 4 The black rectangular frame in the figure corresponds to the outer frame of the figure), and the size is h*w. m It is a binary image, the mask area is 1 (the area inside the bounding box is 1), and the other areas are 0 (the area outside the bounding box is 0). m The coordinates of the center point P of the mask area in is (x c ,y c ), the width and height of the rectangular frame are w m 、h m Thus, the original image and the first mask can be obtained, and then only the abnormal target needs to be generated within the bounding box of the first mask to obtain a new image. As described below, the generation of the image can directly use an image or text-guided image inpainting model (such as Paint-by-example).

[0050] Furthermore, the first expansion unit 2201 in the first acquisition unit 220 can expand the first image and the first mask respectively, so that the first expansion values ​​are increased in the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0. Figure 4 As shown, for the first image I and the first mask I m Expand them all, add h to the left, right, top and bottom p The expanded edge area is set to 0. The first expansion value h p Greater than the width and height of the outer frame of the first mask (w m 、h m ). For example, the first extended value h p It can be equal to the width and height of the outer frame of the first mask (w m 、h m ), but the present disclosure is not limited thereto. For example, the first extended value h p It can also be equal to the width and height (w m 、h m ) is the larger of

[0051] The first interception unit 2202 in the first acquisition unit 220 can intercept an image whose width and height are three times the first expansion value from the expanded first image and the first mask, respectively, with the center of the outer frame of the first mask as the center, to obtain a intercepted image and a intercepted mask. Figure 4 As shown in FIG. 1 , due to the expansion, the coordinate P of the center point of the outer frame area of ​​the original first mask becomes (xc +h p ,y c +h p ). With point P as the center, the expanded I and I m Cut the width and height to h respectively p *Sub-image I s1 and I ms1 (Not shown). Of course, the width and height of the captured image and the captured mask are not limited to three times the first expansion value, and may be other values, as long as the entire bounding box is included in the captured image and the captured image does not exceed the outer boundary of the entire first image after expansion. For example, it may be 2.8 times the first expansion value.

[0052] The first transformation unit 2203 in the first acquisition unit 220 can transform the size of the intercepted image and the intercepted mask to match the size required by the image repair model, thereby obtaining a second image and a second mask. Figure 4 As shown, I s1 and I ms1 The size is transformed into the size L*L required by the image adding unit (image patching model) to obtain the second image and the second mask I s2 and I ms2 For example, for the Paint-by-example model, L can be 512.

[0053] The image adding unit 230 can use the image repair model to add abnormal targets to the corresponding positions of the second image according to the positions of the bounding box in the second mask to obtain a third image. Figure 4 The dark gray blocks in FIG. 1 are only simplified examples of abnormal targets, and the present disclosure is not limited thereto. For example, in a road target detection task, abnormal targets may be small animals other than cars, trucks, etc. that often appear on the road, such as bears, deer, crows, etc. For example, according to the second mask I ms2 The position of the mask area in the second image I s2 The corresponding position of the image is repaired to obtain the new image I s3 .

[0054] Further, the second transformation unit 2401 in the second acquisition unit 240 can transform the size of the third image so that the width and height of the third image are restored to three times of the first expansion value, thereby obtaining a fourth image (not shown). s3 Resize it so that its width and height return to h p *3, get the fourth image I s4 .

[0055] The second intercepting unit 2402 in the second acquiring unit 240 can intercept a sub-image whose width and height are respectively consistent with the size of the bounding box in the first mask, with the center of the fourth image as the center, and the first pasting unit 2403 in the second acquiring unit 240 can paste the sub-image to the corresponding position of the first image to obtain a generated image. s4 The second interception unit 2402 intercepts a region with a width and a height of w and a m 、h m and paste it to the corresponding position of the original first image I (without expansion).

[0056] For each valid mask generated by the generating unit 210, an image with abnormal objects added can be obtained based on the first image. Thus, a batch of images with abnormal objects added can be obtained, and the newly generated images can be used for training.

[0057] Therefore, the image processing device 200 according to the embodiment of the present disclosure can generate new targets with diversity (position diversity, size diversity, morphology diversity), rationality (in accordance with natural laws), and effectiveness (not blocking the original target and not reducing image quality) in the original image without affecting the quality of the original detection target. Using these newly generated image data for training can greatly improve the model's ability to handle abnormal situations.

[0058] Combine the following Figure 5 and Figure 6 An image processing apparatus 300 according to yet another embodiment of the present disclosure will be described. Figure 5 is a block diagram illustrating a structure of an image processing apparatus according to yet another embodiment of the present disclosure, and Figure 6 It is a schematic diagram illustrating the principle of an image processing device according to yet another embodiment of the present disclosure.

[0059] Figure 5 The image processing device 300 in the embodiment may include a generating unit 310, a first acquiring unit 320, an image adding unit 330, and a second acquiring unit 340. The first acquiring unit 320 includes a second expanding unit 3201 and a third intercepting unit 3202, and the second acquiring unit 340 includes a fourth intercepting unit 3402 and a second pasting unit 3403. The generating unit 310 and the image adding unit 330 perform the same Figure 1 and Figure 2 The generation unit 110 and the image adding unit 130 in FIG. 1 are similar processes, so the description thereof is appropriately omitted.

[0060] The generating unit 310 randomly generates a bounding box of the first mask in the image region to be generated of the first image, wherein the bounding box of the first mask does not overlap with the existing detected objects in the first image and is intended to add abnormal objects to the bounding box of the first mask. Figure 6 As shown, assume that there are image I (first image) and binary image I m (The first mask, with Figure 6 The size of the bounding box is h*w. m It is a binary image, the mask area is 1 (the area inside the bounding box is 1), and the other areas are 0 (the area outside the bounding box is 0).

[0061] The second expansion unit 3201 in the first acquisition unit 320 can expand the first image and the first mask respectively, so that the second expansion value is added to the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0. The second expansion value is equal to half of the size L required by the image adding unit (image repair model), that is, L / 2. For example, for the Paint-by-example model, L can be 512 and the second expansion value can be 256.

[0062] The third clipping unit 3202 in the first acquisition unit 320 takes the center of the circumscribed frame of the first mask as the center, and clips the expanded first image and the first mask respectively to obtain images (size is L*L) whose width and height match the size required by the image repair model, to obtain the second image and the second mask.

[0063] The image adding unit 330 uses the image repair model to add an abnormal target to the corresponding position of the second image according to the position of the bounding box in the second mask to obtain a fifth image. Since the fifth image is square, its center coincides with the center of the bounding box.

[0064] The fourth interception unit 3402 in the second acquisition unit 340 takes the center of the fifth image as the center, intercepts a sub-image whose width and height are respectively consistent with the size of the external frame in the first mask, and the second pasting unit 3403 in the second acquisition unit 340 pastes the sub-image to the corresponding position of the first image I to obtain a generated image.

[0065] Therefore, according to the embodiment of the present disclosure, the image processing device 300 can generate new targets with diversity (position diversity, size diversity, morphology diversity), rationality (in accordance with natural laws), and effectiveness (not blocking the original target and not reducing image quality) in the original image without affecting the quality of the original detection target. Using these newly generated image data for training can greatly improve the model's ability to handle abnormal situations.

[0066] The main difference between the image processing apparatus 200 and the image processing apparatus 300 is whether the image intercepted from the first image needs to be resized. If the configuration of the image processing apparatus 200 is adopted, masks of any size can be processed. If the configuration of the image processing apparatus 300 is adopted, the best resolution can be obtained in the generated area. When the bounding box in the image is very small, the image processing apparatus 200 can obtain a good abnormal object filling effect.

[0067] Combine the following Figure 7 A circumscribed frame generation process performed by a generation unit according to an embodiment of the present disclosure will be described.

[0068] like Figure 7 As shown, the circumscribed frame generation process according to the embodiment of the present disclosure starts at step S110. In step S110, the image area to be generated is determined.

[0069] Next, in step S120 , a center point of a bounding box is randomly selected in the image area to be generated.

[0070] Next, in step S130, the size of the outer frame is generated. For example, the initial size (side length) of the square outer frame may be generated first, and then the size of the outer frame is deformed in terms of aspect ratio to obtain the final size of the outer frame.

[0071] Next, in step S140, it is determined whether the mask corresponding to the external frame overlaps with the existing object in the image or exceeds the image area to be generated. When it is determined that the mask corresponding to the external frame overlaps with the existing object in the image or exceeds the image area to be generated, the external frame is determined to be an invalid external frame, and the method returns to step S120. When it is determined that the mask corresponding to the external frame does not overlap with the existing object in the image and does not exceed the image area to be generated, the method proceeds to step S150 to determine that the external frame is a valid external frame.

[0072] More details on the bounding box generation process are given above. Figure 2 It has been described in detail and will not be repeated here.

[0073] Therefore, according to the external frame generation process disclosed in the present invention, an external frame with diversity (position diversity, size diversity, shape diversity), rationality (conforms to natural laws), and effectiveness (does not block the original target and does not reduce image quality) can be generated to fill in abnormal targets.

[0074] Combine the following Figure 8 An image processing method according to an embodiment of the present disclosure is described below.

[0075] like Figure 8As shown, the image processing method according to an embodiment of the present disclosure starts at step S210. In step S210, a bounding box of a first mask is randomly generated in the image area to be generated in the first image, wherein the bounding box of the first mask does not overlap with the existing detection target in the first image, and is intended to add an abnormal target to the bounding box of the first mask.

[0076] Next, in step S220, a second image and a corresponding second mask are obtained based on a partial area of ​​the first image including the outer frame, and a size of the second image meets the input requirement of the model required by the image adding unit.

[0077] Next, in step S230 , the abnormal object is added to the bounding box in the second mask in the second image by the image adding unit.

[0078] Next, in step S240 , the first image to which the abnormal object is added is obtained as a generated image based on the second image to which the abnormal object is added.

[0079] Therefore, the image processing method according to the embodiment of the present disclosure can generate new targets with diversity (position diversity, size diversity, morphology diversity), rationality (in accordance with natural laws), and effectiveness (not blocking the original target and not reducing image quality) in the original image without affecting the quality of the original detection target. Using these newly generated image data for training can greatly improve the model's ability to handle abnormal situations.

[0080] Various specific implementations of the above steps of the image processing method according to the embodiment of the present disclosure have been described in detail above and will not be repeated here.

[0081] Obviously, each operation process of the image processing method according to the present disclosure can be implemented in the form of a computer executable program stored in various machine-readable storage media.

[0082] Moreover, the purpose of the present disclosure can also be achieved in the following manner: a storage medium storing the above-mentioned executable program code is directly or indirectly provided to a system or device, and a computer or a central processing unit (CPU) in the system or device reads out and executes the above-mentioned program code. In this case, as long as the system or device has the function of executing the program, the embodiment of the present disclosure is not limited to the program, and the program can also be in any form, for example, a target program, a program executed by an interpreter, or a script program provided to an operating system, etc.

[0083] The above-mentioned machine-readable storage media include, but are not limited to: various memories and storage units, semiconductor devices, disk units such as optical, magnetic and magneto-optical disks, and other media suitable for storing information.

[0084] In addition, the technical solution of the present disclosure can also be implemented by connecting a computer to a corresponding website on the Internet, downloading and installing the computer program code according to the present disclosure into the computer, and then executing the program.

[0085] Fig. 9 is a block diagram of an exemplary structure of a general-purpose personal computer in which the image processing apparatus and method according to the embodiments of the present disclosure can be implemented.

[0086] like Fig. 9 As shown, a CPU 1301 performs various processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage device 1308 to a random access memory (RAM) 1303. In the RAM 1303, data required when the CPU 1301 performs various processes and the like is also stored as needed. The CPU 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output interface 1305 is also connected to the bus 1304.

[0087] The following components are connected to the input / output interface 1305: an input device 1306 (including a keyboard, a mouse, etc.), an output device 1307 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage device 1308 (including a hard disk, etc.), and a communication device 1309 (including a network interface card such as a LAN card, a modem, etc.). The communication device 1309 performs communication processing via a network such as the Internet. A drive 1310 may also be connected to the input / output interface 1305 as needed. A removable medium 1311 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 1310 as needed, so that a computer program read therefrom is installed in the storage device 1308 as needed.

[0088] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 1311 .

[0089] It should be understood by those skilled in the art that such storage media is not limited to Fig. 9 The removable medium 1311 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 1311 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 1302, a hard disk included in the storage device 1308, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0090] In the system and method of the present disclosure, it is obvious that each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present disclosure. In addition, the steps of performing the above series of processes can naturally be performed in chronological order according to the order of description, but do not necessarily need to be performed in chronological order. Some steps can be performed in parallel or independently of each other.

[0091] Although the embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings, it should be understood that the embodiments described above are only used to illustrate the present disclosure and do not constitute a limitation of the present disclosure. For those skilled in the art, various modifications and changes can be made to the above embodiments without departing from the essence and scope of the present disclosure. Therefore, the scope of the present disclosure is limited only by the attached claims and their equivalent meanings.

[0092] Regarding the implementation methods including the above embodiments, the following additional notes are also disclosed:

[0093] Note 1. A device for generating an image with an abnormal target, comprising:

[0094] a generating unit, which randomly generates a bounding box of a first mask in the image region to be generated of the first image, wherein the bounding box of the first mask does not overlap with the existing detected objects in the first image, and is intended to add the abnormal object to the bounding box of the first mask;

[0095] A first acquisition unit, which obtains a second image and a corresponding second mask based on a partial area of ​​the first image including the external frame, wherein a size of the second image meets an input requirement of a model required by an image adding unit;

[0096] the image adding unit adding the abnormal target into a bounding box in the second mask in the second image; and

[0097] A second acquisition unit is configured to obtain the first image to which the abnormal object is added as a generated image based on the second image to which the abnormal object is added.

[0098] Note 2. The device according to Note 1, wherein the image area to be generated is the entire first image or an area on the first image segmented according to a target detection task.

[0099] Note 3. The device according to Note 1, wherein the generating unit comprises:

[0100] A selection unit, which randomly selects a point in the image area to be generated as the center point of the circumscribed frame;

[0101] a size acquisition unit that generates the size of the circumscribed frame based on a relationship between the position and size of the circumscribed frame, a relationship between the size of the abnormal target and the size of the reference object or the reference frame, the size of the first image, and a random number factor related to the size of the circumscribed frame; and

[0102] A determination unit determines that the circumscribed frame is a valid circumscribed frame only when the circumscribed frame does not overlap with all existing detection targets in the first image and the circumscribed frame does not exceed the area of ​​the image to be generated.

[0103] Note 4. The device according to Note 3, wherein the size acquisition unit establishes the relationship between the position and size of the external frame by the following method:

[0104] Solving the relationship between the position and size of the circumscribed frame through a linear model based on the size of the circumscribed frame of the reference object in the image and the position of the reference object along the axis reflecting the near-far relationship in the image; or

[0105] Two reference frames reflecting the near and far relationship of the detection target are manually set, and the relationship between the position and size of the circumscribed frame is solved through a linear model based on the positions and sizes of the two reference frames.

[0106] Note 5. The device according to Note 1, wherein:

[0107] The first acquiring unit includes:

[0108] A first expansion unit, which expands the first image and the first mask respectively, so that first expansion values ​​are added to the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0;

[0109] A first interception unit, which intercepts images whose width and height are three times the first expansion value from the expanded first image and the first mask, with the center of the circumscribed frame of the first mask as the center, to obtain a intercepted image and a intercepted mask; and

[0110] a first transformation unit, which performs a size transformation on the intercepted image and the intercepted mask to match the size required by the image repair model, thereby obtaining the second image and the second mask, and

[0111] The image adding unit uses the image repair model to add the abnormal target to the corresponding position of the second image according to the position of the external frame in the second mask to obtain a third image, and

[0112] The second acquiring unit includes:

[0113] a second transformation unit, which performs a size transformation on the third image to restore the width and height of the third image to three times of the first expansion value, thereby obtaining a fourth image; and

[0114] a second cropping unit, which crops a sub-image whose width and height are respectively consistent with the size of the bounding box in the first mask, with the center of the fourth image as the center; and

[0115] A first pasting unit is configured to paste the sub-image onto a corresponding position of the first image to obtain the generated image.

[0116] Note 6. The device according to Note 5, wherein the first expansion value is equal to an average value of a width and a height of a bounding box of the first mask.

[0117] Note 7. The device according to Note 1, wherein:

[0118] The first acquiring unit includes:

[0119] a second expansion unit, which expands the first image and the first mask respectively, so that second expansion values ​​are added to the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0; and

[0120] a third interception unit, which intercepts images whose width and height match the size required by the image repair model from the expanded first image and the first mask, with the center of the outer frame of the first mask as the center, to obtain the second image and the second mask, and

[0121] The image adding unit uses the image repair model to add an abnormal target to the corresponding position of the second image according to the position of the external frame in the second mask to obtain a fifth image, and

[0122] The second acquiring unit includes:

[0123] a fourth cropping unit, which crops a sub-image whose width and height are respectively consistent with the size of the bounding box in the first mask, with the center of the fifth image as the center; and

[0124] A second pasting unit is configured to paste the sub-image onto a corresponding position of the first image to obtain the generated image.

[0125] Note 8. The apparatus according to Note 7, wherein the second expansion value is equal to half of the size required for the image repair model.

[0126] Note 9. A method for generating an image with an abnormal target, comprising:

[0127] Randomly generate a bounding box of a first mask in the image area to be generated of the first image, wherein the bounding box of the first mask does not overlap with the existing detection target in the first image, and is intended to add the abnormal target to the bounding box of the first mask;

[0128] Obtaining a second image and a corresponding second mask based on a partial area of ​​the first image including the external frame, wherein a size of the second image meets an input requirement of a model required by an image adding unit;

[0129] adding the abnormal target into a bounding box in the second mask in the second image by the image adding unit; and

[0130] The first image to which the abnormal object is added is obtained as a generated image based on the second image to which the abnormal object is added.

[0131] Note 10. The method according to Note 9, wherein the image area to be generated is the entire first image or an area on the first image segmented according to a target detection task.

[0132] Supplement 11. The method according to Supplement 9, wherein randomly generating an outer frame of the first mask comprises:

[0133] Randomly select a point in the image area to be generated as the center point of the circumscribed frame;

[0134] generating a size of the circumscribed frame based on a relationship between a position and a size of the circumscribed frame, a relationship between a size of the abnormal target and a size of a reference object or a reference frame, a size of the first image, and a random number factor related to the size of the circumscribed frame; and

[0135] The circumscribed frame is determined to be a valid circumscribed frame only when the circumscribed frame does not overlap with all existing detection targets in the first image and the circumscribed frame does not exceed the area of ​​the image to be generated.

[0136] Note 12. The method according to Note 11, wherein the relationship between the position and size of the external frame is established by the following method:

[0137] Solving the relationship between the position and size of the circumscribed frame through a linear model based on the size of the circumscribed frame of the reference object in the image and the position of the reference object along the axis reflecting the near-far relationship in the image; or

[0138] Two reference frames reflecting the near and far relationship of the detection target are manually set, and the relationship between the position and size of the circumscribed frame is solved through a linear model based on the positions and sizes of the two reference frames.

[0139] Note 13. The method according to Note 9 further includes:

[0140] Expanding the first image and the first mask respectively, so that first expansion values ​​are added to the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0;

[0141] Taking the center of the outer frame of the first mask as the center, respectively intercepting images with a width and a height three times the first expansion value from the expanded first image and the first mask to obtain a intercepted image and a intercepted mask;

[0142] Transforming the size of the intercepted image and the intercepted mask to match the size required by the image repair model, thereby obtaining the second image and the second mask;

[0143] Using the image inpainting model, adding the abnormal target to the corresponding position of the second image according to the position of the bounding box in the second mask, to obtain a third image;

[0144] Resizing the third image so that the width and height of the third image are restored to three times of the first expansion value, thereby obtaining a fourth image; and

[0145] Taking the center of the fourth image as the center, a sub-image whose width and height are respectively consistent with the size of the bounding box in the first mask is cut out, and the sub-image is pasted to the corresponding position of the first image to obtain the generated image.

[0146] Note 14. The method according to Note 13, wherein the first expansion value is equal to an average value of a width and a height of a bounding box of the first mask.

[0147] Note 15. The method according to Note 9 further includes:

[0148] Expanding the first image and the first mask respectively, so that second expansion values ​​are added to the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0;

[0149] Taking the center of the outer frame of the first mask as the center, respectively intercepting images whose width and height match the size required by the image repair model from the expanded first image and the first mask, to obtain the second image and the second mask;

[0150] Using the image inpainting model, adding an abnormal target to a corresponding position of the second image according to a position of the bounding box in the second mask, to obtain a fifth image; and

[0151] Taking the center of the fifth image as the center, a sub-image whose width and height are respectively consistent with the size of the bounding box in the first mask is cut out, and the sub-image is pasted to the corresponding position of the first image to obtain the generated image.

[0152] Note 16. The method according to Note 15, wherein the second expansion value is equal to half of the size required for the image repair model.

[0153] Note 17. The method according to any one of Notes 9-16, wherein the image repair model is Paint-by-example.

[0154] Note 18. A method according to any one of Notes 9-16, wherein the generated image is used for training of target detection tasks.

[0155] Note 19. The method according to Note 9, wherein, in a road target detection task, the image area to be generated is an area covered by the road in the first image.

[0156] Note 20. A machine-readable storage medium carrying a program product including machine-readable instruction codes stored therein, wherein the instruction codes, when read and executed by a computer, enable the computer to execute the method described in Notes 9-19.

Claims

1. A device for generating an image with an abnormal target, comprising: a generating unit, which randomly generates a bounding box of a first mask in the image region to be generated of the first image, wherein the bounding box of the first mask does not overlap with the existing detected objects in the first image, and is intended to add the abnormal object to the bounding box of the first mask; A first acquisition unit, which obtains a second image and a corresponding second mask based on a partial area of ​​the first image including the external frame, wherein a size of the second image meets an input requirement of a model required by an image adding unit; the image adding unit adding the abnormal target into a bounding box in the second mask in the second image; and A second acquisition unit is configured to obtain the first image to which the abnormal object is added as a generated image based on the second image to which the abnormal object is added.

2. The device according to claim 1, wherein: The image region to be generated is the entire first image or a region on the first image segmented according to a target detection task.

3. The device according to claim 1, wherein: The generating unit comprises: A selection unit, which randomly selects a point in the image area to be generated as the center point of the circumscribed frame; a size acquisition unit that generates the size of the circumscribed frame based on a relationship between the position and size of the circumscribed frame, a relationship between the size of the abnormal target and the size of the reference object or the reference frame, the size of the first image, and a random number factor related to the size of the circumscribed frame; and A determination unit determines that the circumscribed frame is a valid circumscribed frame only when the circumscribed frame does not overlap with all existing detection targets in the first image and the circumscribed frame does not exceed the area of ​​the image to be generated.

4. The device according to claim 3, wherein: The size acquisition unit establishes the relationship between the position and size of the external frame by the following method: Solving the relationship between the position and size of the circumscribed frame through a linear model based on the size of the circumscribed frame of the reference object in the image and the position of the reference object along the axis reflecting the near-far relationship in the image; or Two reference frames reflecting the near and far relationship of the detection target are manually set, and the relationship between the position and size of the circumscribed frame is solved through a linear model based on the positions and sizes of the two reference frames.

5. The device according to claim 1, wherein: The first acquiring unit includes: A first expansion unit, which expands the first image and the first mask respectively, so that first expansion values ​​are added to the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0; A first interception unit, which intercepts images whose width and height are three times the first expansion value from the expanded first image and the first mask, with the center of the circumscribed frame of the first mask as the center, to obtain a intercepted image and a intercepted mask; and a first transformation unit, which performs a size transformation on the intercepted image and the intercepted mask to match the size required by the image repair model, thereby obtaining the second image and the second mask, and The image adding unit uses the image repair model to add the abnormal target to the corresponding position of the second image according to the position of the external frame in the second mask to obtain a third image, and The second acquiring unit includes: a second transformation unit, which performs a size transformation on the third image to restore the width and height of the third image to three times of the first expansion value, thereby obtaining a fourth image; and a second cropping unit, which crops a sub-image whose width and height are respectively consistent with the size of the bounding box in the first mask, with the center of the fourth image as the center; and A first pasting unit is configured to paste the sub-image onto a corresponding position of the first image to obtain the generated image.

6. The device according to claim 5, wherein: The first expansion value is equal to an average value of a width and a height of a bounding box of the first mask.

7. The device according to claim 1, wherein: The first acquiring unit includes: A second expansion unit is configured to expand the first image and the first mask respectively, so that second expansion values ​​are added to the left, right, top and bottom of the first image and the first mask respectively, wherein the area of ​​the expanded part is set to 0; and a third interception unit, which intercepts images whose width and height match the size required by the image repair model from the expanded first image and the first mask, with the center of the outer frame of the first mask as the center, to obtain the second image and the second mask, and The image adding unit uses the image repair model to add an abnormal target to the corresponding position of the second image according to the position of the external frame in the second mask to obtain a fifth image, and The second acquiring unit includes: a fourth cropping unit, which crops a sub-image whose width and height are respectively consistent with the size of the bounding box in the first mask, with the center of the fifth image as the center; and A second pasting unit is configured to paste the sub-image onto a corresponding position of the first image to obtain the generated image.

8. The device according to claim 7, wherein: The second expansion value is equal to half of the required size of the image inpainting model.

9. A method for generating an image with an abnormal target, comprising: Randomly generate a bounding box of a first mask in the image area to be generated of the first image, wherein the bounding box of the first mask does not overlap with the existing detection target in the first image, and is intended to add the abnormal target to the bounding box of the first mask; Obtaining a second image and a corresponding second mask based on a partial area of ​​the first image including the external frame, wherein a size of the second image meets an input requirement of a model required by an image adding unit; adding the abnormal target into a bounding box in the second mask in the second image by the image adding unit; and The first image to which the abnormal object is added is obtained as a generated image based on the second image to which the abnormal object is added.

10. A machine-readable storage medium carrying a program product including machine-readable instruction codes stored therein, wherein: When the instruction code is read and executed by a computer, the computer can execute the method according to claim 9.

Images