Data proliferation method and arithmetic device thereof

Abstract

The invention discloses a data hyperplasia method and an arithmetic device thereof. The data hyperplasia method comprises the steps of obtaining an input image and generating a plurality of output images corresponding to the input image. At least one first pixel of the input image is respectively displaced to form an output image, and the displacement is random.

Description

TECHNICAL FIELD

[0001] The present application relates to a data augmentation method and an operation device thereof, and in particular to an operation device and a data augmentation method thereof capable of generating a large number of images with sufficient diversity corresponding to an image. BACKGROUND

[0002] Computer vision techniques (e.g., object or boundary recognition, image reconstruction or enhancement, etc.) allow electronic devices to obtain information from images or videos, and can be applied in various fields (e.g., medical image processing, advanced driving assistance systems, automated inspection, etc.). For example, today's industrial production lines can use technologies such as automatic optical inspection or deep learning to automatically check whether products on the production line have any defects, thereby reducing the need for manual inspection. However, in order to establish an automated inspection mechanism, a sufficient number of normal images and defect images need to be collected in advance to provide a standard for the automated inspection machine to recognize defects, but it is difficult to collect a large number of images or images with diversity in the early stages of new products or new manufacturing processes.

[0003] In terms of image generation, deep learning techniques (e.g., generative adversarial networks (GAN), stable diffusion models, etc.) still need to collect a large number of images in advance to train the model, so that the trained model can be used to generate images. Moreover, deep learning techniques are like a black box, and users have difficulty understanding how deep learning techniques generate images, making it difficult to judge their reasonableness. Therefore, how to generate a large number of images with diversity is an important challenge for existing computer vision techniques. SUMMARY

[0004] Therefore, the present application mainly provides a data augmentation method and an operation device thereof to improve the prior art.

[0005] An embodiment of the present application discloses a data augmentation method, comprising obtaining an input image; and generating a plurality of output images corresponding to the input image, wherein at least one first pixel of the input image is respectively displaced to form one of the plurality of output images, and the displacement amount of any of the at least one first pixel has randomness.

[0006] An embodiment of the present application discloses an operation device, comprising a storage circuit configured to store an instruction, the instruction comprising obtaining an input image; and generating a plurality of output images corresponding to the input image, wherein at least one first pixel of the input image is respectively displaced to form one of the plurality of output images, and the displacement amount of any of the at least one first pixel has randomness; and a processing circuit coupled to the storage circuit and configured to execute the instruction. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1Fig. 1 is a schematic diagram of an operation device according to an embodiment of the present application;

[0008] Figure 2 Fig. 2 is a schematic diagram of an operation device according to another embodiment of the present application; Figure 3 Figs. 3 and 4 are schematic diagrams of an operation device according to another embodiment of the present application;

[0009] Figure 4 Fig. 5 is a schematic diagram of an operation device according to another embodiment of the present application;

[0010] Figure 5 Fig. 6 is a schematic diagram of an array according to an embodiment of the present application;

[0011] Figure 6 Fig. 7 is a schematic diagram of different equivalent displacement amplitudes according to an embodiment of the present application;

[0012] Figure 7 Fig. 8 is a schematic diagram of different equivalent deformation amplitudes according to an embodiment of the present application;

[0013] Figure 8 Fig. 9 is a schematic diagram of different equivalent smoothness according to an embodiment of the present application;

[0014] Figure 9 Fig. 10 is a schematic diagram of an operation device according to another embodiment of the present application; Figure 10 Figs. 11 and 12 are schematic diagrams of an operation device according to another embodiment of the present application.

[0015] Symbol explanation

[0016] 10: operation device

[0017] 110: storage circuit

[0018] 120: processing circuit

[0019] 10IN, 20IN, 90IN, 11IN: input image

[0020] 10UT1~10UTn, 20UT1, 30UT1~30UT8, 40UT1~40UT8, 90UT1~90UT8, 11UT1~11UT8: output image

[0021] 210, 210a, 210b, 220, 220a, 610a0~610a50, 710a1~710a50, 810a1~810a10: image

[0022] 230, 240: background region

[0023] 50g1, 50g2: filter

[0024] (2,1),(1,W),(H,W),(211,212),(1W1,1W2),(HW1,HW2),(X1,Y1),(X2,Y2): pixels

[0025]

[0026] element

[0027] S504: Steps

[0028] SR: Region of Interest

[0029] T',T”,g(T'): Array Detailed Implementation

[0030] Figure 1 This is a schematic diagram of the computing device 10 according to Embodiment 1 of the present invention. Figure 1 As shown in (a), the computing device 10 (e.g., a chip, computer, or host) may include a storage circuit 110 and a processing circuit 120, and may be installed in an industrial production line, a drone, or a sensor. The computing device 10 can generate multiple output images 10UT1 to 10UTn (e.g., multiple defective images) corresponding to the input image 10IN by random deformation, provided that only one input image 10IN (e.g., a normal image) is received. The output images 10UT1 to 10UTn can be a large number of images with sufficient diversity, which helps to improve the performance of the deep learning model. In other words, the pixel value distributions of the output images 10UT1 to 10UTn are different from each other and different from the pixel value distribution of the input image 10IN.

[0031] In one embodiment, such as Figure 1As shown in (b), all or partial pixels of an input image (e.g., 10IN) can be respectively displaced to form all or partial pixels of an output image (e.g., 10UT1). The displacement amount of a pixel can be random. For example, the pixel value of pixel (2, 1) of image 10IN is unchanged but displaced by a first displacement amount to form pixels (211, 212) of image 10UT1, and the pixel value of pixel (H, W) of image 10IN is unchanged but displaced by a second displacement amount to form pixels (HW1, HW2) of image 10UT1. Similarly, the pixel value of pixel (2, 1) of image 10IN is unchanged but displaced by a third displacement amount to form a pixel of image 10UT2, and the pixel value of pixel (H, W) of image 10IN is unchanged but displaced by a fourth displacement amount to form another pixel of image 10UT2. The first displacement amount to the fourth displacement amount can be different, or can be independent of each other without correlation. Alternatively, the first displacement amount (relative to at least one of the second displacement amount to the fourth displacement amount) is a random value, or cannot be expressed as a function of at least one of the second displacement amount to the fourth displacement amount. In other words, all or partial pixels of an input image are randomly deformed to be converted into a large number of output images with sufficient diversity.

[0032] The size of an output image is the same as that of its corresponding input image. In an embodiment, input image 10IN and output images 10UT1-10UTn can be two-dimensional images (e.g., grayscale images or color images) with a height of H pixels and a width of W pixels. Alternatively, input image 10IN and output images 10UT1-10UTn can be three-dimensional images (e.g., 3D point cloud images or 3D tomographic images) with a height of H pixels, a width of W pixels, and a depth of D pixels.

[0033] In an embodiment, a data augmentation method can be compiled into a program code and used in computing device 10, and can include at least the following steps:

[0034] Step S102: input an input image (e.g., 10IN) to computing device 10.

[0035] Step S104: computing device 10 determines whether the input image is divided into a plurality of first pixels and a plurality of second pixels. If (computing device 10 or a user) marks a region of interest (ROI) (e.g., 10ROI) in the input image where random deformation is desired (e.g., 10ROI), and encloses the first pixels (e.g., (2, 1) or (H, W)) in the ROI and divides the second pixels (e.g., (1, W)) outside the ROI, step S106 is performed; if (computing device 10 or a user) desires to perform random deformation on the entire input image (e.g., 10ROI), step S106 is performed. Figure 2 Figure 4 ​If the input image is locally warped, the operation device 10 generates a first region image (e.g., 210a or 210b of FIG. 2) after image processing, and performs step S112; if the input image is globally warped, the operation device 10 generates an output image (e.g., 10UT1 or 40UT1 of FIG. 4) corresponding to the input image after image processing, and performs step S114 or S116.

[0036] Step S106: The operation device 10 performs image processing on the second pixels to form a second region image (e.g., 220a of FIG. 2). Figure 2 For example, the operation device 10 repairs the input image around the region of interest or the surrounding of the region of interest with respect to the second pixels to optimize the final image generation effect. Then, step S108 is performed.

[0037] Step S108: The operation device 10 calculates a warping array for the first pixels. Then, step S110 is performed.

[0038] Step S110: The operation device 10 performs image processing on the first pixels. For example, the operation device 10 applies the warping array to the first pixels. In the case of locally warping the input image, the operation device 10 generates a first region image (e.g., 210a or 210b of FIG. 2) after image processing, and performs step S112; in the case of globally warping the input image, the operation device 10 generates an output image (e.g., 10UT1 or 40UT1 of FIG. 4) corresponding to the input image after image processing, and performs step S114 or S116. Figure 2 Figure 1 Figure 4

[0039] Step S112: The operation device 10 performs image synthesis according to the first region image (or the first pixels) and the second region image (or the second pixels) to generate an output image (e.g., 10UT1) corresponding to the input image. For example, the operation device 10 pastes the generated first region image back to the second region image or covers the second region image at the corresponding position. Then, step S114 or S116 is performed.

[0040] Step S114: The operation device 10 determines whether to perform step S104 or S108 again to generate other output images (e.g., 10UT2) corresponding to the input image. Then, step S116 is performed.

[0041] Step S116: The operation device 10 trains, validates, or tests a deep learning model using the input image or the output image.

[0042] ​​​One or more of steps S102 to S116 can be deleted or their order changed depending on different needs. In one embodiment, only steps S104, S110, and S112 can be performed to perform random deformation of a local input image; in another embodiment, only step S110 can be performed to perform random deformation of the entire input image. In one embodiment, if the second pixel meets the requirements and no image processing is needed, step S106 can be omitted. In one embodiment, the order of step S106 can be interchanged or paralleled with steps S108 and S110.

[0043] Figure 2 This is a schematic diagram illustrating how an input image 20IN is locally and randomly deformed to generate an output image 20UT1, according to an embodiment of the present invention. Figure 2 The input image 20IN and output image 20UT1 can be used to implement the input image 10IN and output image 10UT1, respectively. Please note that the illustration (e.g.) Figure 2 or Figure 9 Different mesh backgrounds can be used to represent different objects, but the mesh background is not used to restrict patterns, decorations, or lines on the object. Figure 2 A grid pattern represents screw holes or their material; a dotted pattern represents screws or their material; a horizontal stripe pattern represents washers or their material; and a diagonal stripe pattern represents printed circuit boards or their material. Figure 2 (or Figures 6 to 8 In this example, white is used to represent the background, but in other embodiments, black can be used to represent the background.

[0044] In step S102, the arithmetic device 10 can obtain Figure 2 (a) The input image 20IN. In step S104, the user manually or the computing device 10 automatically (e.g., using the Grounding Dino or Segment anything model) marks a region of interest SR surrounding the first pixel (e.g., (X1, Y1)) in the input image 20IN. The shape of the region of interest SR can be regular (e.g., rectangular) or irregular (e.g., rectangular). Figure 2 (b)). In Figure 2 In (b), the color inside and outside the region of interest SR is white, but the invention is not limited to this. The color outside the region of interest SR (e.g., black) can be filled with a color different from that inside the region of interest SR (e.g., white). The processing device 10 can perform a cropping operation and can use the region of interest SR as a mask to crop or extract multiple first pixels (i.e., ...) within the region of interest SR from the input image 20IN. Figure 2 Image 210 of (c) and multiple second pixels outside the region of interest SR (i.e. Figure 2 Image 220 of (f). For example... Figure 2As shown in (c), the region of interest SR substantially marks the screw hole positions to be randomly deformed, and the computing device 10 can use the region of interest SR to obtain the screw hole position image (i.e., 210) of the input image 20IN.

[0045] In step S110, the arithmetic device 10 can... Figure 2 Image 210 of (c) is processed to produce Figure 2 Image 210a of (d). For example, the processing unit 10 may use the deformation array calculated in step S108 to perform spatial transformation on each first pixel (e.g., (X1,Y1)) within the region of interest SR. For example, the processing unit 10 may perform spatial transformation (e.g., deformation) on different layers (e.g., color channel values ​​(R,G,B) of the RGB layer) of each first pixel (e.g., (X1,Y1)) to ensure that the color of the first pixel does not change, but only the position of the first pixel changes.

[0046] In step S110, the computing device 10 may also apply image erosion or de-friction to slightly remove edge pixels located on the outer periphery of image 210a, thereby generating image 210b. This makes the image after random deformation in step S110 more suitable for image compositing. Image 210a or 210b can be used to realize the first region image.

[0047] In step S106, image 220 can be repaired to optimize the pasting or merging effect. This is because the shape of image 210a after random deformation in step S110 may differ from the shape of image 210, causing images 210a or 210b to be inconsistent with or mismatched with input image 20IN or image 220. Therefore, directly pasting or merging images 210a or 210b back to or into image 20IN or image 220 may not produce good results. For example, in step S106, the computing device 10 can, based on the second pixel (e.g., (X2, Y2)) located around the region of interest SR, perform... Figure 2 Image 220 of (f) is processed to repair, fill, or repaint the background area 240 (presented in white) surrounding the second pixel, thereby producing Figure 2the second region image 220a of (g). In an embodiment, the background region 240 can be filled by blurring or duplicating the second pixels around the region of interest SR, or by averaging or adding random noise values to the second pixels around the region of interest SR, such that the second region image 220a does not include the background region 240. In an embodiment, the inpainting can be sequential-based, CNN-based, GAN-based, or Fast Marching Method-based.

[0048] At step S112, the computing device 10 can paste or synthesize the inpainted image 210b back to the inpainted second region image 220a to generate an output image 20UT1 (may be referred to as a synthesized image).

[0049] At step S116, the computing device 10 can output or provide labeled data or unlabeled data. For example, for an image classification task, the output image provided by the computing device 10 can be considered to include a certain label (e.g., a defect label), and accordingly, the deep learning model has a higher accuracy if it classifies the output image generated by the computing device 10 to a certain (defect) class at step S116. For an image segmentation task, since the deformed region is known, the deformed region (e.g., a defect region, an image 210a, 210b, or an entire output image) can be used as a label, and accordingly, the deep learning model has a higher accuracy if it outputs this (defect) region according to the output image generated by the computing device 10 at step S116.

[0050] Figure 3 Figures 1A-1C show schematic diagrams of the present embodiment for locally deforming the input image 20IN to generate the output images 30UT1-30UT8. Any of the output images 30UT1-30UT8 can be used to implement one of the output images 10UT1-10UTn. As shown in Figures 1A-1C, the input image 20IN is deformed locally to generate the output images 30UT1-30UT8. Figure 2 and Figure 3 As can be seen, the local pixels of the input image 20IN are deformed randomly and can be converted into a large number of output images 20UT1 and 30UT1-30UT8 with sufficient diversity at step S114.

[0051] Figure 4Fig. 1 shows a schematic diagram of an embodiment of the present application for generating output images 40UT1-40UT8 from input images 20IN by overall random warping. Any of the output images 40UT1-40UT8 can be used to realize one of the output images 10UT1-10UTn. Without the region of interest (e.g. SR) being marked, the warping array of step S108 can be used to directly warp the whole input image 20IN in step S110 as another data augmentation method. By Figure 4 It is known that all pixels of the input image 20IN are randomly warped to be converted into a large number of output images 40UT1-40UT8 with sufficient diversity.

[0052] In step S116, a large number of images are needed for training the deep learning model, and the number or efficiency of the output images generated with the region of interest (e.g. SR) being marked can be insufficient. Compared with generating the output images 20UT1 or 30UT1-30UT8 with the region of interest being marked, the output images 40UT1-40UT8 generated without the region of interest being marked can have a lower degree of authenticity, but can also increase the data used for training. Moreover, the pretraining of transfer learning can use the output images generated without the region of interest being marked to omit the marking step of the region of interest to improve the performance.

[0053] In an embodiment, a data augmentation method can be compiled into a program code and used in the computing device 10, and can at least include the following steps:

[0054] Step S502: The computing device 10 generates a first array (e.g. T’ or T”). Then, step S504, S506 or S508 is executed.

[0055] Step S504: The computing device 10 converts the first array into a second array (e.g. g(T’)) using at least one filter. Then, step S506 or S508 is executed.

[0056] Step S506: The computing device 10 vector integrates the first array (or the second array) into a third array (e.g. ∫g(T’)). Then, step S508 is executed.

[0057] Step S508: The computing device 10 determines a warping array (e.g. T) according to the first array, the second array or the third array. Then, step S510 is executed.

[0058] Step S510: The computing device 10 generates at least one output image (e.g. 10UT1) corresponding to an input image (e.g. 10IN) according to the warping array.

[0059] Step S512: The operation device 10 determines whether to perform step S502 again to generate another output image (e.g., 10UT2) corresponding to the input image. Then, step S514 is performed.

[0060] Step S514: The operation device 10 trains a deep learning model using the input image or the output image.

[0061] One or more of steps S502-S514 can be deleted or changed in order according to different requirements. Step S510 can be used to implement step S110, and step S508 can be used to implement step S108.

[0062] Figure 5 Schematic diagram generated for the array of the embodiment of the present application. Figure 5 The arrays T’, T”, g(T’) can be multi-dimensional arrays, respectively. Note that, Figure 5 is drawn for a two-dimensional space, therefore, the array T’ can include elements arranged in a two-dimensional array and elements arranged in a two-dimensional array For example, the array T’ can be expressed as i.e., the number of rows, columns, groups of the array T’ are h, w, 2, respectively. The array T” can include elements of the array T’ and elements surrounding the elements of the array T’, respectively The filter 50g1 can include elements arranged in a two-dimensional array The filter 50g2 can include elements arranged in a two-dimensional array The array g(T’) can include a plurality of elements arranged in two two-dimensional arrays. However, the present application is not limited to a two-dimensional space, but can be applied to a higher-dimensional space.

[0063] In step S502, the operation device 10 can randomly generate the first array (e.g., T’ or T”) according to a normal distribution. In other words, the first array is a random array, and any element of the first array is a random number, respectively. The plurality of elements of the first array can satisfy a normal distribution, and the mean and the standard deviation of the normal distribution can be related to the hyperparameters (e.g., the equivalent displacement amplitude and the equivalent deformation amplitude) of the first pixel, respectively. For example, any element of the first array satisfies a normal distribution with the mean being the equivalent displacement amplitude and the standard deviation being the equivalent deformation amplitude, respectively. Alternatively, all elements of the first array satisfy a normal distribution with the mean being the equivalent displacement amplitude and the standard deviation being the equivalent deformation amplitude, for example, if the first array has a total of K elements, then from N~(equivalent displacement amplitude, equivalent deformation amplitude 2K values are randomly sampled from a normal distribution of the normal distribution of the first array, where different first arrays can be generated under different permutations and combinations. Please note that even if the same mean and the same standard deviation are used by the operation device 10, different first arrays can be output based on randomness, so that the operation device 10 can output various output images corresponding to a single input image.

[0064] Figure 6 Figures of different equivalent displacement amplitudes of an embodiment of the present application. Images 610a0-610a50 can correspond to equivalent displacement amplitudes of 0, 10, 20, 30, and 50, respectively, and can all be used to implement the image 210a. The equivalent displacement amplitude refers to the displacement amplitude of the random deformation region (e.g., all first pixels) as a whole in any direction. The equivalent displacement amplitude can be between 0 and infinity, where 0 represents no displacement, and the larger the equivalent displacement amplitude, the larger the displacement amplitude (e.g., screw misalignment).

[0065] Figure 7 Figures of different equivalent deformation amplitudes of an embodiment of the present application. Images 710a1-710a50 can correspond to equivalent deformation amplitudes of 1, 10, 20, 30, and 50, respectively, and can all be used to implement the image 210a. The equivalent deformation amplitude refers to the deformation intensity of the random deformation region (e.g., all first pixels) as a whole. The equivalent deformation amplitude can be between 1 and infinity, and the larger the equivalent deformation amplitude, the stronger the deformation intensity (e.g., screw slipping). After the selected hyperparameters (e.g., equivalent displacement amplitude and equivalent deformation amplitude) are selected, the first array can be randomly generated, so that the operation device 10 can determine how to move the deformation array T of the pixels of the deformation region.

[0066] Please refer again to Figure 5 To perform step S504, in an embodiment, the operation device 10 can use a normal distribution to randomly generate the array T" and use the array T" as the first array; or the operation device 10 can use a normal distribution to randomly generate the array T' and use the array T' as the first array, and then pad the first array (i.e., T') outward to generate the array T". In an embodiment, zero padding, padding with the mean, or copying the elements at the edges of the array T' can be used for outward padding. The difference in the number of rows and the difference in the number of columns between the array T" and the array T' can be related to the sliding (Stride) number of rows and the sliding number of columns of the filter, and can ensure that after step S504 is performed, the size of the array g(T') can be consistent with the size of the deformation array T or the array T'.

[0067] At step S504, the operation device 10 can apply filters (e.g., 50g1, 50g2) to the array T" (e.g., perform convolution) to calculate a second array g(T'), so as to ensure the smoothness of the movement of a first pixel of the input image (e.g., 10IN) relative to the first pixels around it and ensure the authenticity of the final output image (e.g., 10UT1). The number of filters (e.g., 2) can be determined according to the spatial dimension (e.g., two-dimensional). The filters 50g1, 50g2 can be Gaussian filters, respectively, and the standard deviation or kernel size of the Gaussian filters can be related to the hyperparameter (e.g., equivalent smooth) of the first pixel. For example, the operation device 10 can randomly generate the filters 50g1, 50g2 satisfying N~(0, equivalent smooth) respectively, and the size of the filters 50g1, 50g2 can satisfy round(equivalent smooth x 3) x 2 + 1, wherein the round function is used to round the number to an integer. The equivalent smooth used by the filters 50g1, 50g2 can be different or the same, and even if the equivalent smooth is the same, random sampling can make the filters 50g1, 50g2 different from each other. Alternatively, the filters 50g1, 50g2 can also be the same as each other. Please note that even if the second array g(T') is smoothed, the second array g(T') is still essentially a random array, and the elements thereof can have randomness respectively. 2 ) respectively, and the size of the filters 50g1, 50g2 can satisfy round(equivalent smooth x 3) x 2 + 1, wherein the round function is used to round the number to an integer. The equivalent smooth used by the filters 50g1, 50g2 can be different or the same, and even if the equivalent smooth is the same, random sampling can make the filters 50g1, 50g2 different from each other. Alternatively, the filters 50g1, 50g2 can also be the same as each other. Please note that even if the second array g(T') is smoothed, the second array g(T') is still essentially a random array, and the elements thereof can have randomness respectively.

[0068] Figure 8 The figure shows different equivalent smooths of an embodiment of the present application. The images 810a1-810a10 can correspond to the deformation amplitude of 50 and the equivalent smooth of 1, 3, 6, 8, 10 respectively, and can all be used to realize the image 210a. The equivalent smooth refers to the smoothness of the whole random deformation region (e.g., all first pixels). The equivalent smooth can be between 1 and infinity, and the larger the equivalent smooth, the smoother the deformation amplitude. In an embodiment, the equivalent displacement amplitude can be set to 0, or the ratio of the equivalent deformation amplitude to the equivalent smooth can be 50:8, 50:6 or between the two, which is beneficial to the segmentation and pasting of the region to be deformed back to the input image and ensures the deformation quality.

[0069] Please refer again to Figure 5 At step S506, the operation device 10 can calculate a third array by vector integration of the array g(T') (or T'), for example, the third array can be represented as ∫g(T'). Please note that even if the third array is subjected to vector integration, the third array is still essentially a random array, and the elements thereof can have randomness respectively.

[0070] From another perspective, the array g(T') (or T') can be understood as a velocity field, and the corresponding displacement field or deformation field (i.e., the third array) can be calculated by integrating the velocity field. The topology and its invertibility can be preserved by vector integration, where the invertibility means that the deformed image (e.g., 210a) can be deformed again to recover the pre-deformed image (e.g., 210) by integrating the negative of the array g(T') to get ∫-g(T').

[0071] In an embodiment, the array g(T') (or T') can be vector integrated with respect to time (or space) to calculate the third array. In an embodiment, the array g(T') (or T') can be a function of time, e.g., the function srand() can be used to set a random seed, and the random seed is determined based on time and thus is a function of time, so that the array g(T') (or T') can be a function of time. However, the present application is not limited thereto, and the array g(T') (or T') can also be a constant with respect to time. In an embodiment, the velocity field (e.g., g(T') or T') can involve solving to find the optimized velocity field to utilize the velocity field to vector integrate the corresponding displacement field, where ‖v t ‖ V represents a proper Sobolev norm of the velocity field v t (.), and represents a squared-error norm, I0and I1represent the input image and the deformed image before optimization, respectively, and φ1represents the deformation path φ t at the end point t = 1.

[0072] At step S508, the array T', g(T') or ∫g(T') can be used as the deformation array T (step S508), e.g., the deformation array T can satisfy T = ∫g(T'). Similarly to the arrays T', T", g(T'), the deformation array T is a multi-dimensional array corresponding to a multi-dimensional space. For example, the deformation array T can be expressed as or expressed as to determine the moving manner of each pixel in a two-dimensional space. In an embodiment, the deformation array T can be written in C++ as int T[h][w][2]. Alternatively, for a three-dimensional space, the deformation array T can be expressed as pixels in a displacement three-dimensional space, wherein h, w, d, 3 respectively correspond to height, width, depth, and spatial dimension. In an embodiment, height h can be less than or equal to image height H of the input image and greater than or equal to maximum height of the region of interest SR, width w can be less than or equal to image width W of the input image and greater than or equal to maximum width of the region of interest SR, and depth d can be less than or equal to image depth of the input image and greater than or equal to maximum depth of the region of interest SR, because deforming the background region 230 (presented in white) with the deformation array T does not affect the effect of the output image. In an embodiment, h, H, i, I, j, J, K, n, N, m, M, p, P, q, Q, X1, X2, Y1, Y2, w, or W are positive integers greater than or equal to 1, respectively.

[0073] At step S510, the operation device 10 can generate an output image (e.g., 10UT1) corresponding to an input image (e.g., 10IN) according to the deformation array T. In an embodiment, the displacement amount of a (first) pixel of an input image constitutes a deformation array T or corresponds to an element of a deformation array T, respectively. Alternatively, each element of a deformation array T represents the displacement amount of each coordinate point (i.e., each pixel) of an input image in space. Alternatively, each element of the deformation array T represents the coordinate of each pixel of an input image in an output image. For example, the element of the deformation array T If equal to (5, 3), it represents that the (first) pixel of the input image located at coordinate (2, 1) is deformed to coordinate (5, 3) in the output image. If equal to (5, 3), it represents that the (first) pixel of the input image located at coordinate (2, 1) is deformed to coordinate (5, 3) in the output image.

[0074] Since the array T’, g(T’), or ∫g(T’) is essentially a random array, the deformation array T can be a random deformation field, and each element of the deformation array T can have randomness, respectively. Further, since the deformation array T can be essentially generated by a normal distribution, under the same set of hyperparameters (i.e., the same equivalent displacement amplitude, equivalent deformation amplitude, and equivalent smoothness), different random arrays (e.g., T) can be generated in large quantities, so that the operation device 10 can output a variety and a large number of output images corresponding to a single input image. Moreover, the generation of the deformation array T can be controlled by using the hyperparameters (e.g., the equivalent displacement amplitude, the equivalent deformation amplitude, or the equivalent smoothness), in other words, by changing the hyperparameters, the operation device 10 can also output a variety and a large number of output images corresponding to a single input image.

[0075] Figure 9Fig. 1 is a schematic diagram of an embodiment of the present application for performing random morphing on an input image 10IN to generate output images 10UT1-10UT8. The input image 10IN and the output images 10UT1-10UT8 can be used to implement the input image 9IN and the output images 9UT1, respectively. The input image 10IN is a raw image of a buried wire defect, and the output images 10UT1-10UT8 are images of the buried wire defect generated by the operation device 10 to present defects in which the wire is not inserted into the slot. In other words, the present application can not only generate defect images of screw holes, but also can be applied to other types of parts to generate defect images of other parts. Figure 9 The cross-shaped net bottom represents a circuit or its material, and the triangular net bottom represents a slot or its material. The input image 10IN is a raw image of a buried wire defect, and the output images 10UT1-10UT8 are images of the buried wire defect generated by the operation device 10 to present defects in which the wire is not inserted into the slot. In other words, the present application can not only generate defect images of screw holes, but also can be applied to other types of parts to generate defect images of other parts.

[0076] The present application can generate output images by moving pixels of an input image, and can explain random morphing (or how to move pixels of an input image) in a visual manner, thereby ensuring the interpretability and rationality of random morphing. For example, Figure 10 Fig. 1 is a schematic diagram of an embodiment of the present application for performing random morphing on an input image 10IN to generate output images 10UT1-10UT8. The input image 10IN and the output images 10UT1-10UT8 can be used to implement the input image 9IN and the output images 9UT1, respectively. The input image 10IN is a raw image of a buried wire defect, and the output images 10UT1-10UT8 are images of the buried wire defect generated by the operation device 10 to present defects in which the wire is not inserted into the slot. In other words, the present application can not only generate defect images of screw holes, but also can be applied to other types of parts to generate defect images of other parts.

[0077] To verify the rationality of the present application, experiments can be used to confirm whether the performance of a deep learning model can be improved by adding different numbers of synthetic images (i.e., output images of the operation device 10). For example, according to Table 1, the accuracy of a deep learning model can be improved by training the model using output images generated by the present application.

[0078] Table 1

[0079]

[0080]

[0081] Further, the existing image generation methods (e.g. simple flipping, simple rotation, simple translation or simple color jittering) can not be able to generate enough diverse defect images. For example, after simple rotation, the region of the screw thread slip is still the same shape, and more diverse slip shapes cannot be generated. In contrast, the present application can generate a large number of output images (e.g. defect images of different slip shapes or thread chipping) with high fidelity and sufficient diversity by random deformation method under the condition of only one input image. Moreover, the output images of the present application can be combined with existing image generation methods to derive more output images. From another point of view, the present application is not limited to random translation deformation, but achieves deformation by randomly generating a random matrix, so that the essence of moving pixel positions essentially implies (local or global) random rotation or (local) random flipping.

[0082] In addition, compared with existing deep learning techniques, the present application does not need to collect a large number of images in advance. Even if there is only one input image, the present application can generate multiple output images. Moreover, the present application does not need to train the model and does not need to perform model inference to generate multiple output images, so it can save computing resources. The present application can visually explain how to generate output images, unlike the black box operation of existing deep learning techniques. More importantly, the output images generated by the present application can be used to train the model of deep learning techniques, which can solve the dilemma that existing deep learning techniques cannot overcome when lacking training data.

[0083] The above only describes the preferred embodiments of the present application, and any equivalent changes and modifications made according to the claims of the present application shall be within the scope of the present application.

Claims

1. A data augmentation method, comprising: obtaining an input image; and generating a plurality of output images corresponding to the input image, wherein at least one first pixel of the input image is displaced to form one of the plurality of output images, and a displacement amount of any of the at least one first pixel has randomness. 2.The data augmentation method of claim 1, further comprising: generating a first array, wherein, at least one element of the first array is a random number; and determining a deformation array according to the first array, wherein the deformation array is composed of at least one displacement amount of the at least one first pixel of the input image or at least one coordinate value of the at least one first pixel in the output image.

3. The data propagation method of claim 2, wherein, The at least one element of the first array satisfies a normal distribution, a mean of the normal distribution is related to an equivalent displacement amplitude, and a standard deviation of the normal distribution is related to an equivalent deformation amplitude. 4.The data augmentation method of claim 2, further comprising: converting the first array or the third array into the second array using at least one filter, wherein, the step of determining the deformation array according to the first array comprises determining the deformation array according to the second array.

5. The data propagation method of claim 4, wherein, One of the at least one filter is a Gaussian filter, and a standard deviation or size of the Gaussian filter is related to an equivalent smoothness. 6.The data augmentation method of claim 2, further comprising: vector integrating the first array or the second array into a third array, wherein, the step of determining the deformation array according to the first array comprises determining the deformation array according to the third array. 7.The data augmentation method of claim 1, further comprising: training a deep learning model using the input image or the plurality of output images.

8. The data propagation method of claim 1, wherein, The at least one first pixel is all or part of the pixels of the input image. 9.The data augmentation method of claim 1, further comprising: dividing the input image into the at least one first pixel and at least one second pixel; and performing a first image processing on the at least one first pixel to form a first region image, wherein the first image processing comprises displacing the at least one first pixel respectively; wherein the step of generating the plurality of output images corresponding to the input image comprises performing image synthesis according to the first region image and the at least one second pixel to generate one of the plurality of output images. 10.The data augmentation method of claim 1, further comprising: performing second image processing according to the at least one second pixel to form a second region image; wherein, the step of performing image synthesis according to the first region image and the at least one second pixel comprises synthesizing the first region image and the second region image, wherein the first image processing comprises removing at least one edge pixel of the displaced at least one first pixel to form the first region image, and the at least one edge pixel is located at an edge of the displaced at least one first pixel. 11.An operation apparatus, comprising: a storage circuit configured to store instructions, the instructions comprising: obtaining an input image; and generating a plurality of output images corresponding to the input image, wherein at least one first pixel of the input image is displaced to form one of the plurality of output images, and a displacement amount of any of the at least one first pixel has randomness; and a processing circuit coupled to the storage circuit and configured to execute the instructions.

12. The arithmetic device according to claim 11, wherein the instructions further comprising: generating a first array, wherein at least one element of the first array is a random number; and determining a deformation array according to the first array, wherein the deformation array is composed of at least one displacement amount of the at least one first pixel of the input image or at least one coordinate value of the at least one first pixel in the output image. The deformation array is determined according to the first array, wherein the deformation array is composed of at least one displacement value of the at least one first pixel of the input image or at least one coordinate value of the at least one first pixel in the output image.

13. The arithmetic device according to claim 12, wherein The at least one element of the first array satisfies a normal distribution, wherein a mean of the normal distribution is related to an equivalent displacement amplitude and a standard deviation of the normal distribution is related to an equivalent deformation amplitude.

14. The arithmetic device according to claim 12, wherein The instructions further comprise: The first array or the third array is converted into a second array by using at least one filter, wherein the step of determining the deformation array according to the first array comprises determining the deformation array according to the second array.

15. The arithmetic device according to claim 14, wherein One of the at least one filter is a Gaussian filter, wherein a standard deviation or a size of the Gaussian filter is related to an equivalent smoothness.

16. The arithmetic device of claim 12, wherein, The instructions further comprise: The first array or the second array is vector integrated into a third array, wherein the step of determining the deformation array according to the first array comprises determining the deformation array according to the third array.

17. The arithmetic device of claim 11, wherein, The instructions further comprise: A deep learning model is trained by using the input image or the plurality of output images.

18. The arithmetic device of claim 11, wherein, The at least one first pixel is all or part of the pixels of the input image.

19. The arithmetic device of claim 11, wherein, The instructions further comprise: The input image is divided into the at least one first pixel and at least one second pixel; and The at least one first pixel is subjected to a first image processing to form a first region image, wherein the first image processing comprises respectively displacing the at least one first pixel. The plurality of output images are generated according to the first region image and the at least one second pixel by image synthesis, to generate one of the plurality of output images.

20. The arithmetic device of claim 11, wherein, The instructions further comprise: A second region image is formed by a second image processing according to the at least one second pixel. The first region image and the second region image are synthesized according to the first region image and the at least one second pixel by image synthesis. The first image processing comprises removing at least one edge pixel of the displaced at least one first pixel to form the first region image, wherein the at least one edge pixel is located at an edge of the displaced at least one first pixel.

Images