Image data augmentation method and device based on image destruction processing
By using an image diffusion model to perform continuous destruction and restoration processes on the image, the problem that existing image data enhancement methods cannot simulate changes in real-world application scenarios is solved. The generated images are more consistent with the actual scene, thus improving the effect of data enhancement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING LONGZHI DIGITAL TECH CO LTD
- Filing Date
- 2023-04-07
- Publication Date
- 2026-07-31
AI Technical Summary
Existing image data augmentation methods cannot accurately simulate visual and environmental changes in real-world application scenarios, resulting in generated images that do not conform to the changes in images in real-world application scenarios.
The image is subjected to multiple consecutive destruction and restoration processes through the diffusion and de-diffusion processes of the image diffusion model, generating an augmented image dataset, ensuring that each restoration process corresponds to the inverse process of the destruction process.
This makes the generated image dataset more consistent with the image changes in real-world application scenarios, thus improving the effect of data augmentation.
Smart Images
Figure CN116596813B_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of image processing technology, and in particular to an image data enhancement method and apparatus based on image destruction processing. Background Technology
[0002] In computer vision, image data augmentation is a common method used to enrich training datasets and improve the generalization ability of models. Existing image data augmentation methods typically generate new image data by performing a series of affine transformations on the original image. Common affine transformations include random rotation, flipping, and cropping. For example, existing image data augmentation methods randomly select a region from the original image, crop it, randomly rotate, slightly stretch, or flip the cropped image, and then add the transformed image to the training dataset. However, this method fails to accurately reproduce the changes in images in real-world applications due to random transformations. It cannot effectively simulate visual and environmental changes in real-world applications, such as changes in lighting and viewing angles. Therefore, the generated images often do not conform to the changes in images encountered in real-world application scenarios.
[0003] In realizing the concept disclosed herein, the inventors discovered at least the following technical problems in the related technologies: the images obtained by traditional data augmentation methods do not conform to the changes in images in actual application scenarios. Summary of the Invention
[0004] In view of this, the present disclosure provides an image data enhancement method, apparatus, electronic device, and computer-readable storage medium based on image destruction processing, to solve the problem that the images obtained by traditional data enhancement methods in the prior art do not conform to the changes in images in actual application scenarios.
[0005] A first aspect of this disclosure provides an image data enhancement method based on image destruction processing, comprising: acquiring an image dataset to be enhanced; performing destruction processing on a target image in the image dataset multiple times consecutively using the diffusion process of an image diffusion model to obtain a first destroyed image corresponding to the target image; performing restoration processing on the first destroyed image multiple times consecutively using the reverse diffusion process of an image diffusion model to obtain a first restored image corresponding to the target image, wherein each restoration processing corresponds to one destruction processing, and each restoration processing corresponds to the inverse process of its destruction processing; and generating an enhanced image dataset using the target image and the first restored image.
[0006] A second aspect of this disclosure provides an image data enhancement apparatus based on image destruction processing, comprising: an acquisition module configured to acquire an image dataset to be enhanced; a diffusion module configured to perform continuous destruction processing on a target image in the image dataset multiple times using the diffusion process of an image diffusion model to obtain a first destroyed image corresponding to the target image; an inverse diffusion module configured to perform continuous restoration processing on the first destroyed image multiple times using the inverse diffusion process of an image diffusion model to obtain a first restored image corresponding to the target image, wherein each restoration processing corresponds to one destruction processing, and the destruction processing corresponding to each restoration processing is its inverse process; and an enhancement module configured to generate an enhanced image dataset using the target image and the first restored image.
[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.
[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0009] The beneficial effects of this disclosure embodiment compared with the prior art are as follows: because this disclosure embodiment obtains an image dataset to be data augmented; uses the diffusion process of the image diffusion model to continuously perform multiple destruction processes on the target image in the image dataset to obtain a first destroyed image corresponding to the target image; uses the reverse diffusion process of the image diffusion model to continuously perform multiple restoration processes on the first destroyed image to obtain a first restored image corresponding to the target image, wherein each restoration process corresponds to one destruction process, and the destruction process corresponding to each restoration process is its inverse process; and uses the target image and the first restored image to generate a data-augmented image dataset, therefore, by adopting the above technical means, the problem that the image obtained by the traditional data augmentation method in the prior art does not conform to the changes in the image in the actual application scenario can be solved, thereby making the image obtained by data augmentation conform to the changes in the image in the actual application scenario. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure;
[0012] Figure 2 This is a schematic flowchart of an image data enhancement method based on image destruction processing provided in this disclosure embodiment;
[0013] Figure 3 This is a schematic diagram of the structure of an image data enhancement device based on image destruction processing provided in an embodiment of this disclosure;
[0014] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation
[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.
[0016] The following describes in detail, with reference to the accompanying drawings, an image data enhancement method and apparatus based on image destruction processing according to embodiments of the present disclosure.
[0017] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 101, 102, and 103, server 104, and network 105.
[0018] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays that support communication with server 104, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. Terminal devices 101, 102, and 103 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not impose any limitations on this. Furthermore, various applications can be installed on terminal devices 101, 102, and 103, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.
[0019] Server 104 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 104 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This embodiment of the disclosure does not impose any limitations on these aspects.
[0020] It should be noted that server 104 can be either hardware or software. When server 104 is hardware, it can be various electronic devices that provide various services to terminal devices 101, 102, and 103. When server 104 is software, it can be multiple software programs or software modules that provide various services to terminal devices 101, 102, and 103, or it can be a single software program or software module that provides various services to terminal devices 101, 102, and 103. This disclosure does not limit the scope of the embodiments.
[0021] Network 105 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), Infrared, etc. This disclosure does not limit the scope of the network.
[0022] Users can establish a communication connection with server 104 via network 105 through terminal devices 101, 102, and 103 to receive or send information, etc. It should be noted that the specific types, quantities, and combinations of terminal devices 101, 102, and 103, server 104, and network 105 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any limitations on this.
[0023] Figure 2 This is a schematic flowchart of an image data enhancement method based on image destruction processing provided in an embodiment of this disclosure. Figure 2 Image data augmentation methods based on image destruction processing can be derived from... Figure 1 The computer or server, or the software on the computer or server, executes the command. For example... Figure 2 As shown, this image data enhancement method based on image destruction processing includes:
[0024] S201, Obtain the image dataset to be augmented;
[0025] S202, the target image in the image dataset is repeatedly damaged by the diffusion process of the image diffusion model to obtain the first damaged image corresponding to the target image;
[0026] S203, the first damaged image is restored multiple times using the inverse diffusion process of the image diffusion model to obtain the first restored image corresponding to the target image. Each restoration process corresponds to a destruction process, and the destruction process corresponding to each restoration process is its inverse process.
[0027] S204, using the target image and the first restored image to generate a data-enhanced image dataset.
[0028] The image diffusion model consists of two processes: diffusion and inverse diffusion. Diffusion involves repeatedly destructing the target image in the image dataset to obtain the first destroyed image. Inverse diffusion involves determining the destruction process performed on the target image at each step during diffusion and performing the corresponding restoration process to obtain the first restored image.
[0029] The image dataset contains multiple target images. For ease of understanding, we can first consider each target image as a single image. After a diffusion process, we obtain the first damaged image corresponding to the target image. After a reverse diffusion process, we obtain the first restored image corresponding to the target image. Following this method, we obtain the first restored image corresponding to each target image. Then, all the target images and their corresponding first restored images constitute the data-augmented image dataset.
[0030] The diffusion model is primarily a denoising model in terms of structure, and can be a U-net structure composed of multiple convolutional and deconvolutional layers. This embodiment utilizes a trained image diffusion model for image data augmentation.
[0031] According to the technical solution provided in this disclosure, an image dataset to be augmented is obtained; the target image in the image dataset is subjected to multiple consecutive destructive processes using the diffusion process of an image diffusion model to obtain a first destructive image corresponding to the target image; the first destructive image is subjected to multiple consecutive restoration processes using the reverse diffusion process of an image diffusion model to obtain a first restored image corresponding to the target image, wherein each restoration process corresponds to one destructive process, and the destructive process corresponding to each restoration process is its inverse process; the target image and the first restored image are used to generate an augmented image dataset. Therefore, by adopting the above technical means, the problem that the image obtained by the traditional data augmentation method does not conform to the changes in the image in the actual application scenario can be solved, thereby making the image obtained by data augmentation conform to the changes in the image in the actual application scenario.
[0032] By controlling the number of times the target image is processed using the image diffusion model, controlling the number of first restored images corresponding to the target image, and controlling the size of the data-enhanced image dataset, the image diffusion model undergoes one diffusion process and one reverse diffusion process, which is denoted as processing the target image once.
[0033] Each time the image diffusion model is used to process the target image, a first restored image corresponding to the target image is obtained. When the image diffusion model is used to process the target image multiple times, a large number of first restored images corresponding to the target image are obtained (because the destruction process of the target image can be different each time the image diffusion model is used to process the target image, so the image diffusion model can be used to achieve image data enhancement).
[0034] Before performing multiple consecutive destructive processes on the target image in the image dataset using the diffusion process of the image diffusion model to obtain the first destructive image corresponding to the target image, the method further includes: obtaining a training dataset, training the image diffusion model using the training dataset, so that the image diffusion model can determine and execute the restoration process corresponding to the destructive process performed on the training image in the training dataset by the diffusion process during the reverse diffusion process.
[0035] The trained image diffusion model can perform destructive processing on the training images in the training dataset during the diffusion process, and determine and execute the corresponding restoration processing during the inverse diffusion process. The diffusion process performs multiple destructive processes on the training images in the training dataset, and the inverse diffusion process performs multiple restoration processes. Each restoration process corresponds to one destructive process, and the destructive process corresponding to each restoration process is its inverse process.
[0036] After obtaining the image dataset to be augmented, the method further includes: using the diffusion process of the image diffusion model to repeatedly perform destructive processing and add noise to the target image in the image dataset to obtain a second destructive image corresponding to the target image; using the reverse diffusion process of the image diffusion model to determine the destructive processing and added noise to the target image each time during the diffusion process, and repeatedly performing restoration processing and noise removal corresponding to the destructive processing on the second destructive image to obtain a second restored image corresponding to the target image; and using the target image and the second restored image to generate the augmented image dataset.
[0037] The target image is subjected to one destruction process and noise addition in sequence, which is equivalent to the diffusion process processing the target image once. This diffusion process is repeated multiple times to obtain the second destroyed image corresponding to the target image. The target image is then subjected to one restoration process and noise removal, which is equivalent to the reverse diffusion process processing the target image once. This reverse diffusion process is repeated multiple times to obtain the second restored image corresponding to the target image.
[0038] Before repeatedly performing destructive processing and adding noise to the target image in the image dataset using the diffusion process of the image diffusion model to obtain the second destructive image corresponding to the target image, the method further includes: obtaining a training dataset; repeatedly performing destructive processing and adding noise to the training images in the training dataset using the diffusion process of the image diffusion model to obtain the third destructive image corresponding to the training image; determining the destructive processing and added noise to the training image each time during the diffusion process using the reverse diffusion process of the image diffusion model, and repeatedly performing restoration processing and noise removal corresponding to the destructive processing on the third destructive image to obtain the third restored image corresponding to the training image; calculating the noise loss between the multiple noises added during the diffusion process and the multiple noises determined during the reverse diffusion process, and calculating the destruction loss between the destructive processing performed during the diffusion process and the restoration processing performed during the reverse diffusion process; updating the model parameters of the image diffusion model based on the noise loss and destruction loss to complete the training of the image diffusion model.
[0039] The mean squared error can be used to calculate the loss between each noise added during diffusion and the corresponding noise predicted during reverse diffusion; the sum of all calculated losses is taken as the noise loss. The cross-entropy loss function can be used to calculate the destruction loss between the destruction label and the corresponding recovery treatment.
[0040] The target image is destructed and noise is added according to the following formula:
[0041] x t =F t (x0)+σε,ε~q(ε)
[0042] Among them, F t For the t-th destruction process, the destruction process includes pooling, blurring, and occlusion operations, x t This is the target image after the t-th destruction process and the addition of noise. When t equals 1, x0 is the target image; when t equals N, x... N It is the second damaged image corresponding to the target image, where N is a preset number and ε 为 Noise, q() is the target distribution, which includes Gaussian distribution, uniform distribution and t-distribution, ε~q(ε) 表示 ε 满足 q(), where σ is the variance of q().
[0043] Destruction processing can include operations such as pooling, blurring, and masking, and the target distribution can be Gaussian, uniform, or t-distribution.
[0044] In an optional embodiment, a loss function is used to calculate the loss, and the model parameters of the image diffusion model are updated based on the loss to complete the training of the image diffusion model.
[0045] The loss function is as follows:
[0046]
[0047] Among them, G t For the t-th recovery process, F t For the t-th destruction process, G t With F t It is corresponding, x t-1 This is the target image after the (t-1)th destruction process and the addition of noise. When t equals 1, x0 is the target image; when t equals N, x... N It is the second damaged image corresponding to the target image, N is a preset number, ε is noise, ε satisfies the target distribution, σ is the variance of the target distribution, ||||1 indicates that a norm operation is performed, T is the total number of times the damage processing and noise addition are performed, and T is numerically equal to t.
[0048] In this embodiment, noise loss and damage loss are calculated as a whole.
[0049] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.
[0050] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.
[0051] Figure 3 This is a schematic diagram of an image data enhancement device based on image destruction processing provided in an embodiment of this disclosure. Figure 3 As shown, the image data enhancement device based on image destruction processing includes:
[0052] The acquisition module 301 is configured to acquire the image dataset to be augmented.
[0053] The diffusion module 302 is configured to continuously and repeatedly destroy the target image in the image dataset using the diffusion process of the image diffusion model to obtain the first destroyed image corresponding to the target image.
[0054] The inverse diffusion module 303 is configured to perform restoration processing on the first damaged image multiple times in succession using the inverse diffusion process of the image diffusion model to obtain the first restored image corresponding to the target image. Each restoration process corresponds to one destruction process, and the destruction process corresponding to each restoration process is its inverse process.
[0055] Enhancement module 304 is configured to generate an enhanced image dataset using the target image and the first restored image.
[0056] The image diffusion model consists of two processes: diffusion and inverse diffusion. Diffusion involves repeatedly destructing the target image in the image dataset to obtain the first destroyed image. Inverse diffusion involves determining the destruction process performed on the target image at each step during diffusion and performing the corresponding restoration process to obtain the first restored image.
[0057] The image dataset contains multiple target images. For ease of understanding, we can first consider each target image as a single image. After a diffusion process, we obtain the first damaged image corresponding to the target image. After a reverse diffusion process, we obtain the first restored image corresponding to the target image. Following this method, we obtain the first restored image corresponding to each target image. Then, all the target images and their corresponding first restored images constitute the data-augmented image dataset.
[0058] The diffusion model is primarily a denoising model in terms of structure, and can be a U-net structure composed of multiple convolutional and deconvolutional layers. This embodiment utilizes a trained image diffusion model for image data augmentation.
[0059] According to the technical solution provided in this disclosure, an image dataset to be augmented is obtained; the target image in the image dataset is subjected to multiple consecutive destructive processes using the diffusion process of an image diffusion model to obtain a first destructive image corresponding to the target image; the first destructive image is subjected to multiple consecutive restoration processes using the reverse diffusion process of an image diffusion model to obtain a first restored image corresponding to the target image, wherein each restoration process corresponds to one destructive process, and the destructive process corresponding to each restoration process is its inverse process; the target image and the first restored image are used to generate an augmented image dataset. Therefore, by adopting the above technical means, the problem that the image obtained by the traditional data augmentation method does not conform to the changes in the image in the actual application scenario can be solved, thereby making the image obtained by data augmentation conform to the changes in the image in the actual application scenario.
[0060] Optionally, the enhancement module 304 is further configured to control the number of times the target image is processed using the image diffusion model, control the number of first restored images corresponding to the target image, and control the size of the data-enhanced image dataset, wherein the image diffusion model undergoes one diffusion process and one reverse diffusion process, which is denoted as processing the target image once.
[0061] Each time the image diffusion model is used to process the target image, a first restored image corresponding to the target image is obtained. When the image diffusion model is used to process the target image multiple times, a large number of first restored images corresponding to the target image are obtained (because the destruction process of the target image can be different each time the image diffusion model is used to process the target image, so the image diffusion model can be used to achieve image data enhancement).
[0062] Optionally, the diffusion module 302 is also configured to acquire a training dataset and train the image diffusion model using the training dataset, so that the image diffusion model can determine and execute the restoration process corresponding to the destruction process performed on the training images in the training dataset by the diffusion process during the reverse diffusion process.
[0063] The trained image diffusion model can perform destructive processing on the training images in the training dataset during the diffusion process, and determine and execute the corresponding restoration processing during the inverse diffusion process. The diffusion process performs multiple destructive processes on the training images in the training dataset, and the inverse diffusion process performs multiple restoration processes. Each restoration process corresponds to one destructive process, and the destructive process corresponding to each restoration process is its inverse process.
[0064] Optionally, the diffusion module 302 is further configured to continuously perform destructive processing and add noise to the target image in the image dataset multiple times using the diffusion process of the image diffusion model to obtain a second destructive image corresponding to the target image; determine the destructive processing and added noise to the target image each time during the diffusion process using the reverse diffusion process of the image diffusion model, and continuously perform restoration processing and noise removal corresponding to the destructive processing on the second destructive image multiple times to obtain a second restored image corresponding to the target image; and generate a data-enhanced image dataset using the target image and the second restored image.
[0065] The target image is subjected to one destruction process and noise addition in sequence, which is equivalent to the diffusion process processing the target image once. This diffusion process is repeated multiple times to obtain the second destroyed image corresponding to the target image. The target image is then subjected to one restoration process and noise removal, which is equivalent to the reverse diffusion process processing the target image once. This reverse diffusion process is repeated multiple times to obtain the second restored image corresponding to the target image.
[0066] Optionally, the diffusion module 302 is further configured to: acquire a training dataset; repeatedly perform destructive processing and add noise to the training images in the training dataset using the diffusion process of the image diffusion model to obtain a third destructive image corresponding to the training images; determine the destructive processing and added noise to the training images each time during the diffusion process using the inverse diffusion process of the image diffusion model, and repeatedly perform restoration processing and noise removal corresponding to the destructive processing on the third destructive image to obtain a third restored image corresponding to the training images; calculate the noise loss between the multiple noises added during the diffusion process and the multiple noises determined during the inverse diffusion process, and calculate the destruction loss between the destructive processing performed during the diffusion process and the restoration processing performed during the inverse diffusion process; and update the model parameters of the image diffusion model based on the noise loss and destruction loss to complete the training of the image diffusion model.
[0067] The mean squared error can be used to calculate the loss between each noise added during diffusion and the corresponding noise predicted during reverse diffusion; the sum of all calculated losses is taken as the noise loss. The cross-entropy loss function can be used to calculate the destruction loss between the destruction label and the corresponding recovery treatment.
[0068] Optionally, the diffusion module 302 is also configured to degrade the target image and add noise according to the following formula:
[0069] x t =F t (x0)+σε,ε~q(ε)
[0070] Among them, F t For the t-th destruction process, the destruction process includes pooling, blurring, and occlusion operations, x t This is the target image after the t-th destruction process and the addition of noise. When t equals 1, x0 is the target image; when t equals N, x... N It is the second damaged image corresponding to the target image, N is the preset number, ε is the noise, q() is the target distribution, the target distribution includes Gaussian distribution, uniform distribution and t distribution, ε~q(ε) means that ε satisfies q(), and σ is the variance of q().
[0071] Destruction processing can include operations such as pooling, blurring, and masking, and the target distribution can be Gaussian, uniform, or t-distribution.
[0072] Optionally, the diffusion module 302 is also configured to calculate the loss using a loss function, and update the model parameters of the image diffusion model based on the loss, so as to complete the training of the image diffusion model;
[0073] The loss function is as follows:
[0074]
[0075] Among them, G t For the t-th recovery process, F t For the t-th destruction process, G t With F t It is corresponding, x t-1 This is the target image after the (t-1)th destruction process and the addition of noise. When t equals 1, x0 is the target image; when t equals N, x... N It is the second damaged image corresponding to the target image, N is a preset number, ε is noise, ε satisfies the target distribution, σ is the variance of the target distribution, ||||1 indicates that a norm operation is performed, T is the total number of times the damage processing and noise addition are performed, and T is numerically equal to t.
[0076] In this embodiment, noise loss and damage loss are calculated as a whole.
[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.
[0078] Figure 4 This is a schematic diagram of the electronic device 4 provided in an embodiment of this disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module / unit in the various device embodiments described above.
[0079] Electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 4 may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or different components.
[0080] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0081] The memory 402 can be an internal storage unit of the electronic device 4, such as a hard disk or RAM of the electronic device 4. The memory 402 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 4. The memory 402 can also include both internal and external storage units of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.
[0082] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0083] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0084] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.
Claims
1. An image data augmentation method based on a destructive process on an image, characterized by, include: Obtain the image dataset to be augmented; The target image in the image dataset is subjected to repeated destruction processes using the diffusion process of the image diffusion model to obtain the first destroyed image corresponding to the target image. The first damaged image is restored multiple times using the inverse diffusion process of the image diffusion model to obtain the first restored image corresponding to the target image. Each restoration process corresponds to one destruction process, and the destruction process corresponding to each restoration process is its inverse process. The data-enhanced image dataset is generated using the target image and the first restored image; After obtaining the image dataset to be augmented, the method further includes: repeatedly performing the destruction process and adding noise on the target image in the image dataset using the diffusion process of the image diffusion model to obtain a second destroyed image corresponding to the target image; determining the destruction process and added noise on the target image each time during the diffusion process using the reverse diffusion process of the image diffusion model, and repeatedly performing the restoration process and noise removal corresponding to the destruction process on the second destroyed image to obtain a second restored image corresponding to the target image; and generating the augmented image dataset using the target image and the second restored image. The target image is subjected to the destructive processing and noise addition according to the following formula: Among them, F t The destruction process for the t-th iteration includes pooling, blurring, and masking operations. This is the target image after the t-th destruction process and the addition of noise. When t equals 1, The target image, when t equals N, It is the second damaged image corresponding to the target image, where N is a preset number. Let q() be the noise, and q() be the target distribution, which includes Gaussian distribution, uniform distribution, and t-distribution. express The condition q() is satisfied, and σ is the variance of q().
2. The method of claim 1, wherein, include: By controlling the number of times the target image is processed using the image diffusion model, controlling the number of first restored images corresponding to the target image, and controlling the size of the data-enhanced image dataset, wherein each diffusion process and reverse diffusion process of the image diffusion model is denoted as processing the target image once.
3. The method of claim 1, wherein, Before the diffusion process using the image diffusion model is used to repeatedly destroy the target image in the image dataset to obtain the first destroyed image corresponding to the target image, the method further includes: Obtain a training dataset and train the image diffusion model using the training dataset, so that the image diffusion model can determine and execute the restoration process corresponding to the destruction process performed by the diffusion process on the training images in the training dataset during the reverse diffusion process.
4. The method of claim 1, wherein, Before the second damaged image corresponding to the target image is obtained by repeatedly performing the destruction process and adding noise on the target image in the image dataset using the diffusion process of the image diffusion model, the method further includes: Obtain the training dataset; The training images in the training dataset are subjected to the destruction process and noise addition multiple times using the diffusion process of the image diffusion model to obtain the third destroyed image corresponding to the training image. The inverse diffusion process of the image diffusion model is used to determine the destruction process and noise added to the training image each time during the diffusion process, and the restoration process and noise removal corresponding to the destruction process are performed on the third destroyed image multiple times in succession to obtain the third restored image corresponding to the training image. Calculate the noise loss between multiple noises added during the diffusion process and multiple noises determined during the reverse diffusion process, and calculate the damage loss between the destructive treatment performed during the diffusion process and the recovery treatment performed during the reverse diffusion process; Based on the noise loss and the destruction loss, the model parameters of the image diffusion model are updated to complete the training of the image diffusion model.
5. The method of claim 4, wherein, include: The loss is calculated using a loss function, and the model parameters of the image diffusion model are updated based on the loss to complete the training of the image diffusion model. The loss function is as follows: Among them, G t For the t-th recovery process, F t For the t-th destruction process, G t With F t It is corresponding. This is the target image after the (t-1)th destructive process and the addition of noise. When t equals 1, The target image, when t equals N, It is the second damaged image corresponding to the target image, where N is a preset number. For noise, The target distribution is satisfied, σ is the variance of the target distribution, ||||1 represents the first norm operation, T0 is the total number of times the destructive processing and noise addition are performed, and T is numerically equal to t.
6. An image data enhancement apparatus based on a destructive process of an image, characterized by, include: The acquisition module is configured to acquire the image dataset to be augmented. After acquiring the image dataset to be augmented, the acquisition module is further configured to: perform the destruction process and add noise to the target image in the image dataset multiple times in succession using the diffusion process of the image diffusion model to obtain the second destroyed image corresponding to the target image; The inverse diffusion process of the image diffusion model is used to determine the destruction process and noise added to the target image each time during the diffusion process, and the second destroyed image is repeatedly subjected to the restoration process and noise removal corresponding to the destruction process to obtain the second restored image corresponding to the target image. The data-enhanced image dataset is generated using the target image and the second restored image; The target image is subjected to the destructive processing and noise addition according to the following formula: Among them, F t The destruction process for the t-th iteration includes pooling, blurring, and masking operations. This is the target image after the t-th destruction process and the addition of noise. When t equals 1, The target image, when t equals N, It is the second damaged image corresponding to the target image, where N is a preset number. Let q() be the noise, and q() be the target distribution, which includes Gaussian distribution, uniform distribution, and t-distribution. express The condition q() is satisfied, and σ is the variance of q(). The diffusion module is configured to continuously and repeatedly destroy the target image in the image dataset using the diffusion process of the image diffusion model to obtain the first destroyed image corresponding to the target image. The inverse diffusion module is configured to perform restoration processing on the first damaged image multiple times consecutively using the inverse diffusion process of the image diffusion model to obtain the first restored image corresponding to the target image, wherein each restoration process corresponds to one destruction process, and the destruction process corresponding to each restoration process is its inverse process. The enhancement module is configured to generate the data-enhanced image dataset using the target image and the first restored image.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 7. When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.