Methods, apparatus, and computer program products for data augmentation

Abstract

Embodiments of the present disclosure relate to a method, an electronic device and a computer program product for data augmentation. The method comprises: generating a set of candidate images based on a target image by using a thermodynamic genetic algorithm (TDGA) model, wherein the TDGA model is configured to apply one or more operations in a set of predetermined image processing operations in each evolution process; and determining a plurality of augmented images from the set of candidate images based on a free energy of the set of candidate images, the plurality of augmented images being determined to have a same classification as the target image. In this way, data augmentation can be efficiently implemented by using the thermodynamic genetic algorithm.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of computers, and more specifically, to methods, apparatus, and computer program products for data enhancement. Background Technology

[0002] With the development of computer technology, machine learning technology has been gradually applied to all aspects of people's lives. Computing devices can use machine learning models to perform a wide variety of tasks.

[0003] The performance of machine learning models is closely related to the size and quality of the training dataset. In some fields, it is often difficult to obtain a sufficient amount of training data. Therefore, how to effectively augment data has become a focus of attention. Summary of the Invention

[0004] Embodiments of this disclosure provide a scheme for data augmentation.

[0005] According to a first aspect of this disclosure, a method for task processing is proposed. The method includes: generating a set of candidate images based on a target image using a thermodynamic genetic algorithm (TDGA) model, wherein the TDGA model is configured to apply one or more of a predetermined set of image processing operations in each evolutionary process; and determining multiple enhanced images from the set of candidate images based on the free energy of the candidate images, wherein the multiple enhanced images are determined to have the same classification as the target image.

[0006] According to a second aspect of this disclosure, an electronic device is proposed. The device includes: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the device to perform an action comprising: generating a set of candidate images based on a target image using a thermodynamic genetic algorithm (TDGA) model, wherein the TDGA model is configured to apply one or more of a predetermined set of image processing operations in each evolutionary process; and determining a plurality of enhanced images from the set of candidate images based on the free energy of the set of candidate images, the plurality of enhanced images being determined to have the same classification as the target image.

[0007] In a third aspect of this disclosure, a computer program product is provided. The computer program product is stored in a non-transitory computer storage medium and includes machine-executable instructions that, when executed in a device, cause the device to perform any step of the method described in the first aspect of this disclosure.

[0008] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or essential features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0009] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0010] Figure 1 A schematic diagram of an example environment in which embodiments of the present disclosure may be implemented is shown;

[0011] Figure 2 A flowchart of a data enhancement method according to some embodiments of the present disclosure is shown;

[0012] Figure 3 A schematic diagram illustrating the generation of dynamic saliency maps according to some embodiments of the present disclosure is shown; and

[0013] Figure 4 A block diagram of an example device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0014] Preferred embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0015] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0016] As mentioned earlier, the performance of machine learning models is closely related to the size and quality of the training dataset. In some fields, it is often difficult to obtain a sufficient amount of training data. Therefore, how to effectively augment data (also known as data augmentation) has become a focus of attention.

[0017] According to embodiments of this disclosure, a data augmentation scheme is provided. In this scheme, a set of candidate images is generated based on a target image using a thermodynamic genetic algorithm (TDGA) model, wherein the TDGA model is configured to apply one or more of a predetermined set of image processing operations in each evolutionary process; and multiple augmented images are determined from the set of candidate images based on the free energy of the candidate images, the multiple augmented images being determined to have the same classification as the target image.

[0018] Based on this approach, the embodiments of this disclosure can effectively utilize the crossover and mutation processes in genetic algorithms to efficiently achieve data augmentation.

[0019] The present disclosure will now be described with reference to the accompanying drawings.

[0020] Figure 1 An example environment 100 in which embodiments of this disclosure may be executed is shown. For example... Figure 1 As shown, the environment 100 may include a computing device 120. The computing device 120 can acquire a target image 110 and use a Thermodynamical Genetic Algorithm (TDGA) model 130 to generate multiple enhanced images 140 based on the target image 110.

[0021] Genetic Algorithm (GA) is an optimization method based on the evolutionary process of a system. It has been used to solve discrete optimization problems because it does not require prior knowledge of the reward or differential gradient. However, GA suffers from a phenomenon known as premature convergence. In other words, population diversity is often lost in the early stages of the search. Unlike traditional GA, TDGA addresses this premature convergence problem by incorporating the concepts of temperature and entropy into the selection rules. Details regarding the generation of multiple enhanced images will be described in detail below.

[0022] Figure 2 A flowchart of a data enhancement process 200 according to some embodiments of the present disclosure is shown. Process 200 may, for example, be performed by... Figure 1 The computing device 120 shown is used to implement this.

[0023] like Figure 2 As shown in box 202, computing device 120 uses thermodynamic genetic algorithm TDGA model 130 to generate a set of candidate images based on target image 110, wherein the TDGA model is configured to apply one or more of a set of predetermined image processing operations in each evolution process.

[0024] In some embodiments, computing device 120 may acquire a trained TDGA model 130 and utilize the evolutionary process in TDGA model 130 to generate a set of candidate images based on target image 110.

[0025] Specifically, in the evolutionary process of TDGA model 130, similar to the simulated hottest algorithm, a temperature parameter is introduced to avoid the loss of population diversity. During the evolutionary process, TDGA can add a predetermined number of individuals with the minimum free energy in each evolution to the next generation to achieve the goal of minimizing the population free energy.

[0026] In some embodiments, during the process of generating a set of candidate images using the TDGA model 130, each evolutionary process can perform one or more of a set of predetermined image processing operations on the individual.

[0027] For example, the set of predetermined image processing operations may include sixteen image operations, such as automatic compression, equalization, inversion, rotation, positioning, exposure, exposure increase, color, contrast, brightness, sharpness, shear X, shear Y, cutout, translation X and translation Y.

[0028] Furthermore, for each image processing operation, it can be set with an amplitude M and a probability P of being selected. In some embodiments, the amplitude M can be, for example, a value between 0 and 30. The probability P can be expressed as P = m / L, where m is a positive integer between 1 and 5, and L represents the chromosome length.

[0029] Based on this approach, the search space automatically requested by TDGA is approximately 31×5×10×2. 16 ≈1.0×10 8 It is much smaller than 10 in the automatic reinforcement algorithm. 32 The size of the search space.

[0030] In box 204, computing device 120 determines multiple enhanced images 140 from a set of candidate images based on the free energy of the set of candidate images, wherein the multiple enhanced images 140 are determined to have the same classification as the target image 110.

[0031] In some embodiments, the TDGA model can be trained based on a training dataset D. During training, D can be divided into a training set and a validation set. The free energy of each candidate image can be determined, for example, based on the validation set. Specifically, the accuracy of the TDGA model trained on the training set in processing the validation set can be regarded as the goodness of fit of the individual, and its negative value can be regarded as the free energy of the individual.

[0032] In some embodiments, computing device 120 selects a predetermined number of candidate images with minimum free energy from a set of candidate images to serve as a plurality of enhanced images 140.

[0033] Based on this approach, embodiments of this disclosure can fully utilize various transformations in genetic algorithms to efficiently construct enhanced images.

[0034] In some embodiments, the constructed augmented images can also be used for contrastive learning. Contrastive learning aims to transform samples with the same classification into similar feature representations, while transforming samples with different classifications into dissimilar feature representations.

[0035] In some embodiments, given that the enhanced image generated based on TDGA always has the same category as the target image, the computing device 120 may further construct a training image set based on the target image and a set of candidate images, and train a contrastive learning model based on the training image set, wherein the contrastive learning model is configured to minimize the distance between feature representations corresponding to image samples of the same type, and maximize the minimization of the distance between feature representations corresponding to image samples of different types.

[0036] In this way, embodiments of the present disclosure can improve the quality of contrastive learning models, enabling them to better distinguish between images of the same category and images of different categories.

[0037] In some embodiments, the trained contrastive learning model can also be used to process image-based object detection tasks. Specifically, the computing device 120 can receive a set of input images and utilize the contrastive learning model to determine a set of feature representations for the set of input images. Further, the computing device 120 can also detect objects in the set of input images based on the set of feature representations.

[0038] In some embodiments, the computing device 120 may rely on a dynamic saliency map (DSM) to perform object detection in an image. Specifically, the computing device 120 may construct a dynamic saliency map based on the set of feature representations. Further, the computing device 120 may determine objects in the set of input images based on the dynamic saliency map.

[0039] In some embodiments, the computing device 120 may also utilize a genetic algorithm to optimize the generation process of the dynamic saliency map (DSM) during object detection. The following will refer to... Figure 3 To describe the generation process of DSM, Figure 3 A schematic diagram 300 illustrating the generation of a DSM according to some embodiments of the present disclosure is shown.

[0040] like Figure 3 As shown, unlike the traditional DSM generation process, the computing device 120 can use the GA model 310 to weight the central periphery difference (CSD) map. Specifically, the computing device 120 can determine a set of central periphery difference (CSD) maps based on the set of feature representations, and determine the weights w used to weight the set of CSD maps based on the genetic algorithm GA model 310.

[0041] For GA model 310, its fitness function can be determined as follows:

[0042] fitness function=E(1-overlapratio) (I)

[0043] The overlap ratio can be determined as:

[0044]

[0045] Where A T A represents the truth region of the target object. D Indicates the area defined using DSM, A C This represents the region where the two overlap. The GA model 310 aims to minimize the fitness function as defined in Equation (8).

[0046] In some embodiments, such as Figure 4 As shown, computing device 120 can also construct a dynamic saliency map, also known as genetic DSM 320, based on a weighted set of CSD graphs.

[0047] In some embodiments, conventional DSMs use the maximum entropy between different consecutive frames within a given time period to analyze the dynamic characteristics of the saliency map. However, entropy is calculated based on a histogram of pixel values ​​corresponding to local regions, which is complex and computationally intensive, making it unsuitable for embedded and real-time systems.

[0048] In some embodiments, to reduce computational load, the computing device 120 may also determine the dynamic characteristics associated with the dynamic saliency map based on the variance of pixel values ​​in a local region, without relying on conventional entropy calculations. For example, if the variance of a pixel is greater than a predetermined threshold, the pixel may be identified as foreground.

[0049] Based on this approach, the embodiments of this disclosure can further reduce computational overhead, thereby improving computational efficiency.

[0050] Figure 4A schematic block diagram of an example device 400 that can be used to implement embodiments of the present disclosure is shown. For example, a computing device 120 according to an embodiment of the present disclosure can be implemented by device 400. As shown, device 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 402 or loaded from storage unit 408 into random access memory (RAM) 403. Various programs and data required for the operation of device 400 may also be stored in RAM 403. CPU 401, ROM 402, and RAM 403 are interconnected via bus 404. Input / output (I / O) interface 405 is also connected to bus 404.

[0051] Multiple components in device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0052] The various processes and handling described above, such as process 200 and / or process 300, may be executed by processing unit 401. For example, in some embodiments, process 200 may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by CPU 401, one or more actions of process 200 described above may be performed.

[0053] This disclosure can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0054] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0055] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0056] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0057] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0058] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0059] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0060] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0061] Various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for data augmentation, comprising: Using a first instance of a genetic algorithm based on a thermodynamic genetic algorithm TDGA model, a set of candidate images is generated based on a target image, wherein the TDGA model is configured to apply one or more of a set of predetermined image processing operations in each evolution process; Based on the free energy of the set of candidate images, multiple enhanced images are determined from the set of candidate images, and the multiple enhanced images are determined to have the same classification as the target image; Using a contrastive learning model, based on the multiple enhanced images, at least corresponding first feature representations and second feature representations of first and second different types are generated; Using a second instance of a genetic algorithm, the corresponding weights of the first feature representation and the second feature representation for corresponding first and second different types are determined; and At least one saliency map is generated based on the first feature representation and the second feature representation and their corresponding weights.

2. The method according to claim 1, further comprising: A training image set is constructed based on the target image and the set of candidate images; as well as The contrastive learning model is trained based on the training image set. The contrastive learning model is configured to minimize the distance between feature representations corresponding to image samples of the same type and maximize the distance between feature representations corresponding to image samples of different types.

3. The method according to claim 2, further comprising: Receive a set of input images; Using the contrastive learning model, a set of feature representations for the set of input images is determined; as well as Based on the set of feature representations, objects in the set of input images are detected.

4. The method according to claim 3, wherein detecting objects in the set of input images based on the set of feature representations comprises: Based on the aforementioned set of feature representations, a dynamic saliency map is constructed; as well as Based on the dynamic saliency map, the objects in the set of input images are determined.

5. The method according to claim 4, wherein constructing a dynamic saliency map based on the set of feature representations comprises: Based on the aforementioned set of feature representations, a set of central periphery difference (CSD) maps is determined; Based on the genetic algorithm model, the weights used to weight the set of CSD graphs are determined; as well as Based on the weighted set of CSD graphs, the dynamic saliency graph is constructed.

6. The method of claim 5, wherein the dynamic characteristics associated with the dynamic saliency map are determined based on the variance of pixel values ​​in a local region.

7. An electronic device, comprising: At least one processing unit; At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform an action, the action including: Using a first instance of a genetic algorithm based on a thermodynamic genetic algorithm TDGA model, a set of candidate images is generated based on a target image, wherein the TDGA model is configured to apply one or more of a set of predetermined image processing operations in each evolution process; Based on the free energy of the set of candidate images, multiple enhanced images are determined from the set of candidate images, and the multiple enhanced images are determined to have the same classification as the target image; Using a contrastive learning model, based on the multiple enhanced images, at least corresponding first feature representations and second feature representations of first and second different types are generated; Using a second instance of a genetic algorithm, the corresponding weights of the first feature representation and the second feature representation for corresponding first and second different types are determined; and At least one saliency map is generated based on the first feature representation and the second feature representation and their corresponding weights.

8. The electronic device according to claim 7, wherein the action further includes: A training image set is constructed based on the target image and the set of candidate images; as well as The contrastive learning model is trained based on the training image set. The contrastive learning model is configured to minimize the distance between feature representations corresponding to image samples of the same type and maximize the distance between feature representations corresponding to image samples of different types.

9. The electronic device according to claim 8, wherein the action further includes: Receive a set of input images; Using the contrastive learning model, a set of feature representations for the set of input images is determined; as well as Based on the set of feature representations, objects in the set of input images are detected.

10. The electronic device of claim 9, wherein detecting an object in the set of input images based on the set of feature representations comprises: Based on the aforementioned set of feature representations, a dynamic saliency map is constructed; as well as Based on the dynamic saliency map, the objects in the set of input images are determined.

11. The electronic device of claim 10, wherein constructing a dynamic saliency map based on the set of feature representations comprises: Based on the aforementioned set of feature representations, a set of central periphery difference (CSD) maps is determined; Based on the genetic algorithm model, the weights used to weight the set of CSD graphs are determined; as well as Based on the weighted set of CSD graphs, the dynamic saliency graph is constructed.

12. The electronic device of claim 11, wherein the dynamic characteristics associated with the dynamic saliency map are determined based on the variance of pixel values ​​in a local region.

13. A computer program product stored in a non-transitory computer storage medium and comprising machine-executable instructions that, when executed in a device, cause the device to perform actions, the actions including: Using a first instance of a genetic algorithm based on a thermodynamic genetic algorithm TDGA model, a set of candidate images is generated based on a target image, wherein the TDGA model is configured to apply one or more of a set of predetermined image processing operations in each evolution process; Based on the free energy of the set of candidate images, multiple enhanced images are determined from the set of candidate images, and the multiple enhanced images are determined to have the same classification as the target image; Using a contrastive learning model, based on the multiple enhanced images, at least corresponding first feature representations and second feature representations of first and second different types are generated; Using a second instance of a genetic algorithm, the corresponding weights of the first feature representation and the second feature representation for corresponding first and second different types are determined; and At least one saliency map is generated based on the first feature representation and the second feature representation and their corresponding weights.

14. The computer program product according to claim 13, wherein the action further includes: A training image set is constructed based on the target image and the set of candidate images; as well as The contrastive learning model is trained based on the training image set. The contrastive learning model is configured to minimize the distance between feature representations corresponding to image samples of the same type and maximize the distance between feature representations corresponding to image samples of different types.

15. The computer program product according to claim 14, wherein the action further includes: Receive a set of input images; Using the contrastive learning model, a set of feature representations for the set of input images is determined; as well as Based on the set of feature representations, objects in the set of input images are detected.

16. The computer program product of claim 15, wherein detecting an object in the set of input images based on the set of feature representations comprises: Based on the aforementioned set of feature representations, a dynamic saliency map is constructed; as well as Based on the dynamic saliency map, the objects in the set of input images are determined.

17. The computer program product of claim 16, wherein constructing a dynamic saliency map based on the set of feature representations comprises: Based on the aforementioned set of feature representations, a set of central periphery difference (CSD) maps is determined; Based on the genetic algorithm model, the weights used to weight the set of CSD graphs are determined; as well as Based on the weighted set of CSD graphs, the dynamic saliency graph is constructed.

18. The computer program product of claim 17, wherein the dynamic characteristics associated with the dynamic saliency map are determined based on the variance of pixel values ​​in a local region.

Images