A method and system for expanding domain adaptation training sets
By performing Fourier transform and weighted fusion on images of different domains to generate extended images, the model performance limitations caused by the difference in multi-domain data in image machine learning are solved, and the model training performance and prediction effect are improved.
Patent Information
- Application Number
- CN202011034930.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-27
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-09-27
AI Technical Summary
In image machine learning, the difference in multi-domain data leads to limited model performance. The existing technology expands the sample but has the problem of singularity of training set selection, making it difficult to collect balanced data sets, affecting the model prediction effect.
By acquiring the target images and interference images of different domains, fast Fourier transform is performed separately to obtain high-frequency and low-frequency information, and the fusion image is obtained through weighted fusion. The extended image is generated through inverse Fourier transform, which is used as the extended image of the training set. , improve the training performance of the model.
Through the generation of the diversity training set, the training performance of the model is significantly improved, so that the model can better adapt to data in different domains and improve the prediction effect.
Smart Images

Figure CN112116102B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of machine learning, and in particular to a method and system for expanding a domain adaptation training set. Background Art
[0002] In the field of image-based machine learning, there are often data from multiple domains, which can be specifically manifested as different camera types having different color features, different shooting qualities, different groups of people, etc. When using machine learning methods to learn data from multiple different domains, due to the inherent differences in these data, a large amount of data from different domains is required to converge to a better prediction effect. For example, images from different domains are generally required to be kept at the same order of magnitude, but in actual scenarios, it is difficult to collect a balanced data set, and the performance of the deep learning or machine learning model obtained by training based on the existing training set with different proportions is relatively limited. In the prior art, during training, the training set is expanded by expanding the collected samples, randomly flipping, cropping, color enhancing, etc., and there is a problem of the singleness of the training set selection. Summary of the invention
[0003] Therefore, the method and system for expanding a domain adaptation training set provided by the present invention overcome the defect of the singleness of the training set selection in the prior art.
[0004] In order to achieve the above object, the present invention provides the following technical solutions:
[0005] In a first aspect, an embodiment of the present invention provides a method for expanding a domain adaptation training set, including:
[0006] Acquire a first image and a second image from different domains in a training set, where the first image is a target image trained in the training set, and the second image is an interference image of the target image trained in the training set;
[0007] Performing fast Fourier transform on the first image and the second image respectively to obtain first high-frequency information and first low-frequency information of the first image and second high-frequency information and second low-frequency information of the second image;
[0008] Assigning different weights to the first low-frequency information and the second low-frequency information and performing weighted fusion to obtain low-frequency information of a fused image;
[0009] The low-frequency information of the fused image and the second high-frequency information are transformed by inverse Fourier transform to obtain a fused image as an extended image of the training set.
[0010] In one embodiment, all images in different domains in the training set are fused, and the different domains include at least two domains.
[0011] In one embodiment, the first high-frequency information and the first low-frequency information use a first preset threshold as a dividing line, and the second high-frequency information and the second low-frequency information use a second preset threshold as a dividing line.
[0012] In one embodiment, the low-frequency information of the fused image is calculated by the following formula:
[0013]
[0014] in, To fuse the low-frequency information of the image, is the first low-frequency information, is the second low-frequency information, and λ obeys the uniform distribution U(0,1).
[0015] In one embodiment, the fused image is obtained by calculating the inverse Fourier transform using the following formula:
[0016]
[0017] Among them, x mixed To fuse the images, It is the second high frequency information.
[0018] In one embodiment, the first image and the second image are grayscale images or RGB three-channel images.
[0019] In a second aspect, an embodiment of the present invention provides a system for expanding a domain adaptation training set, including:
[0020] An image acquisition module is used to acquire a first image and a second image from different domains in a training set, wherein the first image is a target image trained in the training set, and the second image is an interference image of the target image trained in the training set;
[0021] A Fourier transform module, used to perform fast Fourier transform on the first image and the second image respectively, to obtain first high-frequency information and first low-frequency information of the first image and second high-frequency information and second low-frequency information of the second image;
[0022] A weight matching module is used to assign different weights to the first low-frequency information and the second low-frequency information and perform weighted fusion to obtain low-frequency information of a fused image;
[0023] The image expansion module is used to obtain a fused image by performing inverse Fourier transform on the low-frequency information of the fused image and the second high-frequency information as an expanded image of the training set.
[0024] In a third aspect, an embodiment of the present invention provides a terminal, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the method for expanding the domain adaptation training set described in the first aspect of the embodiment of the present invention.
[0025] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for expanding the domain adaptation training set described in the first aspect of the embodiment of the present invention.
[0026] The technical solution of the present invention has the following advantages:
[0027] The domain adaptation training set expansion method and system provided by the present invention obtains the first image and the second image from different domains in the training set, wherein the first image is the target image trained in the training set, and the second image is the interference image of the target image trained in the training set; performs fast Fourier transform on the first image and the second image respectively, obtains the first high-frequency information and the first low-frequency information of the first image and the second high-frequency information and the second low-frequency information of the second image; assigns different weights to the first low-frequency information and the second low-frequency information and performs weighted fusion to obtain the low-frequency information of the fused image; uses the low-frequency information of the fused image and the second high-frequency information to obtain the fused image through inverse Fourier transform as the expanded image of the training set. The domain adaptation training set expansion method provided by the present invention makes the images of the training set diverse, and improves the training performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS
[0028] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0029] Figure 1 A flowchart of a specific example of a method for expanding a domain adaptation training set provided by an embodiment of the present invention;
[0030] Figure 2 A module composition diagram of a system for expanding a domain adaptation training set provided by an embodiment of the present invention;
[0031] Figure 3 A composition diagram of a specific example of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0032] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0033] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.
[0034] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0035] Example 1
[0036] An embodiment of the present invention provides a method for expanding a domain adaptation training set, such as Figure 1 As shown, the following steps are included:
[0037] Step S1: Acquire a first image and a second image from different domains in a training set, wherein the first image is a target image trained in the training set, and the second image is an interference image of the target image trained in the training set.
[0038] In an embodiment of the present invention, a first image and a second image from different domains in a training set are obtained, the first image and the second image are grayscale images or RGB three-channel images, and the first image and the second image are selected accordingly according to actual needs, wherein the first image is a target image trained in the training set, and the second image is an interference image of the target image trained in the training set. For example: when the training target of the training model is to identify the category of fruit, at this time, the image representing the category of fruit is the target image of the training set, and the image outside the category of fruit is the interference image. This is only an example and is not limited to this. In actual applications, the target image is changed accordingly according to the training target of the training model.
[0039] Step S2: performing fast Fourier transform on the first image and the second image respectively to obtain first high-frequency information and first low-frequency information of the first image and second high-frequency information and second low-frequency information of the second image.
[0040] In practical applications, the frequency of an image is an indicator of how drastic the grayscale changes in the image. It is the gradient of the grayscale in the plane space. For example, a large area of desert is an area with slow grayscale changes in the image, and the corresponding frequency value is very low; while the edge area with drastic changes in surface properties is an area with drastic grayscale changes in the image, and the corresponding frequency value is higher. Fourier transform is to convert an image from the spatial domain to the frequency domain, and its inverse transform is to convert an image from the frequency domain to the spatial domain. Fourier transform is to transform the grayscale distribution function of an image into the frequency distribution function of the image, and reflect the grayscale distribution of the image through the frequency distribution.
[0041] In the embodiment of the present invention, the first high-frequency information and the first low-frequency information use the first preset threshold as the dividing line, and the second high-frequency information and the second low-frequency information use the second preset threshold as the dividing line. It should be noted that the numerical selection of the first preset threshold and the second preset threshold is reasonably set according to actual needs, and the specific numerical values of the two thresholds can be the same or different, and there is no limitation here.
[0042] Step S3: assigning different weights to the first low-frequency information and the second low-frequency information and performing weighted fusion to obtain low-frequency information of a fused image.
[0043] In the embodiment of the present invention, the low-frequency information of the fused image is calculated by the following formula:
[0044]
[0045] in, To fuse the low-frequency information of the image, is the first low-frequency information, is the second low-frequency information, and λ obeys the uniform distribution U(0,1). In practice, low-frequency information and high-frequency information can reflect the severity of image changes in the spatiotemporal domain, and low-frequency information reflects the overall information of the image. Therefore, the low-frequency information in the first image and the low-frequency information in the second image both contain certain feature information that will affect the training result. The embodiment of the present invention extracts the low-frequency information of the fused image by weight matching.
[0046] Step S4: The low-frequency information of the fused image and the second high-frequency information are subjected to inverse Fourier transform to obtain a fused image as an extended image of the training set.
[0047] In the embodiment of the present invention, the fused image is obtained by calculating the inverse Fourier transform using the following formula:
[0048]
[0049] Among them, x mixed To fuse the images, is the second high-frequency information; in practice, the high-frequency information reflects the mutation information of the image. In the embodiment of the present invention, the first image is the target image for training in the training set, and the second image is the interference image of the target image for training in the training set. The high-frequency information of the first image often does not include or includes relatively small mutation information, which will not have a significant impact on the training result. The second image, as the interference image of the target image for training, has high-frequency information that includes information about drastic image changes. Therefore, the second high-frequency information of the second image and the low-frequency information of the fused image are selected to obtain a fused image through inverse Fourier transform as an extended image of the training set, so that the images of the training set have diversity, thereby improving the training performance of the model.
[0050] In a specific embodiment, all images in different domains in the training set are fused, and the different domains include at least two domains as extended images of the training set, so that the images of the training set have diversity.
[0051] In an embodiment of the present invention, in a machine learning or deep learning model training process, for an image in a training set, assuming there is a source domain image and a target domain image, and the two images may be from different domains, the following processing is performed:
[0052] The source domain image is transformed into the frequency domain using the following formula:
[0053]
[0054] Among them, x source It is a source domain RGB three-channel image. is a two-dimensional Fourier transform.
[0055] The high-frequency and low-frequency information of the source domain image is extracted using the following formula:
[0056]
[0057] in, They are the low-frequency information and high-frequency information of the Fourier transform of the image, r 1 is a given threshold used to divide high-frequency information and low-frequency information. For the two-dimensional Fourier transform, for the general deep learning input image size of 299*299, r 1 Generally, it is set between 10 and 15. At this time, the domain background color of the image can be better extracted. Using the domain background color at this time can improve the adaptability of the model in this domain. This is just an example and is not limited to this. In actual applications, r 1Set the corresponding value.
[0058] The target domain image is transformed into the frequency domain using the following formula:
[0059]
[0060] Among them, x target It is a RGB three-channel image of the target domain.
[0061] The high-frequency and low-frequency information of the target domain image is extracted using the following formula:
[0062]
[0063] in They are the low-frequency information and high-frequency information of the Fourier transform of the target domain image, r 2 is a given threshold, used to divide the high-frequency and low-frequency components, r 1 、r 2 The specific values may be the same or different and are not limited here.
[0064] Fusion of images by weighted fusion:
[0065]
[0066] Here, λ is a random variable that obeys uniform distribution and has a value range between 0 and 1, which can be specifically expressed as λ~U(0,1). The embodiment of the present invention extracts low-frequency information of the fused image by weight matching.
[0067] The fused image is obtained by inverse Fourier transform using the following formula:
[0068]
[0069] Among them, x mixed To fuse the images, The high-frequency information of the target domain image is the Fourier transform of the target domain image. The high-frequency information of the target domain image contains information about drastic image transformations. Therefore, the high-frequency information of the target domain image and the low-frequency information of the fused image are selected and the fused image is obtained through inverse Fourier transform as the extended image of the training set, which makes the images of the training set more diverse and improves the training performance of the model.
[0070] All images in different domains in the training set are fused, and different domains contain at least two domains as extended images of the training set, so that the images of the training set have diversity. In the expansion of the training set, there is no need to annotate the images of the target domain. Using very few target domain pictures can adapt to the image colors of the target domain, which significantly improves the training performance of the model.
[0071] The method for expanding a domain adaptation training set provided in an embodiment of the present invention comprises obtaining a first image and a second image from different domains in a training set, wherein the first image is a target image for training in the training set, and the second image is an interference image of the target image for training in the training set; performing fast Fourier transform on the first image and the second image, respectively, to obtain first high-frequency information and first low-frequency information of the first image and second high-frequency information and second low-frequency information of the second image; assigning different weights to the first low-frequency information and the second low-frequency information and performing weighted fusion to obtain low-frequency information of a fused image; and performing inverse Fourier transform on the low-frequency information of the fused image and the second high-frequency information to obtain a fused image as an expanded image of the training set. The method for expanding a domain adaptation training set provided in an embodiment of the present invention makes the images of the training set diverse, thereby improving the training performance of the model.
[0072] Example 2
[0073] The embodiment of the present invention provides a system for expanding a domain adaptation training set, such as Figure 2 As shown, including:
[0074] Image acquisition module 1 is used to acquire a first image and a second image from different domains in a training set, wherein the first image is a target image trained in the training set, and the second image is an interference image of the target image trained in the training set; this module executes the method described in step S1 of embodiment 1, which will not be repeated here.
[0075] The Fourier transform module 2 is used to perform fast Fourier transform on the first image and the second image respectively to obtain the first high-frequency information and the first low-frequency information of the first image and the second high-frequency information and the second low-frequency information of the second image; this module executes the method described in step S2 of embodiment 1, which will not be repeated here.
[0076] The weight matching module 3 is used to assign different weights to the first low-frequency information and the second low-frequency information and perform weighted fusion to obtain the low-frequency information of the fused image; this module executes the method described in step S3 of embodiment 1, which will not be repeated here.
[0077] The image expansion module 4 is used to obtain a fused image by inverse Fourier transforming the low-frequency information of the fused image and the second high-frequency information as an extended image of the training set; this module executes the method described in step S4 of Example 1 and will not be repeated here.
[0078] The embodiment of the present invention provides an expansion system for a domain adaptation training set. The first image and the second image from different domains in the training set are obtained through an image acquisition module, wherein the first image is a target image for training in the training set, and the second image is an interference image of the target image for training in the training set; the first image and the second image are respectively subjected to fast Fourier transformation in a Fourier transformation module to obtain first high-frequency information and first low-frequency information of the first image and second high-frequency information and second low-frequency information of the second image; the first low-frequency information and the second low-frequency information are assigned different weights in a weight matching module and weighted fused to obtain low-frequency information of a fused image; the low-frequency information of the fused image and the second high-frequency information are subjected to inverse Fourier transformation to obtain a fused image as an expansion image of the training set. The expansion system for a domain adaptation training set provided by the embodiment of the present invention makes the images of the training set diverse, thereby improving the training performance of the model.
[0079] Example 3
[0080] An embodiment of the present invention provides a terminal, such as Figure 3As shown, it includes: at least one processor 401, such as a CPU (Central Processing Unit), at least one communication interface 403, a memory 404, and at least one communication bus 402. Among them, the communication bus 402 is used to realize the connection and communication between these components. Among them, the communication interface 403 may include a display screen (Display), a keyboard (Keyboard), and the optional communication interface 403 may also include a standard wired interface and a wireless interface. The memory 404 may be a high-speed RAM memory (Random Access Memory, volatile random access memory) or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 404 may also be at least one storage device located away from the aforementioned processor 401. Among them, the processor 401 can execute the domain adaptation training set expansion method in Example 1. A set of program codes are stored in the memory 404, and the processor 401 calls the program code stored in the memory 404 to execute the domain adaptation training set expansion method in Example 1. The communication bus 402 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The communication bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one line is used to represent it, but it does not mean that there is only one bus or one type of bus. Among them, the memory 404 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 404 may also include a combination of the above types of memory. Among them, the processor 401 may be a central processing unit (English: central processing unit, abbreviated: CPU), a network processor (English: network processor, abbreviated: NP) or a combination of CPU and NP.
[0081] Among them, the memory 404 may include a volatile memory (English: volatile memory), such as a random access memory (English: random-access memory, abbreviated: RAM); the memory may also include a non-volatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk drive (English: hard disk drive, abbreviated: HDD) or a solid-state drive (English: solid-state drive, abbreviated: SSD); the memory 404 may also include a combination of the above types of memory.
[0082] The processor 401 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP.
[0083] The processor 401 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0084] Optionally, the memory 404 is also used to store program instructions. The processor 401 can call the program instructions to implement the method for expanding the domain adaptation training set in the implementation example 1 of the present application.
[0085] The embodiment of the present invention also provides a computer-readable storage medium, on which computer-executable instructions are stored, and the computer-executable instructions can execute the method for expanding the domain adaptation training set in Example 1. The storage medium may be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memory.
[0086] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from these are still within the scope of protection of the invention.
Claims
1. A method for expanding a domain adaptation training set, characterized in that: include: Acquire a first image and a second image from different domains in a training set, where the first image is a target image trained in the training set, and the second image is an interference image of the target image trained in the training set; Performing fast Fourier transform on the first image and the second image respectively to obtain first high-frequency information and first low-frequency information of the first image and second high-frequency information and second low-frequency information of the second image, wherein the first high-frequency information and the first low-frequency information use a first preset threshold as a dividing line, and the second high-frequency information and the second low-frequency information use a second preset threshold as a dividing line; Assigning different weights to the first low-frequency information and the second low-frequency information and performing weighted fusion to obtain low-frequency information of a fused image; The low-frequency information of the fused image is calculated by the following formula: in, To fuse the low-frequency information of the image, is the first low-frequency information, is the second low frequency information, Obeys uniform distribution U(0, 1); The low-frequency information of the fused image and the second high-frequency information are transformed by inverse Fourier transform to obtain a fused image as an extended image of the training set.
2. The method for expanding a domain adaptation training set according to claim 1, characterized in that: All images in different domains in the training set are fused, and the different domains include at least two domains.
3. The method for expanding a domain adaptation training set according to claim 1, characterized in that: The fused image is obtained by calculating the inverse Fourier transform using the following formula: in, To fuse the images, It is the second high frequency information.
4. The method for expanding a domain adaptation training set according to claim 3, characterized in that: The first image and the second image are grayscale images or RGB three-channel images.
5. A system for expanding a domain adaptation training set, characterized in that: include: An image acquisition module is used to acquire a first image and a second image from different domains in a training set, wherein the first image is a target image trained in the training set, and the second image is an interference image of the target image trained in the training set; A Fourier transform module, used to perform fast Fourier transform on the first image and the second image respectively, to obtain first high-frequency information and first low-frequency information of the first image and second high-frequency information and second low-frequency information of the second image, wherein the first high-frequency information and the first low-frequency information use a first preset threshold as a dividing line, and the second high-frequency information and the second low-frequency information use a second preset threshold as a dividing line; A weight matching module is used to assign different weights to the first low-frequency information and the second low-frequency information and perform weighted fusion to obtain low-frequency information of a fused image; The low-frequency information of the fused image is calculated by the following formula: in, To fuse the low-frequency information of the image, is the first low-frequency information, is the second low frequency information, Obeys uniform distribution U(0, 1); The image expansion module is used to obtain a fused image by performing inverse Fourier transform on the low-frequency information of the fused image and the second high-frequency information as an expanded image of the training set.
6. A terminal, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the method for expanding the domain adaptation training set described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method for expanding the domain adaptation training set according to any one of claims 1 to 4.
Citation Information
Patent Citations
Image fusing method based on multidirectional gradient field
CN102609930A
Multi-exposure image fusion method
CN104835130A
Image fusion method AND apparatus, computer device, and storage medium
CN109360182A
Image fusion method based on Fourier spectrum extraction
CN110503620A
Image super-resolution reconstruction model and method based on different frequency information
CN111461983A
