Training method, recognition method and device for cross-domain small sample recognition model
By replacing the amplitude features and attention matrix of the source domain image in the target domain image, the problem of insufficient generalization ability under small sample target domain data is solved, and better feature extraction and recognition accuracy is achieved.
Patent Information
- Application Number
- CN202411144032.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2044-08-20
AI Technical Summary
Existing target detection models lack generalization ability in the case of small sample target domain data, and face problems such as labeling difficulties, high costs, and domain offset.
By acquiring the amplitude features of source and target domain images, replacing the amplitude features of the source domain image with the amplitude features of the target domain image, and replacing the attention matrix during training, the model's ability to perceive information and extract features from the target domain is enhanced by using the hybrid domain image and feature matrix reconstruction.
It improves the model's generalization ability in the target domain, enhances feature extraction capabilities, reduces bias towards source domain features, and improves recognition accuracy.
Smart Images

Figure CN118675017B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image recognition technology, and in particular to a training method, a recognition method and a device for a cross-domain small sample recognition model. Background Art
[0002] Image data, as an intuitive medium for information transmission, plays an increasingly important role in today's society. Especially in the field of computer vision, how to efficiently extract useful information from massive images has always been one of the research focuses. As one of the important tasks of computer vision, target detection is challenging to accurately identify targets in images and mark their locations and categories. In recent years, with the increasing demand for target detection technology in fields such as intelligent transportation, intelligent manufacturing, and intelligent education, models not only need to cope with more diverse image information, but also need to cope with more complex application scenarios, which brings greater challenges to the generalization of tasks.
[0003] In practical applications, general object detection algorithms usually require a large amount of instance-level annotation information to train the model. However, this annotation process may face a variety of problems. First, due to the long-tail distribution characteristics of the real world, it is often difficult to obtain a sufficient amount of labeled data for some categories, such as rare animals and plants or rare diseases. Secondly, image annotation usually requires a lot of manpower, and the accuracy of the annotation cannot be guaranteed. Missing and mislabeling often occur, especially for some difficult-to-annotate objects. In addition, model training also requires a lot of resources, including expensive GPU equipment and professional domain knowledge. Finally, when the model is actually applied, the problem of domain shift is often encountered, that is, the difference in data distribution will lead to a decline in model performance.
[0004] When the target domain samples have less labeled information, existing mainstream object detection methods often fail to achieve satisfactory results. However, it is worth noting that even a child can learn new categories in different scenes by observing only a few images. Therefore, how to achieve object detection with a small number of target domain samples has become a practical problem. Summary of the invention
[0005] The purpose of this application is to provide a training method, recognition method and device for a cross-domain small sample recognition model to solve the technical problem of how to use source domain data and a small number of target domain samples to improve the generalization ability of the detection network in the target domain. The many technical effects that can be produced by the preferred technical solution among the many technical solutions provided in this application are described in detail below.
[0006] To achieve the above objectives, this application provides the following technical solutions:
[0007] The first aspect of the present application provides a method for training a cross-domain small sample recognition model, comprising: obtaining amplitude features of any source domain image and amplitude features of any target domain image, replacing the amplitude features of the source domain image with the amplitude features of the target domain image, and obtaining a target-like domain image; using multiple source domain images, multiple target domain images, and multiple target-like domain images as training sets; using the training sets to train a recognition model, wherein in the process of training with the source domain images and the target-like domain images, the attention matrix of the source domain image and the attention matrix of the target-like domain image are replaced with the attention matrix of the target domain image to reconstruct the feature matrices of the source domain image and the target-like domain image.
[0008] In some embodiments, the training set also includes mixed domain images; the method also includes: randomly selecting at least two images from the multiple source domain images, the multiple target domain images, and the multiple target-like domain images, performing weighted summation, and splicing the labels of the at least two images to obtain the mixed domain image.
[0009] In some embodiments, the method further includes: selecting at least one image area on any of the source domain images and / or any of the target domain images and / or any of the target-like domain images, overlaying the at least one image area on any of the source domain images and / or any of the target domain images and / or any of the target-like domain images, to obtain the mixed domain image.
[0010] In some embodiments, the acquiring of the amplitude features of any of the source domain images and the amplitude features of any of the target domain images includes: acquiring frequency domain features of the source domain image and the target domain image by Fourier transform, and separating the amplitude features and phase features from the frequency domain features.
[0011] In some embodiments, obtaining the target-domain image-like includes: retaining the phase feature of the source domain image; and performing an inverse Fourier transform on the source domain image retaining the phase feature to obtain the target-domain image-like.
[0012] In some embodiments, the inverse Fourier transform is defined as:
[0013] ,
[0014] in, represents the Fourier transform, represents the mask matrix, represents the source domain image, represents the target domain image, represents the amplitude feature of the target domain image, represents the amplitude feature of the source domain image, represents the phase feature of the source domain image.
[0015] In some embodiments, the process of training using the source domain image and the target domain image includes: obtaining the feature matrix of the source domain image and the target domain image; processing the feature matrix to obtain the attention matrix, wherein the attention matrix includes a query matrix, a key matrix and a value matrix; and performing attention calculation based on the attention matrix.
[0016] In some embodiments, performing attention calculation based on the attention matrix includes: replacing the key matrix of the source domain image and / or the key matrix of the target domain image with the key matrix of the target domain image.
[0017] The second aspect of the present application provides a recognition method, including: obtaining multiple cross-domain sample images; inputting the cross-domain sample images into a recognition model to obtain recognition results of the cross-domain sample images, wherein the recognition model is a model trained according to the training method of the cross-domain small sample recognition model as described above.
[0018] The third aspect of the present application provides a processing device, comprising: one or more processors; a memory for storing one or more computer programs, and one or more of the processors are used to execute the one or more computer programs stored in the memory, so that the one or more processors execute the training method of the cross-domain small sample recognition model as described above and the recognition method as described above.
[0019] Implementing one of the above-mentioned technical solutions of the present application has the following advantages or beneficial effects: since it is often difficult or costly to obtain labeled data in the field of small samples, in the present application, by replacing the amplitude features of the source domain image with the amplitude features of the target domain image, the low-frequency information of the source domain image and the target domain image can be exchanged, and the information perception of the target domain can be realized, thereby enhancing the expressiveness of the source domain image and better enriching the sample features; secondly, in the process of training with source domain images and target domain-like images, by replacing the attention matrix of the source domain image and the attention matrix of the target domain-like image with the attention matrix of the target domain image, the recognition model can automatically pay attention to the features of the target domain and activate the features in the source domain that are similar to the target domain, so that the features extracted by the network will not be overly biased towards the source domain, which helps to improve the feature extraction ability of the recognition model. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. It is obvious that the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0021] Figure 1 It is a flowchart of a method for training a cross-domain small sample recognition model according to an embodiment of the present application;
[0022] Figure 2 is a schematic diagram of obtaining a target domain image in an embodiment of the present application;
[0023] Figure 3 is a schematic diagram of obtaining a mixed domain image in an embodiment of the present application;
[0024] Figure 4 is a schematic diagram of the network structure of an embodiment of the present application;
[0025] Figure 5 is a schematic diagram of Cityscapes and Foggy Cityscapes data of an embodiment of the present application;
[0026] Figure 6 It is a schematic diagram of the experimental results of the embodiment of the present application;
[0027] Figure 7 is a flowchart of the identification method of an embodiment of the present application;
[0028] Figure 8 It is a schematic diagram of a processing device according to an embodiment of the present application.
[0029] In the figure: 1. processing device; 10. memory; 11. processor. DETAILED DESCRIPTION
[0030] In order to make the purpose, technical solutions and advantages of the present application clearer, the various exemplary embodiments to be described below will refer to the corresponding drawings, which constitute a part of the exemplary embodiments, wherein various exemplary embodiments that may be used to implement the present application are described. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present disclosure. It should be understood that they are only examples of processes, methods, and devices that are consistent with some aspects disclosed in the present application as detailed in the attached claims, and other embodiments may also be used, or the embodiments listed herein may be modified in structure and function without departing from the scope and essence of the present application.
[0031] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", etc. indicate the orientation or positional relationship based on the drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the elements referred to must have a specific orientation, be constructed and operated in a specific orientation. The terms "first", "second", etc. are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. The term "multiple" means two or more. The terms "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, an integral connection, a mechanical connection, an electrical connection, a communication connection, a direct connection, an indirect connection through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more related listed items. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.
[0032] In order to illustrate the technical solution described in the present application, a specific embodiment is provided below, and only the parts related to the embodiment of the present application are shown.
[0033] like Figure 1 As shown, the present application provides a method for training a cross-domain small sample recognition model, including steps S1-S3.
[0034] S1. Obtain the amplitude features of any source domain image and the amplitude features of any target domain image, replace the amplitude features of the source domain image with the amplitude features of the target domain image, and obtain a target domain-like image.
[0035] In some embodiments, cross-domain small samples may refer to the ability of a model to quickly adapt and complete tasks using limited data samples when faced with new tasks or domains that are different from the distribution of training data. That is, the target domain images may have a different distribution from the source domain images, and the target domain images are more scarce than the source domain images. This application can be used in various application scenarios of cross-domain small samples.
[0036] In some embodiments, the source domain image may refer to original image data. For example, the source domain image may be at least one of an animal picture, a natural scene, a traffic scene, and a medical image. The source domain image may be labeled, for example, the label may indicate the type of animal, etc.
[0037] In some embodiments, the target domain image may be an image in a new dataset that needs to be adapted. For example, at least one of a zoo environment image, a seasonal change image of a natural scene, a real-time traffic monitoring image, and a medical image of a rare disease. The target domain image may have a different distribution from the source domain image. The target domain image may not have a label.
[0038] In some embodiments, obtaining the amplitude features of any source domain image and the amplitude features of any target domain image may include: obtaining frequency domain features of the source domain image and the target domain image through Fourier transform, and separating the amplitude features and the phase features from the frequency domain features.
[0039] In some embodiments, Figure 2 As shown, the amplitude feature can be global information of the image, such as image texture, color and other information. The phase feature can be local information of the image, such as image contour, shape and other information. In some embodiments, the image features of the source domain image and the image features of the target domain image can be converted into frequency domain features of the source domain image and the frequency domain features of the target domain image by Fourier transform.
[0040] In some embodiments, the Fourier transform may be defined as:
[0041] ,
[0042] in, Represents an image, that is, a source domain image or a target domain image, Representing images The corresponding row and column coordinates, Respectively represent the frequency coordinates in the frequency domain, Representing images The width of Representing images Height, is an imaginary unit, that is , is the base of natural logarithms, represents the Fourier transform, Represents the spectrum. This can reveal the amplitude and phase of various frequency components in the source domain image or the target domain image.
[0043] In some embodiments, obtaining a target-domain image includes: retaining a phase feature of a source-domain image; and performing an inverse Fourier transform on the source-domain image retaining the phase feature to obtain a target-domain image.
[0044] In some embodiments, the phase feature of the source domain image may be kept unchanged, while the amplitude feature of the source domain image may be replaced with the amplitude feature of the target domain image to reform the frequency domain feature of the source domain image.
[0045] In some embodiments, the frequency domain features re-formed from the source domain image can be converted into image features by inverse Fourier transform to obtain a target domain-like image. In this case, the target domain-like image can introduce global information of the target domain image, such as texture, illumination, etc., while retaining the original semantic information of the source domain image. In some embodiments, the target domain-like image can have the phase features of the source domain image and the amplitude features of the target domain image.
[0046] In some embodiments, the inverse Fourier transform may be defined as:
[0047] ,
[0048] in, represents the Fourier transform, represents the mask matrix, represents the source domain image, represents the target domain image, represents the amplitude feature of the target domain image, represents the amplitude feature of the source domain image, Represents the phase characteristics of the source domain image.
[0049] In some embodiments, the mask matrix It can be defined as:
[0050] ,
[0051] in, Represents the corresponding row and column coordinates of the image, Indicates the width of the image. Indicates the height of the image. It can be less than or equal to 0.15.
[0052] As mentioned above, the target-domain image can use the texture, color and other information of the target-domain image to represent the source-domain image, but at the same time does not change the semantic features of the source-domain image. In this case, the target-domain image can take into account the style of the target-domain image and make full use of the annotated data in the source domain, thereby better enriching the sample features of the image.
[0053] In some other embodiments, the amplitude feature of the target domain image may be replaced with the amplitude feature of the source domain image to obtain a quasi-target domain image. In some embodiments, the quasi-target domain image may have the phase feature of the target domain image and the amplitude feature of the source domain image.
[0054] In some other embodiments, the frequency domain features of the source domain image may be replaced by the frequency domain features of the target domain image.
[0055] S2. Use multiple source domain images, multiple target domain images, and multiple target-like domain images as training sets.
[0056] In some embodiments, the training set may include at least one source domain image, at least one target domain image, and at least one target-domain-like image.
[0057] In some embodiments, the training set may also include at least one mixed-domain image.
[0058] In some embodiments, Figure 3 As shown, the training method of the cross-domain small sample recognition model may also include: randomly extracting at least two images from multiple source domain images, multiple target domain images, and multiple target-like domain images for weighted summation, splicing the labels of the at least two images, and obtaining a mixed domain image. In this case, different images can be mixed, which can better highlight the edge features of the image, thereby enhancing the phase features of the image.
[0059] In some embodiments, two images may be randomly selected from a plurality of source domain images, a plurality of target domain images, and a plurality of quasi-target domain images for weighted summation. For example, a source domain image or a target domain image or a quasi-target domain image may be firstly selected, and then another source domain image or a target domain image or a quasi-target domain image may be selected.
[0060] In some embodiments, the weighted sum may be defined as:
[0061] ,
[0062] in, represents a mixed domain image, , represents image i and image j, represents the weight, .
[0063] In some embodiments, the concatenation of tags may be defined as:
[0064] ,
[0065] in, represents the label of the mixed domain image, represents the label of image i, represents the label of image j.
[0066] In some embodiments, multiple images may be randomly selected from multiple source domain images, multiple target domain images, and multiple target-like domain images for weighted summation, and labels of the multiple images may be spliced to obtain at least one mixed domain image. For example, multiple source domain images and / or multiple target domain images and / or multiple target-like domain images may be selected for weighted summation.
[0067] In some embodiments, Figure 3 As shown, the training method of the cross-domain small sample recognition model can also include: selecting at least one image area on any source domain image and / or any target domain image and / or any type of target domain image, overlaying at least one image area on any source domain image and / or any target domain image and / or any type of target domain image to obtain a mixed domain image.
[0068] In some embodiments, the image may have multiple image regions. The image may be segmented based on color, shape or texture to obtain multiple image regions. In some embodiments, the image may be segmented by at least one of threshold segmentation, edge detection, region growing and clustering algorithms to obtain multiple image regions.
[0069] In some embodiments, at least one image area can be selected from multiple source domain images or multiple target domain images or multiple quasi-target domain images, and then the at least one image area can be overlaid onto a source domain image or a target domain image or a quasi-target domain image to obtain a mixed domain image.
[0070] In some embodiments, at least one image region may be selected from a plurality of source domain images and / or a plurality of target-like domain images, and then the at least one image region may be overlaid onto a source domain image or a target-like domain image to obtain a mixed domain image.
[0071] In some embodiments, the image region may be overlaid to the same position of the image. For example, the image region is located at a front image, such as position A of the source domain image, and the image region is overlaid to a back image, such as the corresponding position A of the target domain image. In some embodiments, the image region may also be overlaid to different positions of the image.
[0072] In some embodiments, at least one image region may be selected from a source domain image and / or a target domain image and / or a quasi-target domain image. For example, three image regions may be selected from two source domain images and one target domain image. This is not limited here and may be selected according to actual needs.
[0073] S3. Use the training set to train the recognition model.
[0074] In some embodiments, the recognition model may refer to an algorithm or system that classifies or detects input data. For example, the recognition model may be an image recognition model, such as a convolutional neural network (CNN) or YOLO.
[0075] In some embodiments, in the process of training using source domain images and target domain images, it includes: obtaining feature matrices of the source domain images and target domain images; processing the feature matrices to obtain attention matrices, wherein the attention matrix includes a query matrix, a key matrix and a value matrix; and performing attention calculation based on the attention matrix.
[0076] In some embodiments, the training set may be input into the recognition model for feature extraction. For example, the source domain images in the training set may be input into the recognition model to obtain source domain features; and the target domain images in the training set may be input into the recognition model to obtain target domain features.
[0077] In some embodiments, at least one source domain image or a quasi-target domain image or a mixed domain image and at least one target domain image may be respectively input into a recognition model for feature extraction.
[0078] In some embodiments, source domain features may be extracted from target domain images or mixed domain images.
[0079] In some embodiments, feature extraction can be performed by one of a convolutional neural network, a fully connected neural network, a generative adversarial network, or a convolutional autoencoder.
[0080] In some embodiments, Figure 4 As shown, the extracted features can be divided into blocks. For example, the extracted source domain features and target domain features can be divided into blocks.
[0081] In some embodiments, each block can be used as a row of a feature matrix, and the feature dimension of each block can be used as the number of columns of the feature matrix. , the size of a block is , then the number of blocks can be expressed as:
[0082] ,
[0083] in, represents the width of the feature map, Indicates the height of the feature map.
[0084] In some embodiments, after the features are divided into blocks, the features can be encoded and added by position coding. In some embodiments, the number of blocks can be used as the number of rows of the feature matrix, and the size of the blocks can be used as the number of columns of the feature matrix to obtain the feature matrix.
[0085] In some embodiments, the feature matrix can be expressed as:
[0086] ,
[0087] in, represents the feature matrix, R represents the set, Indicates the number of blocks, Indicates the size of the block.
[0088] In some embodiments, features may be mapped to a latent space to obtain a feature matrix.
[0089] In some embodiments, the feature matrix may be processed to obtain an attention matrix. In some embodiments, the feature matrix may be multiplied by a trained weight matrix through matrix multiplication to obtain an attention matrix.
[0090] In some embodiments, the attention matrix may include a Q (Query) matrix, a K (Key) matrix, and a V (Value) matrix. In some embodiments, the query matrix may represent the ability of each element in the input data to query other elements; the key matrix may represent the key information of each element in the input data when it is queried by other elements; and the value matrix may represent the information that each element in the input data should contribute when being queried.
[0091] In some embodiments, attention calculation can be performed on the feature matrix. The formula for attention calculation is as follows:
[0092] ,
[0093] Where Q represents the query matrix, K represents the key matrix, and V represents the value matrix; softmax refers to the softmax function, which is used for normalization calculation; K T represents the transpose of the key matrix, d k Indicates the dimension size of the feature map.
[0094] In some embodiments, during training using source domain images and target domain images, the attention matrix of the source domain images and the attention matrix of the target domain images can be replaced with the attention matrix of the target domain images to reconstruct the features of the source domain images and the target domain images.
[0095] In some embodiments, before replacing the attention matrix of the source domain image and the attention matrix of the target domain image with the attention matrix of the target domain image, the following steps are further included:
[0096] Select all or part of the target domain images in the training set, and obtain the attention matrix corresponding to all or part of the target domain images. In some embodiments, at least one of the query matrix, key matrix and value matrix of all or part of the target domain images can be obtained.
[0097] In some embodiments, performing attention calculation based on the attention matrix may include: replacing the attention matrix of the source domain image with the attention matrix of the target domain image. In some embodiments, performing attention calculation based on the attention matrix may include: replacing the attention matrix of the target domain image with the attention matrix of the target domain image. In some embodiments, performing attention calculation based on the attention matrix may include: replacing the attention matrix of the mixed domain image with the attention matrix of the target domain image.
[0098] In some embodiments, performing attention calculation based on the attention matrix may include: replacing the key matrix of the source domain image and / or the key matrix of the target domain image with the key matrix of the target domain image. In some embodiments, performing attention calculation based on the attention matrix may include: replacing the key matrix of the mixed domain image with the key matrix of the target domain image.
[0099] In some embodiments, before each iteration starts, all or part of the target domain images in the training set may be selected to obtain attention matrices corresponding to all or part of the target domain images.
[0100] In some embodiments, the feature matrix of the source domain image and the target domain image can be reconstructed by adaptive weighted fusion to obtain a reconstructed feature matrix. The reconstructed feature matrix can be expressed as:
[0101] ,
[0102] in, represents the reconstructed feature matrix, Indicates the number of blocks, Indicates the size of the dimension.
[0103] In some embodiments, the feature dimensions of the reconstructed feature matrix can be mapped back to , and then restore the feature dimension to , and obtain the final feature matrix.
[0104] In some embodiments, a linear layer may be represented as:
[0105] ,
[0106] Among them, proj represents the linear layer, that is, the mapping layer.
[0107] In some embodiments, the final feature matrix can be expressed as:
[0108] ,
[0109] ,
[0110] Among them, MLP means obtaining the attention matrix through the mapping layer; reshape means the reshaping operation, which can be used to restore the feature dimension; represents the query matrix of the source domain image, represents the key matrix of the target domain image, A matrix of values representing the source domain image, represents the final feature matrix.
[0111] In some embodiments, the loss function of the recognition model may be a loss function of YOLOv5. The loss function may be expressed as:
[0112] ,
[0113] in, represents the loss function, represents the regression loss of the detection box, Represents the classification loss of the detection box.
[0114] In some embodiments, the regression loss of the detection box may refer to a loss function for optimizing the position of the detection box (i.e., the coordinates of the bounding box), such as a smooth L1 loss, an IoU loss, a GIoU loss, a DIoU loss, or a CIoU loss. The classification loss of the detection box may refer to a loss function for optimizing the category of the object in the detection box, such as a cross entropy loss, a logarithmic loss, or a focal loss.
[0115] Since it is often difficult or costly to obtain labeled data in the field of small samples, in the present application, by replacing the amplitude features of the source domain image with the amplitude features of the target domain image, the low-frequency information of the source domain image and the target domain image can be exchanged, and the information perception of the target domain can be realized, thereby enhancing the expressiveness of the source domain image and better enriching the sample features; secondly, in the process of training with source domain images and target domain-like images, by replacing the attention matrix of the source domain image and the attention matrix of the target domain-like image with the attention matrix of the target domain image, the recognition model can automatically pay attention to the features of the target domain and activate the features in the source domain that are similar to the target domain, so that the features extracted by the network will not be overly biased towards the source domain, which helps to improve the feature extraction ability of the recognition model.
[0116] Example:
[0117] The datasets selected in the embodiments of the present application are the Cityscapes and Foggy Cityscapes datasets which are very common in cross-domain detection tasks. Figure 5 As shown in the figure. The Cityscapes dataset is a large-scale urban driving scene dataset for semantic segmentation and scene understanding. The Cityscapes dataset contains street view images from 50 different cities. These images were taken under different weather conditions and at different times, covering various urban environments, such as busy city centers, suburbs, and highways. It contains 2975 training set samples and 500 validation set samples. In the cross-domain adaptation experiment, all 2975 training samples are used as source domain samples. The Foggy Cityscapes dataset is based on Cityscapes and is formed by adding artificially synthesized fog. The rest is consistent with Cityscapes.
[0118] Figure 6 Schematic diagram showing the results of the cross-domain detection task.
[0119] In addition to some existing methods, such as FAFRCNN (that is, Figure 6 Method 1), PICA (that is, Figure 6 Method 2), AcroFod (that is, Figure 6 Method 3), AsyFod (that is, Figure 6 In addition to comparing with method 4) in the previous section, we will also compare with the following baselines:
[0120] Source Only (that is, Figure 6 Baseline (source domain only) in ): The result is obtained by training on the source domain and testing on the target domain. This result reflects a lower bound of the network performance, that is, it does not require the task target domain data for training, and ultimately the generalization performance on the target domain.
[0121] Source + Few Samples (that is, Figure 6 The task reflects the result of training with target domain data by simple random sampling, which can directly reflect the generalization ability of the model brought by a small number of target domain samples.
[0122] Pretrain + Fine Tune (that is, Figure 6Pre-training + fine-tuning in ): Train on the source domain, and then use a small amount of target domain data to fine-tune the classification and regression networks. The pre-training + fine-tuning paradigm is also a common method in small sample tasks, which can also intuitively reflect the generalization ability of a small number of target domain samples to the model.
[0123] like Figure 6 As shown, the Source Only method, that is, when no target domain data is added, has the lowest effect. The model has not seen the target domain data during training, and the data distribution of the source domain and the target domain is essentially different, which will lead to poor detection results. During training, the target domain information was introduced by adding target domain data or fine-tuning based on the target domain data, and the results were improved by 3.5% and 1.8%, respectively, but the improvement was very weak. The latest work AsyFOD currently achieves 44.3% mAP. The embodiment of the present application uses the same benchmark model YOLOv5 and achieves a result of 46.5% on this task, which is 17.3% and 2.2% higher than the SourceOnly and AsyFOD models, respectively. At the same time, the embodiment of the present application conducted an ablation experiment. Figure 6 From the last three lines, we can see that the baseline model (Baseline) improves mAP (meanAverage Precision) by 13.3% by adding the multi-layer data-aware enhancement (MDA) method. It can be seen that this data enhancement method can effectively alleviate the problem of sample scarcity and allow the model to learn robust feature expressions. By adding a cross-attention module (CAFS) based on scarce samples to the baseline model, mAP can be improved by about 5%. It can be seen that this method can effectively integrate the features of the target domain samples, but because the samples are relatively scarce, the improvement is relatively limited. The embodiment of the present application performs cross-domain learning at the data and feature levels, and achieves the current optimal performance on the Cityscaps->Foggy Cityscapes dataset.
[0124] like Figure 7 As shown, the present application also provides an identification method, comprising:
[0125] S10: Acquire multiple cross-domain sample images. The cross-domain sample images may include source domain images and target domain images.
[0126] S20, inputting the cross-domain sample image into the recognition model to obtain the recognition result of the cross-domain sample image. The recognition model may be a model trained according to the training method of the cross-domain small sample recognition model as described above.
[0127] like Figure 8As shown, the present application also provides a processing device 1, including one or more processors 11 and a memory 10; wherein the memory 10 is used to store one or more computer programs, and the one or more processors 11 are used to execute one or more computer programs stored in the memory 10, so that the processor 11 executes the above-mentioned training method embodiment of the cross-domain small sample recognition model and the features / steps of the above-mentioned recognition method.
[0128] Those skilled in the art can understand that all or part of the features / steps of the above-mentioned method embodiments can be implemented by methods, data processing systems or computer programs, and these features can be implemented without hardware, entirely by software, or by a combination of hardware and software. The aforementioned computer program can be stored in one or more computer-readable storage media, and the storage medium stores the computer program. When the computer program is executed (such as by a processor), the training method embodiment of the cross-domain small sample recognition model and the steps of the above-mentioned recognition method embodiment are executed.
[0129] The aforementioned storage media that can store program codes include: static hard disks, solid-state hard disks, random access memories (SRAM), electrically erasable programmable read-only memories (EEPROM), erasable programmable read-only memories (EPROM), programmable read-only memories (PROM), read-only memories (ROM), optical storage devices, magnetic storage devices, flash memories, magnetic disks or optical disks and / or combinations of the above devices, that is, they can be implemented by any type of volatile or non-volatile storage devices or combinations thereof.
[0130] The above is only the preferred embodiment of the present application. Those skilled in the art will appreciate that various changes or equivalent substitutions may be made to these features and embodiments without departing from the spirit and scope of the present application. In addition, under the guidance of the present application, these features and embodiments may be modified to adapt to specific circumstances and materials without departing from the spirit and scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of the present application belong to the protection scope of the present application.
Claims
1. A training method for a cross-domain small sample recognition model. It is characterized in that include: Acquire an amplitude feature of any source domain image and an amplitude feature of any target domain image, replace the amplitude feature of the source domain image with the amplitude feature of the target domain image, and obtain a target domain-like image; Using the plurality of source domain images, the plurality of target domain images and the plurality of target-like domain images as training sets; The recognition model is trained using the training set, wherein in the process of training using the source domain image and the target domain image, the attention matrix of the source domain image and the attention matrix of the target domain image are replaced with the attention matrix of the target domain image to reconstruct the feature matrices of the source domain image and the target domain image.
2. The method for training a cross-domain small sample recognition model according to claim 1, It is characterized in that The training set also includes mixed domain images; the method also includes: At least two images are randomly selected from the plurality of source domain images, the plurality of target domain images and the plurality of target-like domain images for weighted summation, and labels of the at least two images are concatenated to obtain the mixed domain image.
3. The training method for a cross-domain small sample recognition model according to claim 2, It is characterized in that The method further comprises: Select at least one image area on any of the source domain images and / or any of the target domain images and / or any of the quasi-target domain images, and overlay the at least one image area on any of the source domain images and / or any of the target domain images and / or any of the quasi-target domain images to obtain the mixed domain image.
4. The method for training a cross-domain small sample recognition model according to claim 1, It is characterized in that The acquiring of the amplitude feature of any of the source domain images and the amplitude feature of any of the target domain images comprises: The frequency domain features of the source domain image and the target domain image are acquired through Fourier transform, and the amplitude feature and the phase feature are separated from the frequency domain features.
5. The method for training a cross-domain small sample recognition model according to claim 4, It is characterized in that The obtaining of the target-domain image includes: retaining the phase feature of the source-domain image; and performing an inverse Fourier transform on the source-domain image retaining the phase feature to obtain the target-domain image.
6. The method for training a cross-domain small sample recognition model according to claim 5, It is characterized in that The inverse Fourier transform is defined as: , in, represents the Fourier transform, represents the mask matrix, represents the source domain image, represents the target domain image, represents the amplitude feature of the target domain image, represents the amplitude feature of the source domain image, represents the phase feature of the source domain image.
7. The method for training a cross-domain small sample recognition model according to claim 1, It is characterized in that The process of training using the source domain image and the target domain image includes: obtaining the feature matrices of the source domain image and the target domain image; processing the feature matrices to obtain the attention matrix, wherein the attention matrix includes a query matrix, a key matrix and a value matrix; and performing attention calculation based on the attention matrix.
8. The method for training a cross-domain small sample recognition model according to claim 7, It is characterized in that The performing attention calculation based on the attention matrix includes: replacing the key matrix of the source domain image and / or the key matrix of the target domain image with the key matrix of the target domain image.
9. A method of identifying It is characterized in that include: Acquire multiple cross-domain sample images; input the cross-domain sample images into a recognition model to obtain recognition results of the cross-domain sample images, wherein the recognition model is a model trained according to the training method for a cross-domain small sample recognition model according to any one of claims 1-8.
10. A processing device, It is characterized in that include: one or more processors; A memory for storing one or more computer programs, and one or more processors for executing one or more computer programs stored in the memory, so that one or more processors execute the training method for the cross-domain small sample recognition model as described in any one of claims 1 to 8 and the recognition method as described in claim 9.
