Model function domain adaptation method and device, storage medium and computer equipment
By grafting the image domain style features of the target domain to the image content features of the source domain in the field of image processing and machine learning, and performing feature-guided training on the model, the problem of degradation of the model in the target domain is solved, and the model is realized flexible domain adaptation and high performance adaptation.
Patent Information
- Application Number
- CN202411013748.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-11-12
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The performance degradation of the model in the target domain in the prior art is mainly due to the domain offset problem caused by the difference in feature distribution between the source domain and the target domain, resulting in poor domain adaptability of the model.
By obtaining the source function model and source image set of the source domain, collecting the target image set of the target domain, extracting image content features and domain style features, and randomly grafting the domain style features into the content features of the source image set, using feature constraint functions Deviation constraints are performed to form grafted feature data, which is used to perform feature-guided training on the source functional model and generate a target functional model that adapts to the target domain.
The effective adaptation of the model among different domains is realized, the flexibility and scalability of domain adaptation is improved, the domain offset problem is solved, and the high performance performance of the model in the target domain is ensured.
Smart Images

Figure CN118941895A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a model function domain adaptation method, device, storage medium and computer equipment. Background Art
[0002] In the field of image processing and machine learning, such as in anti-counterfeiting technology applications, the performance of the model is often highly dependent on the characteristic distribution of the training data. Traditional methods mainly rely on large-scale training models on a specific source domain. By continuously adjusting and optimizing the model parameters, the model can better fit the characteristic distribution of the source domain data, thereby being able to show excellent performance on the image set corresponding to the source domain.
[0003] However, when the model trained based on the source domain is directly applied to other target domains with the same functions as the source domain, due to the significant difference in the feature distribution of the source domain data and the target domain data, there will be a domain shift problem caused by the difference between the two domains, resulting in a sharp drop in the performance of the model. In short, in the field of image processing and machine learning, the domain adaptability of the model function is poor, and it is impossible to maintain the high performance of the model trained in the source domain in the target domain. Summary of the invention
[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the poor domain adaptability of the model function in the prior art, which cannot maintain the high performance of the model trained in the source domain in the target domain.
[0005] The present application provides a model function domain adaptation method, the method comprising:
[0006] Acquire a source function model and a source image set of a source domain, and acquire a target image set of a target domain;
[0007] Extracting image content features of each source image in the source image set and a plurality of image domain style features in the target image set;
[0008] Randomly grafting each image domain style feature with the image content feature in the source image set to obtain an image grafting feature, and using a feature constraint function to perform deviation constraints on the image grafting feature to form grafting feature data;
[0009] The grafted feature data is used to perform feature-guided training on the source function model, and the trained source function model is used as the target function model of the target domain.
[0010] Optionally, the target image set of the acquired target domain includes:
[0011] Determining an image acquisition device used by the target domain, and performing image acquisition under multiple lighting environments by using the image acquisition device to form a target image set containing multiple domain styles;
[0012] The number of images in the target image set is smaller than the number of images in the source image set.
[0013] Optionally, extracting the image content features of each source image in the source image set and the plurality of image domain style features in the target image set includes:
[0014] Determine a feature extraction model; the feature extraction model is used to extract image content features and image domain style features of the image;
[0015] Using the feature extraction model to extract features from the original image set to obtain source feature data, and removing image domain style features from the source feature data to obtain image content features of each source image in the source image set;
[0016] The feature extraction model is used to extract features from the target image set to obtain target feature data, and the target feature data is filtered to obtain a plurality of image domain style features in the target image set.
[0017] Optionally, the step of randomly grafting each image domain style feature with an image content feature in the source image set to form an image grafted feature includes:
[0018] For each image domain style feature, the image domain style feature is randomly grafted with image content features of multiple source images in the source image set to obtain multiple image grafting features corresponding to the image domain style feature.
[0019] Optionally, the feature grafting method used for grafting feature data is specifically expressed as follows:
[0020]
[0021] In the formula, Represents image grafting features; Represents the image content features of the source domain; Representing the image domain style features of the target domain.
[0022] Optionally, the feature constraint function is specifically expressed as follows:
[0023]
[0024] In the formula, represents the constraint value; N represents the total number of images in the target image set; Represents the image domain style features of the i-th target image in the target image set; Represents the image domain style feature of the grafted features of the i-th image after feature grafting.
[0025] Optionally, the source function model is composed of a shallow network and a deep network;
[0026] The step of performing feature-guided training on the source function model using the grafted feature data, and using the trained source function model as the target function model of the target domain, includes:
[0027] Freezing the parameters of the shallow network, and extracting target content features of the grafted feature data using the shallow network;
[0028] Inputting the target content feature into the deep network so that the deep network updates the parameters of the deep network based on the target content feature;
[0029] When the source function model meets the preset training end condition, the trained source function model is used as the target function model of the target domain.
[0030] The present application also provides a model function domain adaptation device, comprising:
[0031] An image acquisition module is used to acquire a source function model and a source image set of a source domain, and to acquire a target image set of a target domain;
[0032] A feature extraction module, used to extract image content features of each source image in the source image set and multiple image domain style features in the target image set;
[0033] A feature grafting module, used for randomly grafting each image domain style feature with the image content feature in the source image set to obtain an image grafting feature, and using a feature constraint function to perform deviation constraints on the image grafting feature to form grafting feature data;
[0034] The functional domain adaptation module is used to use the grafted feature data to perform feature-guided training on the source functional model, and use the trained source functional model as the target functional model of the target domain.
[0035] The present application also provides a storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the model function domain adaptation method as described in any of the above embodiments.
[0036] The present application also provides a computer device, comprising: one or more processors, and a memory;
[0037] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the model function domain adaptation method described in any one of the above embodiments are performed.
[0038] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:
[0039] The model function domain adaptation method, device, storage medium and computer equipment provided by the present application can first obtain the source function model and source image set of the source domain, and collect the target image set of the target domain when the model function needs to be domain adapted, so as to serve as feature learning data for the model function domain adaptation process; then the image content features of each source image in the source image set and the multiple image domain style features in the target image set can be extracted, and these can be used as key features of the target image set to reduce the training scale of the model function domain adaptation so as to quickly adapt to the target domain; then each image domain style feature can be randomly grafted with the image content features in the source image set to obtain image grafted features, and the image grafted features can be bias constrained by a feature constraint function to form grafted feature data, so that by grafting a small number of key features on the target domain to the image content features that have been learned by the source function model, the effective adaptation of the model between different domains can be improved, thereby achieving the flexibility and scalability of domain adaptation; finally, the present application can use the grafted feature data to perform feature-guided training on the source function model, and use the trained source function model as the target function model of the target domain. In this way, the target function model can quickly adapt to the feature distribution of the target domain while retaining the model function of the source domain, thereby solving the domain shift problem and achieving flexible domain adaptation of the model function. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0041] Figure 1 A flow chart of a model function domain adaptation method provided in an embodiment of the present application;
[0042] Figure 2 A schematic diagram of a flow chart of model function domain adaptation provided in an embodiment of the present application;
[0043] Figure 3 A schematic diagram of the structure of a model function domain adaptation device provided in an embodiment of the present application;
[0044] Figure 4 A schematic diagram of the internal flow of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0046] When the model trained on the source domain is directly applied to other target domains with the same functions as the source domain, the feature distribution of the source domain data is significantly different from that of the target domain data, which will cause domain shift problems caused by inter-domain differences, resulting in a sharp drop in model performance. In short, in the field of image processing and machine learning, the domain adaptability of model functions is poor, and it is impossible to maintain the high performance of the model trained on the source domain in the target domain.
[0047] Based on this, this application proposes the following technical solutions, see below for details:
[0048] S110: Acquire a source function model and a source image set of a source domain, and collect a target image set of a target domain.
[0049] In this step, when the user needs to perform domain adaptation of the model between different domains, the corresponding source domain and target domain can be selected in the computer device, so that the computer device can first obtain the source function model and source image set of the source domain, and collect the target image set of the target domain, thereby obtaining the basic data for feature learning of the model function domain adaptation process.
[0050] Among them, the source domain refers to a field that has accumulated a large amount of annotated data and has good model performance. It contains data sets used for model training and corresponding labels or supervision information; in this application, the source functional model is a functional model obtained by model training with the original image set collected from the source domain. The target domain refers to a new domain with the same function as the source domain, but with less data annotation and no model training. However, due to specific differences in the application scenarios of the target domain and the source domain, such as vision, lighting conditions or background, the data feature distribution between the two domains is different.
[0051] For example, in one application scenario, the source domain and the target domain can be face attendance devices connected to different image acquisition sensors. Due to the different performance of the image acquisition sensors, the face images obtained have different image effects, such as surface smoothness, exposure, resolution, color saturation, etc., so they can be regarded as image data in two different domains.
[0052] It is understandable that, although the functions of face attendance devices connected to different image acquisition sensors are the same, the data feature distributions of the image data collected by the sensors are quite different, and the data training of the model cannot fit the data with different feature distributions in other domains. Therefore, if the source function model trained in advance in the source domain is directly applied to the target domain, it will cause domain shift problems and lead to a decrease in model performance. In this regard, the present application can collect a target image set in the target domain to use the data feature distribution in the target image set to feature guide the source data set of the source domain, so as to achieve functional adaptation of the source function model in other domains.
[0053] S120: extracting image content features of each source image in the source image set and multiple image domain style features in the target image set.
[0054] In this step, after obtaining the source image set of the source domain and the target image set of the target domain through step S110, the computer device can perform feature extraction on the images in the source image set and the target image set, respectively, so as to obtain the image content feature of each source image in the source image set and the multiple image domain style features in the target image set. The image domain style features here can be used as the key features of the target image set, so as to reduce the training scale of the model function domain adaptation so as to quickly adapt to the target domain.
[0055] Among them, image content features refer to the information set of objects specifically presented in the image and their mutual relationships, including but not limited to faces, objects, scenes in the image, and their shapes, sizes, layouts, and relative positions, etc., which are the key to understanding and identifying basic image information; and image domain style features refer to the uniqueness and beauty of the image in visual expression, including the exposure level, color matching, texture details, etc. of the image. In this application, the image domain style features of the target image set can refer to the image features of the target domain under multiple lighting environments covered during image acquisition.
[0056] It is understandable that the image content features extracted from images collected from different target objects are different, while the image domain style features extracted from images collected from different target objects under the same lighting environment are the same. Therefore, the present application only needs to collect a small amount of images in the target domain to extract the various image domain style features covered in the target domain, and then implement feature grafting in the source domain, quickly achieving a data feature distribution close to that of the source domain image, so as to improve the flexibility and scalability of model domain adaptation.
[0057] S130: Randomly perform feature grafting on each image domain style feature and the image content feature in the source image set to obtain image grafting features, and use a feature constraint function to perform deviation constraints on the image grafting features to form grafting feature data.
[0058] In this step, after obtaining the image content features of the source image set and the image domain style features of the target domain through step S120, the computer device can randomly graft the image domain style features with the image content features in the source image set to obtain image grafted features, and use feature constraint functions to perform deviation constraints on the image grafted features to form grafted feature data. By grafting the key features on the target domain to the image content features that have been learned by the source function model, the effective adaptability of the model between different domains can be improved.
[0059] It can be understood that in the source domain, by learning the image content features, the model can extract the specific target shape and background information of the image, which can ensure that the model can understand the specific content of the image and is the basis of the model feature orientation; and by grafting the image domain style features of the target domain to the image content features of the source image set for feature learning, the model can introduce the image visual effects of the target domain, and graft the overall visual style such as exposure, color, texture details in the image domain style features of the target domain to the image content features of the source domain, forming an image grafting feature with the image domain style features of the target domain. Therefore, the source function model domain can be quickly adapted to the target domain, thereby improving the robustness and generalization of the model.
[0060] Furthermore, in order to constrain the problem of domain feature deviation in the image domain style features of the target domain during the feature grafting process, the present application can define a domain feature supervision loss as a feature constraint function, so as to minimize the domain feature deviation of the image domain style features during the feature grafting process by minimizing the feature constraint function, thereby ensuring that the domain style features of the grafted feature data generated after the deviation constraint and the image domain style features of the target domain belong to the same feature distribution space, thereby improving the efficiency and high performance of the model function domain adaptation.
[0061] S140: Use the grafted feature data to perform feature-guided training on the source function model, and use the trained source function model as the target function model of the target domain.
[0062] In this step, after the grafted feature data is obtained by grafting in step S130, the computer device can use the grafted feature data to perform feature-guided training on the source function model, and use the trained source function model as the target function model of the target domain. Therefore, while retaining the model function of the source domain, the target function model can also quickly adapt to the feature distribution of the target domain, thereby solving the domain shift problem and achieving flexible domain adaptation of the model function.
[0063] Specifically, the computer device can use the grafted feature data of the target domain as the guidance information for training the target function model, and then perform feature-guided training. Specifically, the computer device can input the grafted feature data of the target domain and the source image set of the source domain into the source function model, so as to adjust the parameters of the source function model through the optimization algorithm, so that the model after parameter adjustment can make full use of the effective information in the grafted feature data during the training process to improve the generalization ability of the model in the target domain.
[0064] It can be understood that the present application guides the training of the source function model by grafting feature data, which can avoid the collection of a large amount of training data in the target domain, and then train the target function model from scratch. Instead, it uses the existing knowledge in the source function model that has been trained in the source domain with the same function to perform functional domain adaptation, thereby accelerating the training process of model functional domain adaptation and reducing the data dependence on the target domain to a certain extent, so that the model can achieve flexible domain adaptation in other domains with limited data.
[0065] In the above embodiment, when it is necessary to perform domain adaptation on the model function, the source function model and the source image set of the source domain can be obtained first, and the target image set of the target domain can be collected, so as to serve as feature learning data for the model function domain adaptation process; then, the image content features of each source image in the source image set and the multiple image domain style features in the target image set can be extracted, and these can be used as key features of the target image set to reduce the training scale of the model function domain adaptation so as to quickly adapt to the target domain; then, each image domain style feature can be randomly grafted with the image content features in the source image set to obtain an image grafted feature, and a feature constraint function can be used to perform deviation constraints on the image grafted feature to form grafted feature data. Therefore, by grafting a small number of key features on the target domain onto the image content features that have been learned by the source function model, the effective adaptation of the model between different domains can be improved, thereby achieving the flexibility and scalability of domain adaptation; finally, the present application can use the grafted feature data to perform feature-guided training on the source function model, and use the trained source function model as the target function model of the target domain. In this way, the target function model can quickly adapt to the feature distribution of the target domain while retaining the model function of the source domain, thereby solving the domain shift problem and achieving flexible domain adaptation of the model function.
[0066] In one embodiment, collecting a target image set of a target domain in step S110 may include:
[0067] S111: Determine an image acquisition device used in the target domain, and use the image acquisition device to acquire images under multiple lighting environments to form a target image set containing multiple domain styles.
[0068] In this embodiment, when the computer device collects the image data set of the target domain, it can first determine the image acquisition device used by the target domain, and then use the image acquisition device to collect images under multiple lighting environments to form a target image set containing multiple domain styles. The images collected under each lighting environment correspond to one domain style.
[0069] It is understandable that under the same lighting environment, there may be some feature differences between images collected at different angles and compositions. Therefore, when collecting the target image set, for each lighting environment, the computer device can collect images from multiple angles and compositions through the image acquisition device to ensure that the computer device can extract more domain style feature details from the images in the same lighting environment, thereby improving the generalization ability of the model in the target domain.
[0070] For example, when the image acquisition device acquires the target image set, it can acquire images in scenes with different lighting conditions, including darkroom, indoors, windows, and outdoors, so as to ensure that the light intensity in each scene increases progressively, thereby forming images with multiple domain styles.
[0071] It should be noted that the domain style in the target domain is limited and only needs to be used as a feature guide in the process of model function domain adaptation. Therefore, the number of images collected in the target image set can be smaller than the number of images in the source image set. In this application, the source image set and the target image set can be obtained in a ratio of 1000:1, and then the image domain style features of the target image set can be grafted onto the image content features of the source image set, so that the image domain style features of the target image set can be adapted in the source function model.
[0072] In one embodiment, step S120 extracts the image content features of each source image in the source image set and the multiple image domain style features in the target image set, which may include:
[0073] S121: Determine a feature extraction model; the feature extraction model is used to extract image content features and image domain style features of the image.
[0074] S122: Extract features from the original image set using a feature extraction model to obtain source feature data, and remove image domain style features from the source feature data to obtain image content features of each source image in the source image set.
[0075] S123: extracting features from the target image set using the feature extraction model to obtain target feature data, and filtering the target feature data to obtain a plurality of image domain style features in the target image set.
[0076] In this embodiment, when the computer device performs feature extraction on the source image set and the target image set, it can first determine a feature extraction model. The feature extraction model is mainly used to extract the image content features and image domain style features of the image. Therefore, after the source image set and the target image set are respectively input into the feature extraction model, the image content features and image domain style features of each image output by the feature extraction model can be obtained. Finally, the computer device can filter the extraction results of the feature extraction model to obtain the image content features of each source image in the source image set and multiple image domain style features in the target image set.
[0077] Among them, the feature extraction model of the present application refers to a model that extracts features from an input image and obtains image content features and image domain style features. When the feature extraction model is training the model, different types of sample images can be used as training samples, and sample labels, i.e., corresponding image content features and image domain style features, can be annotated for each training sample. When all training samples are annotated, the training samples with sample labels can be input into a preset pre-trained model for forward propagation to train the model, and the preset target loss function can be used to tune the model parameters during the back propagation process of the model. When the model meets certain training conditions or parameter convergence conditions, such as when the number of iterations reaches the set value, the training is considered to be completed, and the trained model can be used as the final feature extraction model.
[0078] Specifically, in order to maintain the main information of the training data in the source functional model, the computer device can remove the image domain style features in the source feature data to obtain the image content features of each source image in the source image set, so as to retain the feature distribution learned by the source functional model during the training process and avoid the domain shift problem in the subsequent domain adaptation process. In addition, the computer device can also filter the target feature data to obtain multiple image domain style features in the target image set, so that each image domain style feature can be used as a feature guide in the subsequent domain adaptation process to achieve the functional adaptation of the model from the source domain to the target domain.
[0079] In one embodiment, in step S130, each image domain style feature is randomly grafted with the image content feature in the source image set to form an image grafting feature, which may include:
[0080] S131: For each image domain style feature, randomly perform feature grafting on the image domain style feature with image content features of multiple source images in a source image set to obtain multiple image grafting features corresponding to the image domain style feature.
[0081] In this embodiment, when the computer device performs feature grafting, for each image domain style feature, the computer device can randomly graft the image domain style feature with the image content features of multiple source images in the source image set to obtain multiple image grafting features corresponding to the image domain style feature, thereby achieving a perfect combination of the domain style of the target domain and the main information of the source domain, avoiding the domain offset problem.
[0082] It can be understood that the present application can realize the migration of domain style in the image by feature grafting the image content features of the source domain and the image domain style features of the target domain. The grafted feature data obtained in this way can integrate the domain style of the target image set of the target domain while maintaining the main information of the source image set of the source domain. Therefore, the grafted feature data is used to perform feature-guided training on the source function model of the source domain, so that the function of the model can be quickly adapted to the target domain.
[0083] In one embodiment, the feature grafting method used in grafting feature data in step S130 is specifically represented as follows:
[0084]
[0085] In the formula, Represents image grafting features; Represents the image content features of the source domain; Representing the image domain style features of the target domain.
[0086] In this embodiment, It represents the feature fusion operation of element-by-element addition. Therefore, when the computer device performs feature grafting on the image content features of the source domain and the image domain style features of the target domain, it can first correspond the dimensions of each element in the image content features and the image domain style features, and then perform the addition operation to generate the image grafting features.
[0087] Furthermore, if the image content features and the image domain style features are incompatible in dimension, the computer device may first adjust the dimensions of the image content features and the image domain style features before performing feature grafting, including operations such as scaling, cropping, and padding, and then use an element-by-element addition method to fuse the image content features and the image domain style features to form an image grafted feature. Finally, the computer device may also use a parser to parse the grafted image grafted features into an image representation, so that it can better guide training in the source function model.
[0088] In one embodiment, the feature constraint function in step S130 is specifically expressed as follows:
[0089]
[0090] In the formula, represents the constraint value; N represents the total number of images in the target image set; Represents the image domain style features of the i-th target image in the target image set; Represents the image domain style feature of the grafted features of the i-th image after feature grafting.
[0091] In this embodiment, the feature constraint function can be implemented by calculating the Euclidean distance between the image domain style features of the target image set and the image domain style features after feature grafting. Since the image domain style features after feature grafting are actually the fusion results of the image content features of the source image set and the image domain style features of the target image set, it cannot directly correspond to the feature representation of a separate domain style in the target domain. Therefore, the present application can use a certain method, such as the Gram matrix, to extract the feature representation in the image domain style features after feature grafting, and then compare the feature differences between the two.
[0092] It is understandable that in the process of feature grafting, the feature constraint function can guide the image grafting features to imitate the image domain style features of the target domain as much as possible while retaining the image content features of the source domain. The computer device minimizes the domain feature deviation of the image domain style features during the grafting process by minimizing the feature constraint function, ensuring that the grafted image domain style features and the image domain style features of the target domain belong to the same feature distribution space, thereby improving the learning ability of the model during the domain adaptation process.
[0093] In one embodiment, the source function model in step S140 is composed of a shallow network and a deep network; wherein the step of performing feature-guided training on the source function model using the grafted feature data and using the trained source function model as the target function model of the target domain may include:
[0094] S141: Freeze the parameters of the shallow network, and use the shallow network to extract target content features of the grafted feature data.
[0095] S142: Input the target content feature into the deep network, so that the deep network updates the parameters of the deep network based on the target content feature.
[0096] S143: When the source function model meets the preset training end condition, the trained source function model is used as the target function model of the target domain.
[0097] In this embodiment, when the computer device performs feature-guided training on the source function model, it can first freeze the parameters of the shallow network, and then use the shallow network to extract the target content features of the grafted feature data; then the computer device can input the target content features into the deep network, so that the deep network updates the parameters of the deep network based on the target content features. When the source function model meets the preset training end conditions, the computer device can use the trained source function model as the target function model of the target domain.
[0098] Specifically, the computer device can send the grafted feature data to the shallow network so that the shallow network can extract the features of the grafted feature data and obtain the target content features, so that the target content features can be used as feature references for the deep network. Then, the computer device can use the target content features extracted by the shallow network as the input of the deep network, and use the preset loss function to perform forward propagation to calculate the loss, and then update the parameters of the deep network through the backpropagation algorithm to minimize the loss function. Finally, the computer device can repeat the parameter fine-tuning process until the source function model meets the preset training end conditions, such as reaching the preset number of iterations and the loss value is no longer significantly reduced, the model training is considered to be completed, and it is used as the target function model of the target domain. Among them, the loss function can be designed as the difference between the target content features and the image content features of the pre-input source image set.
[0099] In order to better explain the model function domain adaptation method of this application, the following will be Figure 2 To further illustrate, schematically, Figure 2 As shown, Figure 2 A schematic diagram of a flow chart of model function domain adaptation provided in an embodiment of the present application.
[0100] Figure 2 In the process, the computer device can obtain a batch of source image sets in the source domain and a small number of target image sets in the target domain, and then input the source image sets and the target image sets into the feature extraction model respectively, so that the feature extraction model can extract the image content features of the source image sets. and image domain style features , and the image content features of the target image set and image domain style features ; The computer device can then and image domain style features Feature grafting is performed, and in the feature grafting process, a feature constraint function is used to perform domain feature supervision loss to form grafted feature data. Finally, the computer device can use the source function model trained with the source image set to It is used as a pre-trained model for the target domain, and the grafted feature data is used to perform feature-guided pre-training on it, thereby achieving domain adaptation of the model function.
[0101] The model function domain adaptation device provided in an embodiment of the present application is described below. The model function domain adaptation device described below and the model function domain adaptation method described above can be referenced to each other.
[0102] In one embodiment, Figure 3 As shown, Figure 3A schematic diagram of the structure of a model function domain adaptation device provided in an embodiment of the present application; the present application also provides a model function domain adaptation device, including an image acquisition module 210, a feature extraction module 220, a feature grafting module 230 and a function domain adaptation module 240, specifically including the following:
[0103] The image acquisition module 210 is used to acquire a source function model and a source image set of a source domain, and to collect a target image set of a target domain.
[0104] The feature extraction module 220 is used to extract image content features of each source image in the source image set and multiple image domain style features in the target image set.
[0105] The feature grafting module 230 is used to randomly graft each image domain style feature with the image content feature in the source image set to obtain image grafting features, and use a feature constraint function to constrain the deviation of the image grafting features to form grafting feature data.
[0106] The functional domain adaptation module 240 is used to perform feature-guided training on the source functional model using the grafted feature data, and use the trained source functional model as the target functional model of the target domain.
[0107] In the above embodiment, when it is necessary to perform domain adaptation on the model function, the source function model and the source image set of the source domain can be obtained first, and the target image set of the target domain can be collected, so as to serve as feature learning data for the model function domain adaptation process; then, the image content features of each source image in the source image set and the multiple image domain style features in the target image set can be extracted, and these can be used as key features of the target image set to reduce the training scale of the model function domain adaptation so as to quickly adapt to the target domain; then, each image domain style feature can be randomly grafted with the image content features in the source image set to obtain an image grafted feature, and a feature constraint function can be used to perform deviation constraints on the image grafted feature to form grafted feature data. Therefore, by grafting a small number of key features on the target domain onto the image content features that have been learned by the source function model, the effective adaptation of the model between different domains can be improved, thereby achieving the flexibility and scalability of domain adaptation; finally, the present application can use the grafted feature data to perform feature-guided training on the source function model, and use the trained source function model as the target function model of the target domain. In this way, the target function model can quickly adapt to the feature distribution of the target domain while retaining the model function of the source domain, thereby solving the domain shift problem and achieving flexible domain adaptation of the model function.
[0108] In one embodiment, the image acquisition module 210 may include:
[0109] The image acquisition submodule is used to determine the image acquisition device used in the target domain, and to acquire images under multiple lighting environments through the image acquisition device to form a target image set containing multiple domain styles. The number of images in the target image set is less than the number of images in the source image set.
[0110] In one embodiment, the feature extraction module 220 may include:
[0111] The extraction model determination submodule is used to determine the feature extraction model; the feature extraction model is used to extract the image content features and image domain style features of the image.
[0112] The first feature extraction submodule is used to extract features from the original image set using a feature extraction model to obtain source feature data, and to remove image domain style features from the source feature data to obtain image content features of each source image in the source image set.
[0113] The second feature extraction submodule is used to extract features from the target image set using the feature extraction model to obtain target feature data, and filter the target feature data to obtain multiple image domain style features in the target image set.
[0114] In one embodiment, the feature grafting module 230 may include:
[0115] The style feature grafting submodule is used to randomly graft each image domain style feature with image content features of multiple source images in a source image set to obtain multiple image grafting features corresponding to the image domain style feature.
[0116] In one embodiment, the feature grafting module 230 may further include:
[0117]
[0118] In the formula, Represents image grafting features; Represents the image content features of the source domain; Representing the image domain style features of the target domain.
[0119] In one embodiment, the feature grafting module 230 may further include:
[0120]
[0121] In the formula, represents the constraint value; N represents the total number of images in the target image set; Represents the image domain style features of the i-th target image in the target image set; Represents the image domain style feature of the grafted features of the i-th image after feature grafting.
[0122] In one embodiment, the functional domain adaptation module 240 may include:
[0123] The parameter freezing submodule is used to freeze the parameters of the shallow network and use the shallow network to extract the target content features of the grafted feature data.
[0124] The parameter updating submodule is used to input the target content features into the deep network so that the deep network updates the parameters of the deep network based on the target content features.
[0125] The model generation submodule is used to use the trained source function model as the target function model of the target domain when the source function model meets the preset training end conditions.
[0126] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the model function domain adaptation method as described in any of the above embodiments.
[0127] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the model function domain adaptation method as described in any one of the above embodiments.
[0128] Indicatively, if Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 4 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301 for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the model function domain adaptation method of any of the above embodiments.
[0129] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.
[0130] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0131] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0132] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.
[0133] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A model function domain adaptation method, characterized in that: The method comprises: Acquire a source function model and a source image set of a source domain, and acquire a target image set of a target domain; Extracting image content features of each source image in the source image set and a plurality of image domain style features in the target image set; Randomly grafting each image domain style feature with the image content feature in the source image set to obtain an image grafting feature, and using a feature constraint function to perform deviation constraints on the image grafting feature to form grafting feature data; The grafted feature data is used to perform feature-guided training on the source function model, and the trained source function model is used as the target function model of the target domain.
2. The model function domain adaptation method according to claim 1, characterized in that: The target image set of the target domain is acquired, including: Determining an image acquisition device used by the target domain, and performing image acquisition under multiple lighting environments by using the image acquisition device to form a target image set containing multiple domain styles; The number of images in the target image set is smaller than the number of images in the source image set.
3. The model function domain adaptation method according to claim 1, characterized in that: The extracting of the image content features of each source image in the source image set and the plurality of image domain style features in the target image set comprises: Determine a feature extraction model; the feature extraction model is used to extract image content features and image domain style features of the image; Using the feature extraction model to extract features from the original image set to obtain source feature data, and removing image domain style features from the source feature data to obtain image content features of each source image in the source image set; The feature extraction model is used to extract features from the target image set to obtain target feature data, and the target feature data is filtered to obtain a plurality of image domain style features in the target image set.
4. The model function domain adaptation method according to claim 1, characterized in that: The step of randomly grafting each image domain style feature with the image content feature in the source image set to form an image grafting feature includes: For each image domain style feature, the image domain style feature is randomly grafted with image content features of multiple source images in the source image set to obtain multiple image grafting features corresponding to the image domain style feature.
5. The model function domain adaptation method according to claim 1 or 4, characterized in that: The feature grafting method used in the grafted feature data is specifically shown as follows: In the formula, Represents image grafting features; Represents the image content features of the source domain; Representing the image domain style features of the target domain.
6. The model function domain adaptation method according to claim 1, characterized in that: The characteristic constraint function is specifically expressed as follows: In the formula, represents the constraint value; N represents the total number of images in the target image set; Represents the image domain style features of the i-th target image in the target image set; Represents the image domain style feature of the grafted features of the i-th image after feature grafting.
7. The model function domain adaptation method according to claim 1, characterized in that: The source function model is composed of a shallow network and a deep network; The step of performing feature-guided training on the source function model using the grafted feature data, and using the trained source function model as the target function model of the target domain, includes: Freezing the parameters of the shallow network, and extracting target content features of the grafted feature data using the shallow network; Inputting the target content feature into the deep network so that the deep network updates the parameters of the deep network based on the target content feature; When the source function model meets the preset training end condition, the trained source function model is used as the target function model of the target domain.
8. A model function domain adaptation device, characterized in that: include: An image acquisition module is used to acquire a source function model and a source image set of a source domain, and to acquire a target image set of a target domain; A feature extraction module, used to extract image content features of each source image in the source image set and multiple image domain style features in the target image set; A feature grafting module, used for randomly grafting each image domain style feature with the image content feature in the source image set to obtain an image grafting feature, and using a feature constraint function to perform deviation constraints on the image grafting feature to form grafting feature data; The functional domain adaptation module is used to use the grafted feature data to perform feature-guided training on the source functional model, and use the trained source functional model as the target functional model of the target domain.
9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the model function domain adaptation method as described in any one of claims 1 to 7.
10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the model function domain adaptation method as described in any one of claims 1 to 7 are performed.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and computer readable medium
CN112581355A
Image processing method and device and computer readable storage medium
CN113111947A
Cross-domain target detection method based on multi-level domain adaptive weak supervised learning
CN116342942A
Image processing method and apparatus, and computer readable storage medium
WO2022218396A1
