Unsupervised adversarial adaptive learning system and method based on conditional channel attention

Abstract

The present invention discloses a learning method for an unsupervised adversarial adaptive learning system based on conditional channel attention, comprising the following steps: Step S100: embedding a domain-conditional channel attention module in a feature extractor E based on the classical residual; Step S200: setting a domain-conditional adaptive module at the front end of the feature extractor E, and setting an inter-domain hybrid generation module R at the back end of the feature extractor E; Step S300: using the inter-domain hybrid generation module R at the sign language semantic layer to generate new features; Step S400: setting the target loss function of the deep network to obtain a universal sign language representation. The present invention also discloses an unsupervised adversarial adaptive learning system based on conditional channel attention. The present invention utilizes a large number of unlabeled sign language videos on the Internet and designs a method for adversarial domain adaptive networks based on channel attention, so that the feature extraction network can obtain a universal sign language representation and improve the generalization ability of the sign language translation system.

Description

Technical Field

[0001] The present invention relates to an unsupervised adversarial adaptive learning system and method based on conditional channel attention, belonging to the technical fields of image processing, computer vision and pattern recognition. Background Art

[0002] In recent years, deep neural networks (DNNs) have achieved significant performance improvements in many different learning tasks, such as network design, computer vision, and natural language processing. However, most state-of-the-art models with good performance rely heavily on large amounts of annotated data. However, obtaining sufficient amounts of labeled training data is always time-consuming and expensive. Furthermore, traditional machine learning methods often share a common assumption that training and test data come from the same or similar probability distributions, which is often not met in practical applications. Therefore, we need to leverage useful knowledge from the relevant labeled source domain to learn classification models that conform to the characteristics of unlabeled sign language in real-world application scenarios based on different feature distributions.

[0003] Domain adaptation is an effective approach to addressing the domain shift problem. Mainstream approaches to domain adaptation are based on instance reweighting, which assigns different weights to labeled source samples, or on learning domain-invariant features to reduce the distributional discrepancy between the two domains. In contrast, recent advances in deep learning have demonstrated that deep architectures can extract more transferable and domain-invariant representations when addressing domain adaptation. Furthermore, deep learning-based approaches have shown superior performance compared to most shallow domain adaptation methods. However, most current deep domain adaptation methods still force the lab and application domain data to share the same deep convolutional network backbone, often ignoring the mismatch or mismatch between domain data during the initial input convolutional layer. Furthermore, because the learning capabilities of the semantic layer of online sign language cannot compensate for the side effects of excessive feature distribution shifts in the initial input convolutional layer, the performance of such methods deteriorates when there is a significant difference between the lab and application domains.

[0004] Recently, a more challenging and practical scenario, called partial domain adaptation, has attracted much attention, where the diversity of a large-scale laboratory scene domain is sufficient to classify all classes into a small-scale application scene domain of interest. Furthermore, if the target data is unlabeled, it is impossible to know the size of the target classes or their corresponding categories. Intuitively, solving the partial domain adaptation problem cannot simply align the entire source and application scene domains, because irrelevant source subclasses will be mixed with the target data, resulting in degraded target classification performance. Therefore, removing irrelevant source classes and enhancing the source classes that are most similar to the application scene domain is the key to effective knowledge transfer.

[0005] It can be seen from this that there is an urgent need for an unsupervised adversarial adaptive learning system and method based on conditional channel attention that can effectively extract the initial layer features of the video image network and maintain diversity, so that the feature extraction network can obtain a universal sign language semantic representation and improve the generalization ability of the sign language translation system. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that, in response to the current problem of being unable to utilize a large number of unlabeled sign language videos on the Internet, a Conditional Channel Attention-based Adversarial Domain Adaptation Network (CTADAN) is proposed, which can effectively extract the initial layer features of the video image network and maintain diversity, so that the feature extraction network can obtain a universal sign language semantic representation, thereby improving the generalization ability of the sign language translation system.

[0007] At the same time, the present invention provides an unsupervised adversarial adaptive learning method based on conditional channel attention.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] An unsupervised adversarial adaptive learning system based on conditional channel attention, including the following modules:

[0010] The feature extractor E consists of a convolutional layer connected to a max pooling layer, which is connected to an attention module, which is connected to an encoding layer for output;

[0011] The attention module consists of connected residual neurons and domain-conditional channel attention modules;

[0012] The number of attention modules is the same as the number of domains. Each attention module is used to collect global information of different domains and input different domains into their respective activation paths according to their labels. The activation paths include shared routing paths and separate routing paths.

[0013] The front end of the feature extractor E is provided with a method for inputting the laboratory scene domain feature D s Or application scene domain feature D t Domain-conditional adaptation module;

[0014] The domain condition adaptation module is used to determine the laboratory scene domain feature D s and application scenario domain features D t Enter shared routing path or separate routing path;

[0015] The backend of the feature extractor E is equipped with an inter-domain hybrid generation module R;

[0016] The inter-domain mixture generation module R is connected to the domain discriminator D and the labeled laboratory scene domain classifier C.

[0017] Global information includes features of the signer's expressions and gestures.

[0018] A learning method for an unsupervised adversarial adaptive learning system based on conditional channel attention, comprising the following steps:

[0019] Step S100: Based on the classical residual, a domain-conditional channel attention module is embedded in the feature extractor E;

[0020] Step S200: setting a domain condition adaptation module at the front end of the feature extractor E, and setting an inter-domain hybrid generation module R at the back end of the feature extractor E;

[0021] Step S300: using an inter-domain hybrid generation module R at the sign language semantic layer to generate new features;

[0022] Step S400: Setting the target loss function of the deep network to obtain a universal sign language representation.

[0023] In step S100, a lightweight domain-conditional channel attention module is embedded in the classic residual convolutional network. The domain-conditional channel attention module counts and averages the two channel information of the laboratory scene domain and the application scene domain to collect the features of the expressions and gestures of the sign language speakers in each domain, and inputs them into their respective activation paths according to the domain labels.

[0024] In step S200, the mean and variance of the convolution representation between the laboratory scene domain and the application scene domain are calculated.

[0025]

[0026] In formula (1-1), μ s and μ t are the mean values ​​of the feature distributions of the laboratory scenario domain and the application scenario domain respectively; σ s and σ t is the variance of the feature distribution in the laboratory scenario domain and the application scenario domain, ε is a small constant, a random number in the interval (0, 0.001) to avoid division by zero; m t is the statistics of application scenario domain; m s It is the laboratory scene domain statistics;

[0027] Cross-domain differences through m s and m t The absolute value of the difference between them is expressed as the cross-domain statistical distance The application scenario domain statistics m t As a relative value, the normalized difference is restricted to the interval [0, 1] by the tanh function;

[0028] Cross-domain statistical distance It can be simply expressed as:

[0029]

[0030] The cross-domain statistical distance is mapped to a value in the interval [0, 1], so it is reasonable to set a threshold λ to adaptively control routing selection; if If it is less than λ, it means that in this convolution stage, the representations of the laboratory scene domain and the application scene domain are very similar. Then, the application scene domain and the laboratory scene domain can share the source channel attention calculation weight through the same source branch; otherwise, they use separate branches to obtain the attention weights w for their specific respective fields. t , which can be expressed as:

[0031]

[0032] Among them, d s ,d t They are the signal sources of the laboratory scene domain and the application scene domain respectively, FC s , FC t is the global pooling mean layer in each domain, ReLU is the activation function, FC is the shared fully connected network layer, σ(x)=1 / (1+e -x ) is the sigmoid function.

[0033] In step S300, in the laboratory scene domain, there is a set of labeled laboratory scene domain data, and in the application scene domain, there is a set of unlabeled data, and the distributions obeyed by the two domain data are different;

[0034] Assume laboratory scene domain data represents the i-th sample in the laboratory scene domain feature, represents the label of the i-th sample in the laboratory scene domain, N s Represents the number of samples in the laboratory scene domain, and the distribution of the laboratory scene domain features is recorded as P s ;

[0035] Assume the application scenario domain characteristics Represents the i-th sample in the application scenario domain feature. The application scenario domain has no label. N t represents the number of application scenario domain samples, and the distribution of application scenario domain features is recorded as P t ;

[0036] definition N is the total number of samples in the two domains;

[0037] Definition d i ∈{0,1} is the domain label of the i-th sample, d i=1 means it belongs to the laboratory scene domain, d i =0 represents the application scenario domain;

[0038] Assume that the feature mixture data in the inter-domain mixture generation module R has a feature distribution, denoted as P r ;

[0039] The samples of the laboratory scene domain and the application scene domain are respectively obtained through the feature extractor E to obtain the laboratory scene domain features F s and application scenario domain features F t , after the inter-domain mixing generation module R, the inter-domain mixing feature F is obtained mix , while giving F mix Label the domain;

[0040] Laboratory scene domain features F s Enter classifier C, perform supervised learning, and generate isolated sign language words;

[0041] At the same time, the laboratory scene domain feature F s , inter-domain hybrid feature F mix and application scenario domain features F t Enter the domain discriminator D and perform adversarial training;

[0042] Finally, the laboratory scene domain feature F s and application scenario domain features F t In the domain discriminator D, maximum mean difference (MMD) alignment optimization is performed at the class probability level to improve the system accuracy.

[0043] Generate inter-domain hybrid features F mix The purpose is to improve the application scenario domain feature F of few samples t Quantity, random in the application scenario domain feature F t Select F t,i , by the nearest neighbor method, with Euclidean distance Dist ed As a performance metric, find the distance F t,i The most recent application scenario domain feature F t The sample point F with the smallest performance metric t,j , which is expressed as formula (1-4), formula (1-5) and formula (1-6):

[0044]

[0045] In order to make the generated inter-domain hybrid feature F mix With stronger robustness, a random perturbation term ε:N(0,1) that obeys the normal distribution is added.

[0046] Generate mixed features F in the few-sample domain t,mix :

[0047] F t,mix =F t,i +ε*(F t,i -F t,j ) (1-5)

[0048] Generate the corresponding inter-domain hybrid feature F mix and the corresponding domain label L mix , which is expressed as formula (1-6), formula (1-7) and formula (1-8):

[0049]

[0050] F mix =S(z)F s +(1-S(z))F t，mix (1-7)

[0051]

[0052] Where z is the mixing coefficient, and the clip threshold is set to clip_th = 0.2;

[0053]

[0054] In formula (1-9), when 0<z<0.2, z=0.2; when -0.2<z<0, z=-0.2; when z is within other value ranges, z=z.

[0055] Step S400 includes the following steps: classification loss of supervised learning and adversarial learning loss on classifier C.

[0056] The classification loss for supervised learning on classifier C consists of the following steps:

[0057] Using cross entropy loss L entropy To optimize:

[0058]

[0059] Among them, θ E is the network weight parameter of the feature extractor E, θ C is the network weight parameter of classifier C, L entropy Represents the cross entropy loss function used.

[0060] Adversarial learning loss consists of the following steps:

[0061] The adversarial learning loss function consists of three components, which come from the laboratory scene domain features, application scene domain features, and inter-domain mixed features, as shown in formula (1-11):

[0062]

[0063] Among them, the weight and They represent the laboratory scene domain loss function, the application scene domain loss function and the inter-domain mixed loss function, respectively. adv represents the adversarial learning loss function;

[0064] Quantity and The calculation formulas are:

[0065]

[0066]

[0067]

[0068] Among them, x s , x t are the feature samples of the laboratory scene domain and the application scene domain, d m Domain labels for inter-domain mixed generation data.

[0069] A Conditional Channel Attention-Based Adversarial Domain Adaptation Network (CTADAN) method includes the following steps:

[0070] Based on the classic residual network, a local shared network architecture is constructed;

[0071] Using the domain-conditional channel attention module, the processed data stream is divided into the domain features of the labeled laboratory scenario and the domain features of the unlabeled application scenario, and the two domains are calibrated and characterized separately;

[0072] A domain-conditional adaptation module is proposed in the front-end of the feature extraction network, and an adversarial training method based on domain mixing generation is adopted in the sign language semantic layer in the back-end of the feature extraction network.

[0073] Adopt inter-domain hybrid feature generation module to generate new features;

[0074] The difference penalty term and adversarial loss quantization are used to overcome the problem that adversarial generation methods cannot obtain stable solutions.

[0075] Furthermore, the domain-conditional channel attention module includes:

[0076] Channel statistics are averaged to collect global information for each domain and fed into their respective activation paths according to the domain label.

[0077] Furthermore, the feature extraction network front-end and back-end concepts include: adaptively modeling the individual activations of domain channels in each attention module based on the statistical differences of the attention modules of each channel;

[0078] The adversarial generation approach constrains the optimization of sign language semantic features, which can explicitly reduce domain shift;

[0079] That is, the domain-specific knowledge is retained in the initial input stage of the deep network, and the domain-general features are fully learned through adversarial training at the sign language semantic layer.

[0080] Furthermore, the inter-domain hybrid feature generation module includes:

[0081] We mainly designed data mixing of task-level features to find common features and potential relationships between the laboratory scenario domain and various application scenario domains;

[0082] A simple linear interpolation method is used to mix the extracted high-dimensional features to generate the corresponding inter-domain mixture F mix and the corresponding domain label L mix .

[0083] Furthermore, the difference penalty term and adversarial loss quantization overcome the problem of adversarial generation methods being unable to obtain stable solutions, including:

[0084] The classification loss metric constructed by cascading the feature extraction network E to the classification module C is mainly used to constrain the network to stably obtain the feature expression of the laboratory scene domain;

[0085] The adversarial learning loss function for adversarial training between the feature network E and the discriminator D aims to enable the extraction network E to obtain common features of both the laboratory and application scenario domains.

[0086] The present invention has the following beneficial effects:

[0087] The present invention is a sign language translation system based on deep learning (i.e., an unsupervised adversarial adaptive learning system and method based on conditional channel attention). It passed training tests on the CSLR500 and created datasets, and the test accuracy rate obtained using the cross-validation method was 93%. In terms of recognition effect, it has reached commercial level. However, when tested in various scenarios and environments, it was found that the accuracy of this system has decreased. By looking back at the cause of the problem, although the present invention uses massive data, each word and sentence has 50 different signers demonstrating, but the videos in the dataset are all in the same environment, with the same lighting and background, the distance between the sign language demonstrator and the camera is fixed, and the signers' speaking speed is kept in a similar range. These various conditions limit the generalization ability of the model learned using the dataset. To solve this problem, the present invention uses a large number of unlabeled sign language videos on the Internet and designs a method of adversarial domain adaptive network based on channel attention. The conditional attention module is embedded in the original network to keep the extracted network underlying features diverse. Then, a domain mixing feature module is designed to perform domain adversarial learning, so that the feature extraction network can obtain a universal sign language representation, thereby improving the generalization ability of the sign language translation system. The videos in the current laboratory sign language scene dataset are mostly taken in the same environment, with the same lighting and background, the distance between the sign language presenter and the camera is fixed, and the signer's speaking speed is kept in a similar range. These restrictions reduce the generalization ability of the model learned using the dataset. To solve this problem, the presenter designs a channel-attention-based adversarial domain adaptation network method using a large number of unlabeled sign language videos on the Internet. This method allows the feature extraction network to obtain a universal sign language representation, improving the generalization ability of the sign language translation system.

[0088] Compared with the prior art, the present invention has the following advantages:

[0089] The videos in the current laboratory sign language scene dataset are mostly taken in the same environment, with the same lighting and background, the distance between the sign language presenter and the camera is fixed, and the signer's speaking speed is kept in a similar range. These restrictions reduce the generalization ability of the model learned using the dataset. To solve this problem, the presenter designs a channel-attention-based adversarial domain adaptation network method using a large number of unlabeled sign language videos on the Internet. This method allows the feature extraction network to obtain a universal sign language representation, improving the generalization ability of the sign language translation system.

[0090] In transfer learning, due to the difference between the data distribution of the laboratory scene domain and the application scene domain, and the lack of supervised information to guide training in the application scene domain, the above-mentioned problem will have a relatively obvious negative effect on the training task. In order to reduce the useless parts of the features extracted in the laboratory scene domain, the present invention introduces a channel attention mechanism in training. Its goal is to perform a weighted calibration mechanism at the channel level of the features extracted by the convolutional network. If weighting is performed at the element level, the number of parameters will be too large, so weighting is performed at the channel level to prevent an increase in excessive computational complexity. Through this mechanism, the feature weights of the parts that contribute greatly to the model training task are increased, and the parts that contribute little to the model training task or even have a negative effect are reduced, so as to selectively enhance the features and suppress the features that are not very useful, thereby improving the quality of the features generated by the convolutional network.

[0091] The present invention further proposes a domain-conditional adaptation module at the front end of the feature extraction network. Based on the statistical differences between the attention modules of each channel, the individual activations of the domain channels in each attention module are adaptively modeled. Unlike most domain adaptation methods that only implement difference loss at the specific task layer, the present invention uses an adversarial generation approach at the back end of the feature extraction network to constrain the optimization of sign language semantic layer features, which can explicitly reduce domain shift. The proposed method provides a new concept, namely, retaining domain-specific knowledge at the initial input stage of the deep network, while fully learning domain-general features through adversarial training at the specific task layer.

[0092] The adversarial domain adaptation method based on inter-domain mixing in the present invention not only increases the data features of training, but also accelerates the convergence of the adversarial loss function. This is because when there is a huge difference in the feature distribution between the laboratory scene domain and the application scene domain, the inter-domain mixing feature generation adds constraints to the gradient descent of the loss function, which is more conducive to converging towards the global optimal direction. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 is a flow chart of the present invention;

[0094] Figure 2 This is a structural diagram of the Conditional Channel Attention-Based Adversarial Domain Adaptation Network (CTADAN) in the present invention;

[0095] Figure 3 This is the architecture diagram of the attention module in the present invention;

[0096] Figure 4 This is a diagram of the adversarial domain adaptation structure generated based on inter-domain mixing in the present invention. DETAILED DESCRIPTION

[0097] The present invention will be further described below with reference to the accompanying drawings.

[0098] like Figures 1 to 4 As shown in FIG, an unsupervised adversarial adaptive learning method based on conditional channel attention includes the following steps:

[0099] Step S100: Based on the classic residual, a domain-conditional channel attention module is embedded.

[0100] like Figure 2 Specifically, this embodiment embeds a lightweight domain-conditional channel attention module in the classic residual convolutional network. The domain-conditional channel attention module counts and averages the two channel information of the laboratory scene domain and the application scene domain to collect the features of the expression and gesture of the sign language speaker in each domain, and inputs them into their respective activation paths (shared routing path and separate routing path) according to the domain label. Figure 3 ). With the full utilization of domain-related knowledge, this method can be applied to problems where there is a large deviation in representation between different domains. This method can improve the diversity of network representation. At the same time, regarding the bottom layer in the deep convolutional network, it usually encodes general low-level features that lack domain distinction, such as the color and outline of the target in the image. It is unreasonable to implement routing separation for each domain in all domain-conditional channel attention modules. Ideally, if the statistical difference in cross-domain representation is small, making the laboratory scene domain and the application scene domain share the same channel activation structure can better improve the transferability of deep features.

[0101] Step S200: New concept of feature extraction network front-end and back-end.

[0102] This embodiment further proposes a domain condition adaptation module at the front end of the feature extraction network, such as Figure 3 As shown in Figure 2, the network uses an adaptive routing strategy to decide whether to separate domain channel activations in each domain-conditional channel attention module. The attention computation path for the application scenario domain is determined based on the defined cross-domain statistical distance. If the distance is small enough, the two domains share the attention computation route (i.e., the source branch). Otherwise, the laboratory scenario domain and the application scenario domain will be processed separately.

[0103] Specifically, the mean and variance of the intermediate convolution representations of the laboratory scene domain and the application scene domain are calculated, which reflects the feature distribution information of the two domains to a certain extent and is used as domain statistics estimation:

[0104]

[0105] In formula (1-1), μ s and μ t are the mean values ​​of the feature distributions of the laboratory scenario domain and the application scenario domain, respectively. s and σ t are their corresponding variances, and ε is a small constant to avoid division by zero.

[0106] Then, cross-domain differences can be obtained by m s and m t The absolute value of the difference between them is used to express the statistical distance. So far, the statistical distance has not been fully defined due to the uncertainty of the range of the difference. In order to specify the normalized difference, the scene domain statistic m is applied. t As a relative value, the normalized difference is limited to the interval [0, 1] by the tanh function. Cross-domain statistical distance It can be simply expressed as:

[0107]

[0108] The statistical difference is mapped to a value in the interval [0, 1], so it is reasonable to set a threshold λ to adaptively control routing selection. If it is less than λ, it means that in this convolution stage, the representations of the laboratory scene domain and the application scene domain are very similar. Then, the application scene domain and the laboratory scene domain can share the source channel attention calculation weight through the same source branch; otherwise, they use separate branches to obtain the attention weight w for their specific respective fields. t , which can be expressed as:

[0109]

[0110] Among them, d s ,d t They are the signal sources of the laboratory scene domain and the application scene domain respectively, FC s , FC t is the global pooling mean layer in each domain, ReLU is the activation function, FC is the shared fully connected network layer, σ(x)=1 / (1+e -x ) is the sigmoid function.

[0111] The dynamic attention module in CTADAN (Conditional Channel-Attention-Based Adversarial Domain Adaptation Network (CTADAN)) employs an adaptive routing strategy to selectively and non-mandatorily perform domain processing separation. This further enhances the flexibility and ability to model complex data from diverse domains, addressing the imbalance in data volume between the laboratory and application domains, which prevents the network from learning the feature distribution in the application domain with less data.

[0112] In fact, the choice of cross-domain statistical measure and the value of λ are crucial for the proposed adaptive domain-conditional channel attention module. If the similarity of domain statistics is lower than the threshold λ, it means that the domain difference is small enough to share a processing path. Setting a larger threshold λ means increasing the upper limit of the single routing strategy, thus triggering fewer separation paths.

[0113] Data features are more general in lower-level networks, allowing the source and destination to share a path with high probability (λ∈(0,0.5)). In higher convolutional layers, features are more specific to abstract tasks, triggering separate routing for the laboratory and application domains (λ∈[0.5,1]). The adaptive routing strategy helps capture the entire distribution of information in both domains and makes appropriate routing choices, significantly saving computational resources in large-scale convolutional activation networks and improving system responsiveness.

[0114] Step S300: At the sign language semantic layer ( Figure 4 ) Adopt the inter-domain hybrid feature generation module to generate new features.

[0115] This embodiment uses an adversarial generation approach to constrain and optimize the sign language semantic layer features in the backend of the feature extraction network, which can explicitly reduce domain shift. The proposed method provides a novel concept: retaining domain-specific knowledge in the initial input stage of the deep network, while using adversarial training at the sign language semantic layer to fully learn the semantic characteristics of signers in different scenarios.

[0116] In the laboratory scene domain, there is a set of labeled laboratory scene domain data, and in the application scene domain, there is a set of unlabeled data. The distributions of the two domains are different. The goal of the algorithm design is to learn a universal representation so that the classifier can perform well in both the laboratory scene domain and the application scene domain. In this embodiment, the laboratory scene domain data represents the i-th sample in the laboratory scene domain feature, represents the label of the i-th sample in the laboratory scene domain, N s Represents the number of samples in the laboratory scene domain, and the distribution of the laboratory scene domain features is recorded as P s At the same time, the application scenario domain features Represents the i-th sample in the application scenario domain feature. The application scenario domain has no label. N t represents the number of application scenario domain samples, and the distribution of application scenario domain features is recorded as P t . And define is the total number of samples in the two domains, and defines d i ∈{0,1} is the domain label of the i-th sample, d i =1 means it belongs to the laboratory scene domain, d i =0 represents the application scenario domain. In the network, the feature extractor is E, the labeled laboratory scenario domain classifier is C, and the inter-domain feature mixing module is R. Assume that the feature mixing data has a feature distribution, which is denoted as P r , the domain discriminator is D.

[0117] In the sign language semantic layer, a hybrid adversarial network based on inter-domain generation is designed, such as Figure 4 As shown. The samples of the laboratory scene domain and the application scene domain are respectively obtained through the feature extractor E to obtain the laboratory scene domain features F s and application scenario domain features F t , after the inter-domain hybrid feature module, we get F mix , while giving F mix Label the domain. Laboratory scenario domain F s Enter the classifier C, conduct supervised learning, and generate sign language isolated words. At the same time, the laboratory scene domain F s , inter-domain hybrid F mix and application scenario domain F t Enter the domain discriminator D and conduct adversarial training with the feature extraction network E through the gradient flipping layer. The training steps are based on existing technologies. Please refer to the literature Yaroslav Ganin etc., Domain-Adversarial Training of Neural Networks[J].Journal of Machine Learning Research 17(2016)1-35. Finally, the laboratory scene domain F s and application scenario domain F t In the domain discriminator network, maximum mean divergence (MMD) alignment optimization is performed at the class probability level to further improve the model accuracy.

[0118] This embodiment proposes a domain adaptive network based on adversarial generation, which mainly designs data mixing of features at the semantic level of sign language, and seeks common representations and potential relationships between the laboratory scene domain and the application scene domain. The present invention uses a method combining random perturbation generation and linear interpolation to mix the extracted high-dimensional features. mix The purpose is to improve the application scenario domain sample F of few samples t Number, random in the sample space F t Select F t,i , by the nearest neighbor method, with Euclidean distance Dist ed As a performance metric, find the distance F t,i The sample point F with the smallest performance metric in the nearest sample space t,j , and its formulas are expressed as formula (1-4), formula (1-5) and formula (1-6).

[0119]

[0120] In order to generate the mixed feature F mix With stronger robustness, a random perturbation term ε:N(0,1) that obeys the normal distribution is added.

[0121] Generate the corresponding mixed feature F in the few-sample domain t,mix :

[0122] F t,mix =F t,i +ε*(F t,i -F t,j ) (1-5)

[0123] Generate the corresponding inter-domain hybrid F mix and the corresponding domain label L mix , and its formulas are expressed as formula (1-6), formula (1-7) and formula (1-8).

[0124]

[0125] F mix =S(z)F s +(1-S(z))F t，mix (1-7)

[0126]

[0127] Where z is the mixing coefficient. The purpose of designing the generation of mixed features is: (1) to generate more samples similar to the application scenario domain and achieve a balance between the feature samples of the laboratory scenario domain and the application scenario domain; (2) the generated new features should be representative and as close to the domain feature information as possible, so that they can have a positive effect on adversarial training. It is necessary to limit the range of z change so that the generated mixed feature F mix It is not in the middle state between the two domains, because the probability of generating domain labels in the middle state not corresponding to the mixed domain features is high. Set the clipping threshold bit clip_th = 0.2.

[0128]

[0129] In formula (1-9), when 0<z<0.2, z=0.2; when -0.2<z<0, z=-0.2; when z is within other value ranges, z=z, that is, equal to itself.

[0130] Through the above design, the results obtained can achieve the design purpose, and it is possible to obtain a smooth, continuous, and domain-invariant latent distribution. When the difference in feature distribution between the two domains is too large, the hybrid feature generation module is used to assist the traditional adversarial training method to perform more effective domain adaptation and produce better universal features.

[0131] Step S400: Design a target loss function of the deep network to obtain a universal sign language representation.

[0132] The objective loss function designed for the deep network in this paper consists of two main parts: 1) a classification loss metric, constructed by cascading the feature extraction network E to the classification module C, which primarily constrains the network to stably represent features in the laboratory scene domain; and 2) an adversarial learning loss function, which performs adversarial training between the feature network E and the discriminator D. This part aims to enable the extraction network E to capture common features in both the laboratory scene domain and the application scene domain.

[0133] (1) Classification loss of supervised learning on classifier C

[0134] The design goal of this link is to let the network learn features that are invariant to the laboratory scene domain and have discriminative features for vocabulary classification tasks. The labeled data of the laboratory scene domain provides a strong constraint on the optimization direction of the network. The labeled laboratory scene domain features are used to perform supervised learning in the classifier C. In this embodiment, the cross entropy loss L is used. entropy To optimize:

[0135]

[0136] Among them, θ E is the network weight parameter of feature extractor E, θ C is the network weight parameter of classifier C, L entropy Represents the cross entropy loss function used.

[0137] (2) Adversarial learning loss (including original data and inter-domain mixed data)

[0138] Compared with traditional adversarial generative learning, the adversarial learning method based on hybrid generation designed in this embodiment generates new features through the inter-domain mixing module while using the data of the laboratory scene domain and the application scene domain in adversarial training. The loss function of adversarial learning consists of three components, which come from the laboratory scene domain features, the application scene domain features and the two-domain mixed features, as shown in Formula (1-11).

[0139]

[0140] Among them, the weight and They represent the laboratory scene domain loss function, the application scene domain loss function and the two-domain mixed loss function, respectively. adv represents the adversarial learning loss function;

[0141] Quantity and The calculation formulas are:

[0142]

[0143]

[0144]

[0145] Among them, x s , x t are the feature samples of the laboratory scene domain and the application scene domain, d m Domain labels for inter-domain mixed generation data.

[0146] This method of the present embodiment not only increases the data features of the training, but also accelerates the convergence of the adversarial loss function. Because there is a huge difference in the feature distribution between the laboratory scene domain and the application scene domain, the gradient descent of the loss function is constrained by generating mixed features between different domains, which is more conducive to convergence towards the global optimum.

[0147] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0148] Similarly, it should be understood that in order to streamline the present disclosure and aid understanding of one or more of the various inventive aspects, in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the invention.

[0149] Those skilled in the art will appreciate that the modules, units, or groups of devices in the examples disclosed herein may be arranged in the device described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the aforementioned examples may be combined into one module or further divided into multiple submodules.

[0150] It will be appreciated by those skilled in the art that the modules in the devices of the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or groups in the embodiments may be combined into one module or unit or group, and further may be divided into a plurality of submodules or subunits or subgroups. All features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any method or device so disclosed may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0151] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0152] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions described. Thus, a processor having the necessary instructions for implementing the method or method element forms a device for implementing the method or method element. Furthermore, the elements described herein of the device embodiments are examples of devices for implementing the functions performed by the elements for the purpose of implementing the invention.

[0153] The various techniques described herein may be implemented in conjunction with hardware or software, or a combination thereof. Thus, the methods and apparatus of the present invention, or certain aspects or portions of the methods and apparatus of the present invention, may take the form of program code (i.e., instructions) embedded in a tangible medium, such as a floppy disk, CD-ROM, hard drive, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes an apparatus for practicing the present invention.

[0154] When the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present invention according to the instructions in the program code stored in the memory.

[0155] By way of example and not limitation, computer-readable media include computer storage media and communication media. Computer-readable media include computer storage media and communication media. Computer storage media stores information such as computer-readable instructions, data structures, program modules, or other data. Communication media generally embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and includes any information delivery media. Combinations of any of the above are also included within the scope of computer-readable media.

[0156] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely indicates that different instances of similar objects are involved and are not intended to imply that the objects so described must have a given order in time, space, ranking, or in any other manner.

[0157] Although the present invention has been described with respect to a limited number of embodiments, it will be apparent to those skilled in the art, having benefit of the foregoing description, that other embodiments are contemplated within the scope of the invention thus described. Furthermore, it should be noted that the language used in this specification has been selected primarily for readability and didactic purposes, rather than for the purpose of explaining or limiting the subject matter of the present invention. Consequently, many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the appended claims. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. An unsupervised adversarial adaptive learning system based on conditional channel attention, characterized in that The image receiving module and the feature extractor E are used to receive video image data. The feature extractor E includes a convolution layer, the convolution layer is connected to the maximum pooling layer, the maximum pooling layer is connected to the attention module, and the attention module is connected to the encoding layer for output; The attention module includes connected residual neurons and a domain-conditional channel attention module. The domain-conditional channel attention module collects statistics and averages the information of two channels, namely the laboratory scene domain and the application scene domain, to collect the features of the signer's expression and gesture in each domain, and inputs them into their respective activation paths according to the domain label. The number of attention modules is the same as the number of domains. Each attention module is used to collect global information of different domains and input different domains into their respective activation paths according to their labels. The activation paths include shared routing paths and separate routing paths. The front end of the feature extractor E is provided with a method for inputting the laboratory scene domain feature D s Or application scene domain feature D t Domain-conditional adaptation module; The domain condition adaptation module is used to determine the laboratory scene domain feature D s and application scenario domain features D t Enter shared routing path or separate routing path; The backend of the feature extractor E is equipped with an inter-domain hybrid generation module R; The inter-domain mixture generation module R is connected to the domain discriminator D and the labeled laboratory scene domain classifier C.

2. The unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 1, characterized in that: Global information includes features of the signer's expressions and gestures.

3. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 1 or 2, characterized in that: The following steps are involved: Step S100: Based on the classical residual, a domain-conditional channel attention module is embedded in the feature extractor E; Step S200: setting a domain condition adaptation module at the front end of the feature extractor E, and setting an inter-domain hybrid generation module R at the back end of the feature extractor E; Step S300: using an inter-domain hybrid generation module R at the sign language semantic layer to generate new features; Step S400: Setting the target loss function of the deep network to obtain a universal sign language representation.

4. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 3 is characterized in that: In step S100, a lightweight domain-conditional channel attention module is embedded in the classic residual convolutional network.

5. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 4 is characterized in that: In step S200, the mean and variance of the convolution representation between the laboratory scene domain and the application scene domain are calculated. In formula (1-1), μ s and μ t are the mean values ​​of the feature distributions of the laboratory scenario domain and the application scenario domain respectively; σ s and σ t is the variance of the feature distribution in the laboratory scenario domain and the application scenario domain, ε is a small constant, a random number in the interval (0, 0.001) to avoid division by zero; m t It is the statistics of application scenario domain; m s It is the laboratory scene domain statistics; Cross-domain differences through m s and m t The absolute value of the difference between them is expressed as the cross-domain statistical distance The application scenario domain statistics m t As a relative value, the normalized difference is restricted to the interval [0, 1] by the tanh function; Cross-domain statistical distance It can be simply expressed as: The cross-domain statistical distance is mapped to a value in the interval [0, 1], so it is reasonable to set a threshold λ to adaptively control routing selection; if If it is less than λ, it means that in this convolution stage, the representations of the laboratory scene domain and the application scene domain are very similar. Then, the application scene domain and the laboratory scene domain can share the source channel attention calculation weight through the same source branch; otherwise, they use separate branches to obtain the attention weights w for their specific respective fields. t , which can be expressed as: Among them, d s ,d t They are the signal sources of the laboratory scene domain and the application scene domain respectively, FC s , FC t is the global pooling mean layer in each domain, ReLU is the activation function, FC is the shared fully connected network layer, σ(x)= / 1(1+e -x ) is the sigmoid function.

6. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 5, characterized in that: In step S300, in the laboratory scene domain, there is a set of labeled laboratory scene domain data, and in the application scene domain, there is a set of unlabeled data, and the distributions obeyed by the two domain data are different; Assume laboratory scene domain data represents the i-th sample in the laboratory scene domain feature, represents the label of the i-th sample in the laboratory scene domain, N s Represents the number of samples in the laboratory scene domain, and the distribution of the laboratory scene domain features is recorded as P s ; Assume the application scenario domain characteristics Represents the i-th sample in the application scenario domain feature. The application scenario domain has no label. N t represents the number of application scenario domain samples, and the distribution of application scenario domain features is recorded as P t ; definition N is the total number of samples in the two domains; Definition d i ∈{0,1} is the domain label of the i-th sample, d i =1 means it belongs to the laboratory scene domain, d i =0 represents the application scenario domain; Assume that the feature mixture data in the inter-domain mixture generation module R has a feature distribution, denoted as P r ; The samples of the laboratory scene domain and the application scene domain are respectively obtained through the feature extractor E to obtain the laboratory scene domain features F s and application scenario domain features F t , after the inter-domain mixing generation module R, the inter-domain mixing feature F is obtained mix , while giving F mix Label the domain; Laboratory scene domain features F s Enter classifier C, perform supervised learning, and generate isolated sign language words; At the same time, the laboratory scene domain feature F s , inter-domain hybrid feature F mix and application scenario domain features F t Enter the domain discriminator D and perform adversarial training; Finally, the laboratory scene domain feature F s and application scenario domain features F t In the domain discriminator D, maximum mean difference (MMD) alignment optimization is performed at the class probability level to improve the system accuracy.

7. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 6, characterized in that: Generate inter-domain hybrid features F mix The purpose is to improve the application scenario domain feature F of few samples t Quantity, random in the application scenario domain feature F t Select F t,i , by the nearest neighbor method, with Euclidean distance Dist ed As a performance metric, find the distance F t,i The most recent application scenario domain feature F t The sample point F with the smallest performance metric t,j , which is expressed as formula (1-4), formula (1-5) and formula (1-6): In order to make the generated inter-domain hybrid feature F mix With stronger robustness, a random disturbance term ε~Ν(0,1) that obeys the normal distribution is added. Generate mixed features F in the few-sample domain t,mix : F t,mix =F t,i +ε*(F t,i -F t,j ) (1-5) Generate the corresponding inter-domain hybrid feature F mix and the corresponding domain label L mix , which is expressed as formula (1-6), formula (1-7) and formula (1-8): F mix =S(z)F s +(1-S(z))F t，mix (1-7) Where z is the mixing coefficient, and the clip threshold is set to clip_th = 0.2; In formula (1-9), when 0<z<0.2, z=0.2; when -0.2<z<0, z=-0.2; when z is within other value ranges, z=z.

8. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 7, characterized in that: Step S400 includes the following steps: classification loss of supervised learning and adversarial learning loss on classifier C.

9. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 8, characterized in that: The classification loss for supervised learning on classifier C consists of the following steps: Using cross entropy loss L entropy To optimize: Among them, θ E is the network weight parameter of feature extractor E, θ C is the network weight parameter of classifier C, L entropy Represents the cross entropy loss function used.

10. The learning method of the unsupervised adversarial adaptive learning system based on conditional channel attention according to claim 9, characterized in that: Adversarial learning loss consists of the following steps: The adversarial learning loss function consists of three components, which come from the laboratory scene domain features, application scene domain features, and inter-domain mixed features, as shown in formula (1-11): Among them, the weight and They represent the laboratory scene domain loss function, the application scene domain loss function and the inter-domain mixed loss function, respectively. adv represents the adversarial learning loss function; Quantity and The calculation formulas are: Among them, x s , x t are the feature samples of the laboratory scene domain and the application scene domain, d m Domain labels for inter-domain mixed generation data.

Images