A process collaborative learning method based on multi-modal feature bias Gaussianization

Abstract

The application discloses a process collaborative learning method based on multi-modal feature deviation Gaussianization, which constructs a multi-flow student network team, and the image features, text features and sound features of each multi-flow student network in the team. Then, a multi-modal feature converter converts the features of each flow to obtain multi-modal features that can be adapted between the multi-flow student networks. Next, the deviation between the output features of each sub-network of the multi-flow student network and the output features of each sub-network of the other student networks is calculated, and the deviation is Gaussianized by using a segmented power level conversion. Finally, for the feature signals using the multi-modal feature deviation Gaussianization technology, a process collaborative learning function is established to complete the optimization of the multi-flow student network.

Description

Technical Field

[0001] This invention relates to a process collaborative learning method based on Gaussianization of multimodal feature bias. Background Technology

[0002] With the widespread application of deep learning across various fields, model compression technology is crucial for achieving real-time inference and reducing energy consumption on resource-constrained devices. Model compression transforms large models into smaller ones, making them easier to deploy on mobile devices, embedded systems, or IoT devices, thereby driving the practical application of deep learning technology in various scenarios. Knowledge distillation is a model compression method that transfers knowledge from a large, complex model (teacher model) to a small, simple model (student model), aiming to achieve similar performance to the large model but with lower computational complexity and storage requirements.

[0003] Knowledge distillation methods can be divided into online distillation and offline distillation. Offline distillation involves first training a teacher model, then fixing its parameters, and using it to guide the training of the student model. In online distillation, both the teacher and student models are updated simultaneously. It can be viewed as a collaborative learning process with multiple deep neural networks, where any one network can serve as a student model, and the others as teacher models.

[0004] Online distillation methods have attracted widespread attention in the field of deep learning research due to their excellent compression performance. However, current offline knowledge distillation methods still lack deep interaction in the learning process, thus affecting the model compression performance. To address this issue, this invention discloses a process-based collaborative learning algorithm based on Gaussianization of multimodal feature bias, which improves the deep interactive learning between student networks in online distillation algorithms. Summary of the Invention

[0005] To address the lack of deep interaction in the learning process in current offline knowledge distillation methods, this invention provides a process-based collaborative learning method based on Gaussianization of multimodal feature bias, which improves the deep interactive learning between student networks in online distillation algorithms.

[0006] The technical solutions adopted in this invention are as follows:

[0007] A process-based collaborative learning method based on Gaussianization of multimodal feature bias includes the following steps:

[0008] S1: Construct a multi-stream student network group consisting of several multi-stream student networks. Each multi-stream student network includes an image sub-network, a text sub-network, and an audio sub-network. Input the image samples, text samples, and audio samples into the corresponding sub-network of each multi-stream student network to obtain the image, text, and audio feature outputs of each multi-stream student network.

[0009] S2: Receive the feature output of each multi-stream student network through the constructed multimodal feature converter, perform corresponding feature transformation, and output the transformed multimodal features;

[0010] S3: For each student network in a multi-stream student network group, calculate the deviation between the output features of each subnetwork in each multi-stream student network and the output features of the corresponding subnetworks in other multi-stream student networks, and use piecewise power transformation to Gaussianize the deviation;

[0011] S4: For feature signals that use multimodal feature bias Gaussianization, establish a process co-learning function.

[0012] Furthermore, S1 specifically includes:

[0013] S11: Initialize a total of S multi-stream student networks. The s-th multi-stream student network is represented as... in, For the image subnetwork with parameter θ, For a text subnetwork with parameter μ, For parameters The sound sub-network; S multi-stream student networks form a multi-stream student network group.

[0014] S12: Randomly select a batch of N samples from the image sample set, text sample set, and audio sample set, respectively, where any one sample is represented as x. i y i , z i The values ​​are input into the subnetwork of the s-th multi-stream student network, respectively, and the following features are obtained:

[0015]

[0016] Furthermore, the multimodal feature converter is a fully connected layer with multiple signal inputs and multiple output signals. The multiple signal inputs respectively receive the feature outputs of each subnetwork of the multi-stream student network, and the multiple signal outputs respectively output the converted multimodal features.

[0017] Furthermore, S2 specifically includes:

[0018] S21: Represent the multimodal feature converter as Tran = {O} w1 O w2 O w3 B w ,P w1 ,P w2 ,P w3}, where O w1 O w2 O w3These represent the feature input converters with parameters w1, w2, and w3, respectively. B w For a multimodal feature mixing converter with parameter w, P w1 ,P w2 ,P w3 These represent the characteristic output converters with parameters w1, w2, and w3, respectively.

[0019] S22: The feature first enters the feature input converter to prepare for entering the multimode mixing converter. Each stream signal feature undergoes the following conversion at the input:

[0020]

[0021]

[0022] Then, the characteristics of each stream signal enter the multi-mode feature mixing converter for the following transformation:

[0023]

[0024] Finally, the hybrid multimodal features F i The following decoupling operation is performed on the characteristic output converter:

[0025]

[0026] Furthermore, S3 specifically includes:

[0027] S31: Randomly select a multi-stream student network from the multi-stream student network group, denoted as... In any sample x from the image sample set, text sample set, and sound sample set i y i , z i The data are then fed into subnetworks of the multi-stream student network M to obtain features. After the S2 multimodal feature converter operation, the converted feature representation is obtained as follows:

[0028] S32: Calculate the deviation between the transformed features of the multi-stream student network M and the s-th multi-stream student network, using the following formula:

[0029]

[0030] S33: Gaussianize the multimodal feature bias using piecewise power transformation, where the expression for Gaussianizing the image feature bias is as follows:

[0031]

[0032] In the formula, γ is an adjustable parameter;

[0033] The expression for Gaussianization of text feature bias is as follows:

[0034]

[0035] The expression for Gaussianizing the sound feature deviation is as follows:

[0036]

[0037] Furthermore, S4 specifically includes:

[0038] S41: For a multi-stream student network M, which is its image subnetwork, establish the following procedural collaborative learning function:

[0039]

[0040] For its text subnetwork, establish the following procedural collaborative learning function:

[0041]

[0042] For its vocal subnetwork, establish the following collaborative learning function:

[0043]

[0044] S42: For a multi-stream student network M, establish the overall objective function for process collaborative learning:

[0045] L = L I +L T +L V .

[0046] The present invention has the following beneficial effects:

[0047] (1) The present invention designs a collaborative learning algorithm, which can realize collaborative learning between multimodal features between student networks, thereby improving the deep interaction between student networks and enhancing the feature representation performance of student networks.

[0048] (2) This invention uses Gaussianization of intermediate hidden layer features to correct the distribution of multimodal features, thereby improving the ability of student networks to resist abnormal samples;

[0049] (3) The online test of this invention uses only one trained student network, without introducing additional parameter capacity, which can improve the model compression efficiency. Attached Figure Description

[0050] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0051] The invention will now be further described with reference to the accompanying drawings.

[0052] This invention discloses a process collaborative learning method based on Gaussianization of multimodal feature bias, comprising the following steps:

[0053] S11: Initialize a total of S multi-stream student networks. The s-th multi-stream student network is represented as... in, For the image subnetwork with parameter θ, For a text subnetwork with parameter μ, For parameters The sound sub-network; S multi-stream student networks form a multi-stream student network group.

[0054] S12: Randomly select a batch of N samples from the image sample set, text sample set, and audio sample set, respectively, where any one sample is represented as x. i y i , z i The values ​​are input into the subnetwork of the s-th multi-stream student network, respectively, and the following features are obtained:

[0055]

[0056] S13: Construct a multimodal feature converter. The multimodal feature converter is a fully connected layer with multiple signal inputs and multiple output signals. The multiple signal inputs receive the feature outputs of each subnetwork of the multi-stream student network, and the multiple signal outputs output the converted multimodal features.

[0057] S14: Represent the multimodal feature converter as Tran = {O} w1 O w2 O w3 B w ,P w1 ,P w2 ,P w3}, where O w1 O w2 O w3 These represent the feature input converters with parameters w1, w2, and w3, respectively. B w For a multimodal feature mixing converter with parameter w, P w1 ,P w2 ,P w3 These represent the characteristic output converters with parameters w1, w2, and w3, respectively.

[0058] S15: The feature first enters the feature input converter to prepare for entering the multimode mixing converter. Each stream signal feature undergoes the following conversion at the input:

[0059]

[0060] Then, the characteristics of each stream signal enter the multi-mode feature mixing converter for the following transformation:

[0061]

[0062] Finally, the hybrid multimodal features F i The following decoupling operation is performed on the characteristic output converter:

[0063]

[0064] S16: Randomly select a multi-stream student network from the multi-stream student network group, denoted as... In any sample x from the image sample set, text sample set, and sound sample set i y i , z i The data are then fed into subnetworks of the multi-stream student network M to obtain features. After the S2 multimodal feature converter operation, the converted feature representation is obtained as follows:

[0065] S17: Calculate the deviation between the transformed features of the multi-stream student network M and the s-th multi-stream student network, using the following formula:

[0066]

[0067] S18: Gaussianize the multimodal feature bias using piecewise power transformation, where the expression for Gaussianizing the image feature bias is as follows:

[0068]

[0069] In the formula, γ is an adjustable parameter;

[0070] The expression for Gaussianization of text feature bias is as follows:

[0071]

[0072] The expression for Gaussianizing the sound feature deviation is as follows:

[0073]

[0074] S19: For a multi-stream student network M, which is its image subnetwork, establish the following procedural collaborative learning function:

[0075]

[0076] For its text subnetwork, establish the following procedural collaborative learning function:

[0077]

[0078] For its vocal subnetwork, establish the following collaborative learning function:

[0079]

[0080] S20: For a multi-stream student network M, establish the overall objective function for process collaborative learning:

[0081] L = L I +L T +L V .

[0082] The implementation of this invention is illustrated using an image classification task as an example. Figure 1 This is a flowchart of the collaborative learning algorithm based on Gaussianization of multimodal feature bias in this invention, as shown below. Figure 1 The specific steps are as follows:

[0083] (1): Initialize a total of S multi-stream student networks. The s-th multi-stream student network is represented as: in For the image subnetwork with parameter θ, For a text subnetwork with parameter μ, For parameters The sound subnetwork. S multi-stream student networks form a multi-stream student network group. A batch of samples, numbered N, is randomly selected from the image sample set, text sample set, and audio sample set, respectively. Any one of these samples is represented as x. i y i , z i The features are respectively input into the subnetwork of the s-th multi-stream student network to obtain the features. as well as

[0084] (2): The feature first enters the input converter. Transformed into It is then fed into a multimodal feature mixing converter to obtain F. si Finally, the output converter decouples it into

[0085] (3): Randomly select a student network M from the multi-stream student network group. This network extracts image features, text features, and sound features. After the multimodal feature converter operation in the second step, Convert to Calculate the deviation between the transformed features of the multi-stream student network M and the s-th multi-stream student network. The piecewise power transform is Gaussianized as follows:

[0086] (4): For a multi-stream student network M, the collaborative learning function L for establishing its image subnetwork is... I The collaborative learning function L for building its text subnetwork T The collaborative learning function L is used to establish the sound subnetwork. V Based on L I L T and L V Establish the overall objective function for collaborative learning and use the steepest Newton descent method to optimize the parameters in the multi-stream student network M.

[0087] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements without departing from the principle of the present invention, and these improvements should also be considered within the scope of protection of the present invention.

Claims

1. A process-based collaborative learning method in the field of computer deep learning model compression based on multimodal feature bias Gaussianization, characterized in that: Includes the following steps: S1: Construct a multi-stream student network group consisting of several multi-stream student networks. Each multi-stream student network includes an image sub-network, a text sub-network, and an audio sub-network. Input the image samples, text samples, and audio samples into the corresponding sub-network of each multi-stream student network to obtain the image, text, and audio feature outputs of each multi-stream student network. S2: The multimodal feature converter, built on a fully connected layer of a neural network, receives the feature output of each multi-stream student network, performs the corresponding feature conversion, and outputs the converted multimodal features. S3: For each student network in a multi-stream student network group, calculate the deviation between the output features of each subnetwork in each multi-stream student network and the output features of the corresponding subnetworks in other multi-stream student networks, and use piecewise power transformation to Gaussianize the deviation; S4: For feature signals using multimodal feature bias Gaussianization, establish a process co-learning function and iteratively optimize the parameters of the multi-stream student network based on the steepest Newton descent method to achieve real-time inference testing of a single trained student network.

2. The process collaborative learning method based on multimodal feature bias Gaussianization in the field of computer deep learning model compression as described in claim 1, characterized in that: S1 specifically includes: S11: Initialize a total of S multi-stream student networks. The s-th multi-stream student network is represented as... ;in, For the image subnetwork with parameter θ, For a text subnetwork with parameter μ, For parameters The sound sub-network; S multi-stream student networks form a multi-stream student network group. ; S12: Randomly select a batch of N samples from the image sample set, text sample set, and audio sample set, respectively. Any one of these samples is represented as... , , The values ​​are then input into the subnetwork of the s-th multi-stream student network, resulting in the following features: ， ， 。 3. The process collaborative learning method based on multimodal feature bias Gaussianization in the field of computer deep learning model compression as described in claim 1, characterized in that: The multimodal feature converter is a fully connected layer with multiple signal inputs and multiple output signals. The multiple signal inputs receive the feature outputs of each subnetwork of the multi-stream student network, and the multiple signal outputs output the converted multimodal features.

4. The process collaborative learning method based on multimodal feature bias Gaussianization in the field of computer deep learning model compression as described in claim 3, characterized in that: S2 specifically includes: S21: Represent the multimodal feature converter as ,in The parameters are respectively , , Feature input converter, For parameters A multimodal feature mixing converter, The parameters are respectively , , Feature output converter; S22: The feature first enters the feature input converter to prepare for entering the multimode mixing converter. Each stream signal feature undergoes the following conversion at the input: ， ， ， Then, the characteristics of each stream signal enter the multi-mode feature mixing converter for the following transformation: ， Finally, hybrid multimodal features The following decoupling operation is performed on the characteristic output converter: ， ， 。 5. The process collaborative learning method based on multimodal feature bias Gaussianization in the field of computer deep learning model compression as described in claim 4, characterized in that: S3 specifically includes: S31: Randomly select a multi-stream student network from the multi-stream student network group, denoted as... In any sample from the image sample set, text sample set, and audio sample set , , The data are then fed into subnetworks of the multi-stream student network M to obtain features. , , After the S2 multimodal feature converter operation, the converted feature representation is obtained as follows: , , ; S32: Calculate the deviation between the transformed features of the multi-stream student network M and the s-th multi-stream student network, using the following formula: ， ， ；， S33: Gaussianize the multimodal feature bias using piecewise power transformation, where the expression for Gaussianizing the image feature bias is as follows: ； In the formula, These are adjustable parameters; The expression for Gaussianization of text feature bias is as follows: ； The expression for Gaussianizing the sound feature deviation is as follows: 。 6. The process collaborative learning method based on multimodal feature bias Gaussianization in the field of computer deep learning model compression as described in claim 5, characterized in that: S4 specifically includes: S41: For a multi-stream student network M, which is its image subnetwork, establish the following procedural collaborative learning function: ， For its text subnetwork, establish the following procedural collaborative learning function: ， For its vocal subnetwork, establish the following collaborative learning function: ， S42: For a multi-stream student network M, establish the overall objective function for process collaborative learning: 。

Images