Model training method and apparatus based on multi-modal data, and device and storage medium

Abstract

Disclosed in the present application are a model training method and apparatus based on multi-modal data, and a device and a storage medium. The method comprises: acquiring training sample sets (201); performing acoustic feature extraction on an acoustic sample comprised in a first training sample set, and performing optical feature extraction on an optical sample comprised therein (202); acquiring a fusion weight for multi-modal fusion (203); and under the constraint of the fusion weight, performing model training on the basis of labeling information of the training sample sets, and an extracted acoustic feature and optical feature, so as to obtain a multi-modal fusion recognition model (204).

Description

Model training method, device, equipment and storage medium based on multimodal data

[0001] This application claims priority to Chinese patent application No. 202311809192.6, filed on December 26, 2023, entitled “Model training method, device, equipment and storage medium based on multimodal data”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus, device and storage medium based on multimodal data. Background Art

[0003] In recent years, with the rapid development of artificial intelligence (AI), machine learning, as a key branch of AI, has been applied in numerous fields. Currently, various machine learning methods often perform NLP (natural language processing) tasks based on single-modal data. For example, NLP tasks are performed solely on optical data (such as images and videos) or acoustic data (such as speech). Summary of the Invention

[0004] The present invention provides a method, apparatus, device, and storage medium for model training based on multimodal data. The technical solution is as follows:

[0005] In one aspect, a model training method based on multimodal data is provided, the method comprising:

[0006] Acquire a first training sample set; wherein the first training sample set includes a labeled acousto-optic sample combination, an unlabeled acoustic sample, and an unlabeled optical sample; and one of the acousto-optic sample combinations includes an acoustic sample and an optical sample;

[0007] performing acoustic feature extraction on the acoustic samples included in the first training sample set;

[0008] performing optical feature extraction on the optical samples included in the first training sample set;

[0009] Obtaining fusion weights for multimodal fusion; wherein the fusion weights are dynamically adjusted based on model training parameters and the external environment during data collection;

[0010] Under the constraint of the fusion weight, model training is performed based on the annotation information of the first training sample set and the extracted acoustic features and optical features to obtain a multimodal fusion recognition model.

[0011] In another aspect, a model training device based on multimodal data is provided, the device comprising:

[0012] A first acquisition module is configured to acquire a first training sample set; wherein the first training sample set includes a labeled acoustic and optical sample combination, an unlabeled acoustic sample, and an unlabeled optical sample; and one acoustic and optical sample combination includes an acoustic sample and an optical sample;

[0013] a first feature extraction module, configured to extract acoustic features from the acoustic samples included in the first training sample set;

[0014] a second feature extraction module, configured to extract optical features from the optical samples included in the first training sample set;

[0015] A second acquisition module is configured to acquire fusion weights for multimodal fusion; wherein the fusion weights are dynamically adjusted according to model training parameters and the external environment during data collection;

[0016] The training module is configured to perform model training based on the annotation information of the first training sample set, the extracted acoustic features and the extracted optical features under the constraint of the fusion weight to obtain a multimodal fusion recognition model.

[0017] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the above-mentioned model training method based on multimodal data.

[0018] On the other hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the above-mentioned model training method based on multimodal data.

[0019] On the other hand, a computer program product or computer program is provided, which includes a computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the above-mentioned model training method based on multimodal data.

[0020] The embodiment of the present application trains a model based on multimodal data, realizing multimodal fusion recognition based on acoustic input and optical input. Since multimodal data can provide richer and more comprehensive information understanding and processing capabilities than single modality data, the model trained based on multimodal data has a wider range of applications. For example, it can adapt to complex environments such as noisy or low-light environments, ensuring the task processing effect when executing task processing. In other words, the multimodal fusion recognition model trained by the embodiment of the present application can achieve efficient recognition and accurate response in complex environments, improving the applicability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] FIG1 is a schematic diagram of an implementation environment involved in a model training method based on multimodal data provided in an embodiment of the present application;

[0022] FIG2 is a flow chart of a model training method based on multimodal data provided in an embodiment of the present application;

[0023] FIG3 is a flowchart of another model training method based on multimodal data provided in an embodiment of the present application;

[0024] FIG4 is a schematic diagram of the structure of a model training device based on multimodal data provided in an embodiment of the present application;

[0025] FIG5 is a schematic structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The first point that needs to be clarified is that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of the relevant regions.

[0027] The second point that needs to be explained is that the pre-trained model mentioned in this article is also called a large language model, a large language model or a large model, and this application does not limit this.

[0028] As mentioned earlier, various machine learning methods often perform NLP tasks based on single-modal data. However, the data humans receive and process in daily life is often multimodal. For example, when communicating, we convey information not only through words but also through facial expressions, gestures, and body language.

[0029] In order to enable machines to better imitate and understand similar complex communication methods, it becomes crucial to perform NLP tasks based on multimodal data.

[0030] Acoustics and optics are two common and information-rich modalities. Acoustic data plays a key role in many applications, such as speech recognition, voiceprint recognition, and noise identification. Optical data also plays a vital role in areas such as image recognition and object tracking.

[0031] To summarize, since task processing based on single-modal data is usually difficult to adapt to complex environments, the embodiment of the present application proposes a model training scheme based on multimodal data, which can integrate acoustic data and optical data, train the model through pre-training strategies and optimization methods, and provide more accurate and efficient recognition and processing capabilities for complex environments.

[0032] The following implementation method is used to introduce the model training scheme based on multimodal data in detail.

[0033] FIG1 is a schematic diagram of an implementation environment involved in a model training method based on multimodal data provided in an embodiment of the present application.

[0034] Referring to Figure 1 , during the training phase, a first computer device 110 performs model training based on multimodal data to obtain a pre-trained model. This pre-trained model is also referred to herein as a multimodal fusion recognition model. During the inference phase, a second computer device 120 uses the fine-tuned model to perform NLP tasks. Fine-tuning refers to retraining the pre-trained model based on a small number of labeled training samples to apply the trained model to a specific task.

[0035] It should be noted that the first computer device 110 and the second computer device 120 are computer devices with machine learning capabilities. The first computer device 110 and the second computer device 120 can be the same device, or the first computer device 110 and the second computer device 120 can be different devices, which is not limited in this application.

[0036] Based on the above implementation environment, an embodiment of the present application provides a pre-training system for multimodal fusion recognition. Exemplarily, the system includes an acoustic input module, an optical input module, and a fusion recognition module.

[0037] Among them, the acoustic input module is used to extract the acoustic features of acoustic samples; the optical input module is used to extract the optical features of optical samples; and the fusion recognition module is used to perform multimodal fusion recognition based on the extracted acoustic features and optical features.

[0038] FIG2 is a flow chart of a model training method based on multimodal data provided in an embodiment of the present application. The execution subject of the method is a computer device, such as the first computer device in FIG1. ​​Referring to FIG2, the method flow provided in an embodiment of the present application includes the following steps.

[0039] 201. A computer device obtains a first training sample set; wherein the first training sample set includes a labeled acousto-optic sample combination, an unlabeled acoustic sample, and an unlabeled optical sample; and an acousto-optic sample combination includes an acoustic sample and an optical sample.

[0040] An acoustic sample can be a sound, and an optical sample can be an image or a video. Acoustic samples can be captured using a high-sensitivity microphone array, and optical samples can be captured using a high-resolution camera, which is not limited in this application.

[0041] In addition, during the process of collecting training samples, a minimum length of acoustic samples and optical samples in video form, or a minimum resolution of optical samples in image form, may be set, which is not limited in this application.

[0042] It should be noted that the first training sample set is used in the pre-training phase. In this first training sample set, the number of labeled samples is much smaller than the number of unlabeled samples. In other words, this training sample set includes a small number of labeled acoustic and optical sample combinations for cross-modal association learning, and a large number of unlabeled acoustic and optical samples for self-supervised learning.

[0043] 202. The computer device extracts acoustic features from the acoustic samples included in the first training sample set and extracts optical features from the optical samples.

[0044] This step is performed by the acoustic input module and the optical input module. Regarding acoustic feature extraction, the extracted acoustic features can be MFCC (Mel Frequency Cepstrum Coefficient) features or spectrograms, which are not limited in this application. The horizontal axis of the spectrogram is time, the vertical axis is frequency, and the coordinate point value is the sound data energy.

[0045] Taking the extracted acoustic features as MFCC features as an example, the MFCC feature extraction process includes: preprocessing (such as pre-emphasis, framing, windowing or denoising, etc.), fast Fourier transform, Mel filter bank, logarithmic operation, discrete cosine transform and dynamic feature extraction, etc.

[0046] For optical feature extraction, taking an optical sample as an image, image preprocessing is first performed. The purpose of preprocessing is to suppress unwanted deformation or enhance certain image features that are important for subsequent processing. For example, image preprocessing includes histogram equalization to enhance image contrast.

[0047] In one possible implementation, the embodiment of the present application extracts image features through a convolutional neural network.

[0048] Convolutional neural networks (CNNs) effectively extract features from images through a variety of methods, including convolution, feature mapping, and pooling. Through layer-by-layer abstraction, CNNs can gradually understand low-level features and high-level concepts, revealing the rich information behind the image.

[0049] In other words, a convolutional neural network typically consists of multiple convolutional layers and fully connected layers. By stacking them layer by layer, the network can gradually abstract higher-level features. Shallow convolutional layers can capture low-level features of the image (such as edges and corners), while deep convolutional layers can capture more abstract features (such as the shape and texture of objects).

[0050] 203. The computer device obtains a fusion weight for multimodal fusion; wherein the fusion weight is dynamically adjusted according to the model training parameters and the external environment during data collection.

[0051] This step is performed by the fusion recognition module. In the embodiment of the present application, the fusion strategy includes feature-level fusion (also called early fusion) and decision-level fusion (also called late fusion).

[0052] For feature-level fusion, after extracting the acoustic and optical features, they are fused at the feature level. Assuming that the extracted acoustic feature is A and the extracted optical feature is B, the fusion weights for multimodal fusion include w A and w B Among them, w A Corresponding to the extracted acoustic features, w B Corresponding to the extracted optical features.

[0053] For decision-level fusion, after extracting the acoustic and optical features, the acoustic and optical subsystems make decisions separately, and then fuse them at the decision level. The advantage of this fusion method is that each modality retains its independent decision-making ability.

[0054] Assuming that DA is the decision output based on acoustic features and DB is the decision output based on optical features, the fusion weights for multimodal fusion include w DA and w DB Among them, w DA Corresponding to the decision output based on acoustic features, wDB Corresponding to the decision output based on optical features.

[0055] The first point to be made is that A and w DA Also referred to as the first fusion weight in this paper, w B and w DB It is also referred to as the second fusion weight in this article.

[0056] The second point that needs to be explained is that A and w B These two weights are used to control the importance of acoustic feature A and optical feature B. For example, the embodiment of the present application sets w according to the degree of trust in the two inputs. A and w B If the acoustic features are considered to be more important in the initial stage, w A Set to a larger value, and w B Set it to a smaller value and vice versa.

[0057] The third point that needs to be explained is that DA and w DB These two weights are used to adjust the reliance of the fusion recognition module on each modality data. For example, the embodiment of the present application sets w according to the initial trust level of each modality data. DA and w DB If the acoustic data is considered to be more critical in the overall fusion, w DA Set to a larger value, and w DB Set it to a smaller value and vice versa.

[0058] In an embodiment of the present application, the fusion weights used for multimodal fusion can be dynamically adjusted based on model training parameters and the external environment during data collection. For example, the model training parameter is a learning rate, and the external environment can be the noise level of the current environment, which is not limited in this application.

[0059] 204. Under the constraint of the fusion weight, the computer device performs model training based on the labeling information of the first training sample set and the extracted acoustic features and optical features to obtain a multimodal fusion recognition model.

[0060] In the embodiment of the present application, in order to ensure that the model can accurately perform fusion recognition, a loss function L is defined, which calculates the loss value based on the fused output and the true label:

[0061] Where N is the total number of samples included in the first training sample set; y iis the true label, that is, the annotation information of the first training sample set; It is the fused output, which is obtained based on the extracted acoustic and optical features.

[0062] For example, the goal of model training is to minimize the loss value of this loss function, thereby achieving more accurate fusion recognition. In addition, in order to achieve the best fusion recognition effect, the embodiment of the present application also uses stochastic gradient descent or Adam optimizer to minimize this loss function. At the same time, early stopping strategy and learning rate annealing technology are also introduced to prevent overfitting and accelerate model convergence.

[0063] The embodiment of the present application trains a model based on multimodal data, realizing multimodal fusion recognition based on acoustic input and optical input. Since multimodal data can provide richer and more comprehensive information understanding and processing capabilities than single modality data, the model trained based on multimodal data has a wider range of applications. For example, it can adapt to complex environments such as noisy or low-light environments, ensuring the task processing effect when executing task processing. In other words, the multimodal fusion recognition model trained by the embodiment of the present application can achieve efficient recognition and accurate response in complex environments, improving the applicability and robustness of the model.

[0064] The above briefly introduces the model training scheme provided in the embodiment of the present application. The following is a detailed introduction to the model training scheme based on multimodal data provided in the embodiment of the present application in conjunction with Figure 3.

[0065] FIG3 is a flow chart of another model training method based on multimodal data provided in an embodiment of the present application. The execution subject of the method is a computer device, such as the first computer device in FIG1. ​​Referring to FIG3, the method flow provided in an embodiment of the present application includes the following steps.

[0066] 301. A computer device obtains a first training sample set; wherein the first training sample set includes a labeled acousto-optic sample combination, an unlabeled acoustic sample, and an unlabeled optical sample; and an acousto-optic sample combination includes an acoustic sample and an optical sample.

[0067] This step can refer to the aforementioned step 201 and will not be repeated here. In addition, in order to enhance the robustness of the model, the embodiment of the present application will also perform data enhancement on the originally collected training samples, as detailed in the following step 302.

[0068] 302. The computer device performs sample expansion on the first training sample set, and adds the expanded acoustic samples and optical samples to the first training sample set to obtain an updated first training sample set.

[0069] In one possible implementation, the embodiment of the present application performs sample expansion in the following manner:

[0070] 3021. Perform first data enhancement on the acoustic samples included in the first training sample set to obtain expanded acoustic samples.

[0071] Exemplarily, the first data enhancement includes at least: changing the pitch and changing the speaking speed, which is not limited in this application.

[0072] 3022. Perform second data enhancement on the optical samples included in the first training sample set to obtain expanded optical samples.

[0073] Exemplarily, the second data enhancement includes at least: image rotation, image flipping, image scaling, and image cropping, which is also not limited in this application.

[0074] 3023. Add the expanded acoustic samples and the expanded optical samples to the first training sample set to obtain an updated first training sample set. That is, perform model training based on the updated first training sample set.

[0075] 303. The computer device extracts acoustic features from the acoustic samples included in the updated first training sample set and extracts optical features from the optical samples.

[0076] This step can refer to the aforementioned step 202 and will not be repeated here.

[0077] 304. The computer device obtains a fusion weight for multimodal fusion; wherein the fusion weight is dynamically adjusted according to the model training parameters and the external environment during data collection.

[0078] In an embodiment of the present application, the environmental information during data collection can be obtained by external sensors or by analyzing the collected acoustic data or optical data, and this application does not limit this. After obtaining the environmental information, the embodiment of the present application can dynamically adjust the fusion weight based on the obtained environmental information and model training parameters. For example, when the above-mentioned environmental information indicates that the external environment during data collection is a high-noise environment, the acoustic data may not be very reliable, so the fusion weight corresponding to the optical data can be increased.

[0079] It should be noted that, for the first training sample set consisting of acoustic samples and optical samples collected in the same environment, the fusion weight includes a first fusion weight corresponding to the acoustic sample and a second fusion weight corresponding to the optical sample.

[0080] Based on the above description, the embodiments of the present application obtain fusion weights for multimodal fusion through the following two methods.

[0081] Method 1: In response to the external environment being a noisy environment when data collection is performed, a weight adjustment value δ is determined based on the ambient noise level when data collection is performed; then, the second fusion weight is adjusted according to the model training parameters and the weight adjustment value δ to obtain a new second fusion weight.

[0082] The new second fusion weight is greater than the previous second fusion weight. This means that in a noisy environment, the δ value will increase the weight corresponding to the optical sample. For example, embodiments of the present application analyze acoustic samples using a fast Fourier transform to determine whether the environment is noisy; or analyze optical samples using optical flow change analysis to determine whether the environment is noisy. After determining the ambient noise level, the δ value is then determined based on the ambient noise level. The greater the ambient noise, the larger the δ value.

[0083] In a possible implementation, the embodiment of the present application implements weight adjustment through a feedback mechanism, which is expressed by a calculation formula: new =w old +α*δ

[0084] Among them, w new Refers to the new weight, w old refers to the original weight and α refers to the learning rate.

[0085] Method 2: Obtain the signal quality of the acoustic sample and the signal quality of the optical sample; determine the weight adjustment value δ based on the obtained signal quality; then, adjust the target fusion weight according to the model training parameters and the weight adjustment value δ to obtain a new target fusion weight.

[0086] The target fusion weight corresponds to the sample with good signal quality among the acoustic samples and the optical samples. The new target fusion weight is greater than the target fusion weight before adjustment.

[0087] In this approach, the δ value is calculated by detecting the signal quality of acoustic samples or optical samples, with the goal of increasing the weight corresponding to the sample of the modality with better signal quality.

[0088] Exemplarily, the signal quality of the acoustic sample is blurriness, signal-to-noise ratio, etc., and the signal quality of the optical sample is clarity, brightness, etc., which is not limited in this application.

[0089] 305. Under the constraint of the fusion weight, the computer device performs model training based on the updated annotation information of the first training sample set and the extracted acoustic features and optical features to obtain a multimodal fusion recognition model.

[0090] In one possible implementation, for feature-level fusion, the model training process includes:

[0091] Step a: Under the constraint of the fusion weight, the extracted acoustic features and optical features are feature-connected to obtain fusion features.

[0092] For example, the present embodiment utilizes a fully connected layer to connect acoustic and optical features. Connecting features through a fully connected layer can introduce more parameters and nonlinear activation functions, allowing for more complex feature fusion and helping the model learn more complex patterns and relationships.

[0093] Step b: input the fusion feature into the recognition network for recognition, and obtain the prediction result output by the recognition network.

[0094] The recognition network includes a deep neural network and a nonlinear transformation layer. For example, the deep neural network is a multi-layer perceptron, and the nonlinear transformation layer performs nonlinear transformation via a ReLU activation function, which is not limited in this application.

[0095] Step c: construct a first loss function based on the annotation information and prediction results of the first training sample set; iteratively obtain the loss value through the first loss function until the training stop condition is met, thereby obtaining a multimodal fusion recognition model.

[0096] Among them, the label information of the first training sample set is y i , that is, the true label; the above prediction result is , that is, the fused output. In addition, the above training stop condition can be that the loss value calculated by the first loss function is the minimum, which is not limited in this application.

[0097] In another possible implementation, for decision-level fusion, the model training process includes:

[0098] Step 1: Input the extracted acoustic features into the first recognition network for recognition, and obtain the prediction results output by the first recognition network; and input the extracted optical features into the second recognition network for recognition, and obtain the prediction results output by the second recognition network.

[0099] The first recognition network includes a deep neural network and a nonlinear transformation layer for independent recognition in the acoustic mode. The second recognition network also includes a deep neural network and a nonlinear transformation layer for independent recognition in the optical mode.

[0100] Step 2: Under the constraint of the fusion weight, the prediction results output by the first recognition network and the prediction results output by the second recognition network are fused, and the fusion results are passed through a nonlinear transformation layer to obtain the fused prediction results.

[0101] This step is used to perform weighted summation of the prediction results output by the two recognition networks using the fusion weights, thereby obtaining a fusion result. For example, the final decision output D is described by the following calculation formula: D = σ(w DA *DA+w DB *DB)

[0102] Here, σ refers to the activation function, such as the Sigmoid function or the Softmax function.

[0103] Step 3: Based on the annotation information of the first training sample set and the fused prediction results, a second loss function is constructed; the loss value is iteratively obtained through the second loss function until the training stop condition is met, thereby obtaining a multimodal fusion recognition model.

[0104] Among them, the label information of the first training sample set is y i , that is, the true label; the above prediction result is In addition, the above training stop condition can be that the loss value calculated by the second loss function is the minimum, which is not limited in this application.

[0105] For example, a multi-task learning approach may be used during the training process, that is, the recognition task for the acoustic modality and the recognition task for the optical modality are combined into a joint training task, which is not limited in this application.

[0106] In another possible implementation, the embodiment of the present application uses knowledge distillation technology to migrate the knowledge of a large model into a smaller and more efficient model.

[0107] Among them, the embodiment of the present application uses a multimodal fusion recognition model as a teacher model.

[0108] During the training process of the student model, the embodiment of the present application determines a weight factor for each training sample according to the sample difficulty and feature importance of each training sample in the second training sample set.

[0109] Exemplarily, the second training sample set and the first training sample set are the same sample set, or the second training sample set is a subset of the first training sample set, which is not limited in this application.

[0110] It should be noted that the weight factor is used to reflect the degree of attention paid by the student model to the output of the teacher model during the knowledge distillation process. This helps to improve the efficiency of knowledge transfer and can adapt to different types of samples more flexibly. For each training sample, the embodiment of the present application determines the sample difficulty of the training sample based on the difference between the output probability distribution of the teacher model for the training sample and the annotation information. In addition, feature importance can be evaluated using feature selection or similar methods, which is not limited in this application.

[0111] Next, under the constraint of the weight factor, the model is trained based on the annotation information of each training sample and the output probability distribution of the teacher model to obtain the student model for multimodal fusion recognition.

[0112] Exemplarily, during the training process of the student model, an embodiment of the present application obtains a soft target loss based on the output probability distribution of the teacher model and the predicted output of the student model; and obtains a hard target loss based on the labeling information of each training sample and the predicted output of the student model; then, under the constraint of the weight factor, a third loss function is constructed based on the soft target loss and the hard target loss; and then the loss value is iteratively obtained through the third loss function until the training stop condition is met to obtain the student model.

[0113] Among them, the third loss function is the cross entropy loss function, which is not limited in this application.

[0114] The embodiment of the present application trains a model based on multimodal data, realizing multimodal fusion recognition based on acoustic input and optical input. Since multimodal data can provide richer and more comprehensive information understanding and processing capabilities than single modality data, the model trained based on multimodal data has a wider range of applications. For example, it can adapt to complex environments such as noisy or low-light environments, ensuring the task processing effect when executing task processing. In other words, the multimodal fusion recognition model trained by the embodiment of the present application can achieve efficient recognition and accurate response in complex environments, improving the applicability and robustness of the model.

[0115] FIG4 is a schematic diagram of the structure of a model training device based on multimodal data provided in an embodiment of the present application. Referring to FIG4 , the device includes:

[0116] The first acquisition module 401 is configured to acquire a first training sample set; wherein the first training sample set includes a labeled acoustic and optical sample combination, an unlabeled acoustic sample, and an unlabeled optical sample; an acoustic and optical sample combination includes an acoustic sample and an optical sample;

[0117] A first feature extraction module 402 is configured to extract acoustic features from acoustic samples included in the first training sample set;

[0118] The second feature extraction module 403 is configured to extract optical features from the optical samples included in the first training sample set;

[0119] The second acquisition module 404 is configured to acquire fusion weights for multimodal fusion; wherein the fusion weights are dynamically adjusted according to model training parameters and the external environment during data collection;

[0120] The training module 405 is configured to perform model training based on the annotation information of the first training sample set, the extracted acoustic features and the extracted optical features under the constraint of the fusion weight to obtain a multimodal fusion recognition model.

[0121] The embodiment of the present application trains a model based on multimodal data, realizing multimodal fusion recognition based on acoustic input and optical input. Since multimodal data can provide richer and more comprehensive information understanding and processing capabilities than single modality data, the model trained based on multimodal data has a wider range of applications. For example, it can adapt to complex environments such as noisy or low-light environments, ensuring the task processing effect when executing task processing. In other words, the multimodal fusion recognition model trained by the embodiment of the present application can achieve efficient recognition and accurate response in complex environments, improving the applicability and robustness of the model.

[0122] In one possible implementation, for the acoustic samples and optical samples collected in the same environment in the first training sample set, the fusion weight includes a first fusion weight corresponding to the acoustic sample and a second fusion weight corresponding to the optical sample; the second acquisition module 404 is configured to:

[0123] In response to the external environment when the data is collected being a noisy environment, determining a weight adjustment value based on an environmental noise level when the data is collected;

[0124] Adjust the second fusion weight according to the model training parameters and the weight adjustment value to obtain a new second fusion weight;

[0125] The new second fusion weight is greater than the second fusion weight before adjustment.

[0126] In one possible implementation, for the acoustic samples and optical samples collected in the same environment in the first training sample set, the fusion weight includes a first fusion weight corresponding to the acoustic sample and a second fusion weight corresponding to the optical sample; the second acquisition module 404 is configured to:

[0127] obtaining the signal quality of the acoustic sample and the signal quality of the optical sample;

[0128] Determining a weight adjustment value based on the acquired signal quality;

[0129] According to the model training parameters and weight adjustment values, the target fusion weight is adjusted to obtain a new target fusion weight;

[0130] Among them, the new target fusion weight is greater than the target fusion weight before adjustment; the target fusion weight corresponds to the sample with good signal quality among the acoustic samples and the optical samples.

[0131] In a possible implementation, the first obtaining module 401 is further configured to:

[0132] Performing first data enhancement on the acoustic samples included in the first training sample set to obtain expanded acoustic samples; wherein the first data enhancement at least includes: changing the pitch and changing the speech speed;

[0133] Performing second data enhancement on the optical samples included in the first training sample set to obtain expanded optical samples; wherein the second data enhancement at least includes: image rotation, image flipping, image scaling, and image cropping;

[0134] The augmented acoustic samples and the augmented optical samples are added to the first training sample set.

[0135] In a possible implementation, the training module 405 is configured to:

[0136] Under the constraint of fusion weight, the extracted acoustic features and optical features are connected to obtain fusion features;

[0137] Input the fused features into the recognition network for recognition, and obtain the prediction results output by the recognition network;

[0138] Constructing a first loss function based on the labeling information and prediction results of the first training sample set;

[0139] The loss value is obtained by iteratively obtaining the first loss function until the training stop condition is met, and a multimodal fusion recognition model is obtained.

[0140] In a possible implementation, the training module 405 is configured to:

[0141] Inputting the extracted acoustic features into the first recognition network for recognition, and obtaining a prediction result output by the first recognition network;

[0142] Inputting the extracted optical features into the second recognition network for recognition, and obtaining a prediction result output by the second recognition network;

[0143] Under the constraint of the fusion weight, the prediction results output by the first recognition network and the prediction results output by the second recognition network are fused, and the fusion results are passed through a nonlinear transformation layer to obtain a fused prediction result;

[0144] Constructing a second loss function based on the labeled information of the first training sample set and the fused prediction results;

[0145] The loss value is obtained by iteratively obtaining the second loss function until the training stop condition is met, and a multimodal fusion recognition model is obtained.

[0146] In a possible implementation, the training module 405 is further configured to:

[0147] Use the multimodal fusion recognition model as the teacher model;

[0148] During the training of the student model, a weight factor is determined based on the sample difficulty and feature importance of each training sample in the second training sample set. The weight factor is used to reflect the degree of attention paid by the student model to the output of the teacher model during the knowledge distillation process.

[0149] Under the constraint of the weight factor, model training is performed based on the labeling information of each training sample in the second training sample set and the output probability distribution of the teacher model to obtain a student model for multimodal fusion recognition.

[0150] In a possible implementation, the training module 405 is further configured to:

[0151] Obtain the soft target loss based on the output probability distribution of the teacher model and the predicted output of the student model;

[0152] Obtaining a hard target loss based on the labeling information of each training sample in the second training dataset and the predicted output of the student model;

[0153] Under the constraint of the weight factor, a third loss function is constructed based on the soft target loss and the hard target loss;

[0154] The loss value is obtained by iteratively obtaining the third loss function until the training stop condition is met to obtain the student model.

[0155] In a possible implementation, the training module 405 is further configured to:

[0156] For each training sample in the second training sample set, the sample difficulty of the training sample is determined according to the difference between the output probability distribution of the teacher model for the training sample and the labeling information of the training sample.

[0157] All of the above optional technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described in detail here.

[0158] It should be noted that the model training device based on multimodal data provided in the above embodiment only uses the division of the above functional modules as an example when performing model training. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model training device based on multimodal data provided in the above embodiment and the model training method based on multimodal data are of the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0159] FIG5 is a schematic structural diagram of a computer device 500 provided in an embodiment of the present application.

[0160] The computer 500 may be a server. The computer device 500 may have relatively large differences due to different configurations or performances, and includes one or more processors (Central Processing Units, CPU) 501 and one or more memories 502, wherein at least one program code is stored in the memory 502, and at least one program code is loaded and executed by the processor 501 to implement the model training method based on multimodal data provided by the above-mentioned various method embodiments. Of course, the computer device 500 also has components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The computer device 500 also includes other components for realizing the functions of the device, which will not be described in detail here.

[0161] In an exemplary embodiment, a computer-readable storage medium is further provided, such as a memory including program code, which can be executed by a processor in a computer device to perform the model training method based on multimodal data in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0162] In an exemplary embodiment, a computer program product or computer program is also provided, which includes a computer program code, which is stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device performs the above-mentioned model training method based on multimodal data.

[0163] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0164] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A model training method based on multi-modal data, the method comprising: Obtaining a first training sample set; wherein, the first training sample set includes labeled acoustic-optical sample combinations, unlabeled acoustic samples, and unlabeled optical samples; one acoustic-optical sample combination includes one acoustic sample and one optical sample; Performing acoustic feature extraction on the acoustic samples included in the first training sample set; Performing optical feature extraction on the optical samples included in the first training sample set; Obtaining a fusion weight for multi-modal fusion; wherein, the fusion weight is dynamically adjusted according to model training parameters and the external environment during data collection; Under the constraint of the fusion weight, performing model training based on the annotation information, the extracted acoustic features, and the optical features of the first training sample set to obtain a multi-modal fusion recognition model.

2. The method according to claim 1, wherein, For the acoustic samples and optical samples collected in the same environment in the first training sample set, the fusion weight includes a first fusion weight corresponding to the acoustic sample and a second fusion weight corresponding to the optical sample; The obtaining of the fusion weight for multi-modal fusion includes: In response to the external environment during data collection being a noisy environment, determining a weight adjustment value based on the environmental noise level during data collection; Adjusting the second fusion weight according to the model training parameters and the weight adjustment value to obtain a new second fusion weight; Wherein, the new second fusion weight is greater than the second fusion weight before adjustment.

3. The method according to claim 1, wherein, For the acoustic samples and optical samples collected in the same environment in the first training sample set, the fusion weight includes a first fusion weight corresponding to the acoustic sample and a second fusion weight corresponding to the optical sample; The obtaining of the fusion weight for multi-modal fusion includes: Obtaining the signal quality of the acoustic sample and the signal quality of the optical sample; Determining a weight adjustment value based on the obtained signal quality; Adjusting the target fusion weight according to the model training parameters and the weight adjustment value to obtain a new target fusion weight; Wherein, the new target fusion weight is greater than the target fusion weight before adjustment; the target fusion weight corresponds to the sample with good signal quality among the acoustic sample and the optical sample.

4. The method according to claim 1, wherein The method further includes: Performing first data augmentation on the acoustic samples included in the first training sample set to obtain augmented acoustic samples; wherein, the first data augmentation at least includes: changing the pitch, changing the speech rate; Performing second data augmentation on the optical samples included in the first training sample set to obtain augmented optical samples; wherein, the second data augmentation at least includes: image rotation, image flipping, image scaling, and image cropping; Adding the augmented acoustic samples and the augmented optical samples to the first training sample set.

5. The method according to claim 1, wherein The performing of model training based on the annotation information, the extracted acoustic features, and the optical features of the first training sample set under the constraint of the fusion weight to obtain a multi-modal fusion recognition model includes: Under the constraint of the fusion weights, concatenate the extracted acoustic features and optical features to obtain fused features; Input the fused features into an identification network for identification, and obtain the prediction result output by the identification network; Construct a first loss function based on the annotation information of the first training sample set and the prediction result; Iteratively obtain the loss value through the first loss function until the training stop condition is satisfied, and obtain the multi-modal fusion identification model.

6. The method according to claim 1, wherein, The method of training a multi-modal fusion identification model based on the annotation information of the first training sample set, the extracted acoustic features and optical features under the constraint of the fusion weights includes: Input the extracted acoustic features into a first identification network for identification, and obtain the prediction result output by the first identification network; Input the extracted optical features into a second identification network for identification, and obtain the prediction result output by the second identification network; Under the constraint of the fusion weights, fuse the prediction result output by the first identification network and the prediction result output by the second identification network, and pass the fusion result through a non-linear transformation layer to obtain the fused prediction result; Construct a second loss function based on the annotation information of the first training sample set and the fused prediction result; Iteratively obtain the loss value through the second loss function until the training stop condition is satisfied, and obtain the multi-modal fusion identification model.

7. The method according to any one of claims 1 to 6, wherein The method further includes: Use the multi-modal fusion identification model as a teacher model; During the training process of the student model, determine a weight factor according to the sample difficulty and feature importance of each training sample in the second training sample set; wherein, the weight factor is used to reflect the attention degree of the student model to the output of the teacher model during the knowledge distillation process; Under the constraint of the weight factor, train the model based on the annotation information of each training sample in the second training sample set and the output probability distribution of the teacher model, and obtain the student model for multi-modal fusion identification.

8. The method according to claim 7, wherein, The method of training the student model for multi-modal fusion identification based on the annotation information of each training sample in the second training sample set and the output probability distribution of the teacher model under the constraint of the weight factor includes: Obtain a soft target loss based on the output probability distribution of the teacher model and the predicted output of the student model; Obtain a hard target loss based on the annotation information of each training sample in the second training data set and the predicted output of the student model; Under the constraint of the weight factor, construct a third loss function based on the soft target loss and the hard target loss; Iteratively obtain the loss value through the third loss function until the training stop condition is satisfied, and obtain the student model.

9. The method according to claim 7, wherein The method further includes: For each training sample in the second training sample set, determine the sample difficulty of the training sample according to the difference between the output probability distribution of the teacher model for the training sample and the annotation information of the training sample.

10. A model training device based on multi-modal data, the device includes: A first acquisition module, configured to acquire a first training sample set; wherein, the first training sample set includes labeled acoustic-optical sample combinations, unlabeled acoustic samples, and unlabeled optical samples; one acoustic-optical sample combination includes one acoustic sample and one optical sample; A first feature extraction module, configured to perform acoustic feature extraction on the acoustic samples included in the first training sample set; A second feature extraction module, configured to perform optical feature extraction on the optical samples included in the first training sample set; A second acquisition module, configured to acquire fusion weights for multimodal fusion; wherein, the fusion weights are dynamically adjusted according to model training parameters and the external environment during data acquisition; A training module, configured to perform model training based on the annotation information, extracted acoustic features, and optical features of the first training sample set under the constraint of the fusion weights to obtain a multimodal fusion recognition model.

11. A computer device, the device includes a processor and a memory, and at least one program code is stored in the memory, and the at least one program code is loaded and executed by the processor to implement the model training method based on multimodal data according to any one of claims 1 to 9.

12. A computer-readable storage medium, at least one program code is stored in the storage medium, and the at least one program code is loaded and executed by a processor to implement the model training method based on multimodal data according to any one of claims 1 to 9.

Images