Robust multimodal emotion recognition method and device, computer equipment, readable storage medium and program product

Abstract

The application relates to a robust multi-modal emotion recognition method and device, computer equipment, a computer readable storage medium and a computer program product, and relates to the technical field of natural language processing. The method comprises the following steps: obtaining to-be-recognized corpus under a target scene; performing speech recognition processing on the to-be-recognized corpus to obtain to-be-recognized text; inputting the to-be-recognized corpus and the to-be-recognized text into a pre-trained multi-modal emotion recognition model to obtain an emotion label corresponding to the to-be-recognized corpus; the pre-trained multi-modal emotion recognition model is trained through implicit feature reconstruction and feature alignment of a preset corpus library; the preset corpus library comprises clean corpus and noisy corpus; the implicit feature reconstruction is used for complementing damaged data of the noisy corpus relative to the clean corpus; and the feature alignment is used for reducing distribution differences of feature spaces of the noisy corpus and the clean corpus. The method can improve the emotion recognition accuracy of speech.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a robust multi-modal sentiment recognition method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] Speech emotion recognition is a key task in intelligent speech assistance, which involves determining the emotional state of the speaker from the speech signal, such as joy, anger, sadness, and fear. The most mainstream method at present is to extract features based on a pre-trained model, and then perform deep fusion on the features of the speech and text two modalities to obtain the emotion recognition result. However, most of these methods only consider relatively ideal and static scenarios, while in real application scenarios, multi-modal signals are easily contaminated by various noises, such as speech signals that may be affected by silence or background noise, and transcribed text that may be disturbed by automatic speech recognition errors, thereby causing a serious decline in system performance. Therefore, the current emotion recognition method has the problem of low accuracy. SUMMARY

[0003] Therefore, it is necessary to provide a robust multi-modal sentiment recognition method, device, computer equipment, computer readable storage medium and computer program product capable of improving the emotion recognition accuracy of speech in view of the above technical problems.

[0004] In a first aspect, the present application provides a robust multi-modal sentiment recognition method, comprising:

[0005] obtaining to-be-recognized corpus under a target scene;

[0006] performing speech recognition processing on the to-be-recognized corpus to obtain to-be-recognized text;

[0007] inputting the to-be-recognized corpus and the to-be-recognized text into a pre-trained multi-modal sentiment recognition model to obtain a sentiment label corresponding to the to-be-recognized corpus; the pre-trained multi-modal sentiment recognition model is trained through implicit feature reconstruction and feature alignment of a pre-set corpus library; the pre-set corpus library includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space of the noisy corpus and the clean corpus.

[0008] In one of the embodiments, the pre-trained multi-modal sentiment recognition model includes a pre-trained speech encoder and a pre-trained text encoder, and the inputting the to-be-recognized corpus and the to-be-recognized text into the pre-trained multi-modal sentiment recognition model includes:

[0009] inputting the to-be-recognized corpus into the pre-trained speech encoder to obtain speech primary features;

[0010] inputting the text to be recognized into the pre-trained text encoder to obtain text primary features.

[0011] In one of the embodiments, the pre-trained multi-modal sentiment recognition model comprises a pre-trained multi-modal Transformer encoder and a sentiment classifier, and the inputting the corpus to be recognized and the text to be recognized into the pre-trained multi-modal sentiment recognition model to obtain the sentiment label corresponding to the corpus to be recognized comprises:

[0012] inputting the speech primary features and the text primary features into the pre-trained multi-modal Transformer encoder respectively to obtain speech hidden features and text hidden features;

[0013] performing average pooling and feature splicing operations on the speech hidden features and the text hidden features in sequence to obtain multi-modal high-level features;

[0014] inputting the multi-modal high-level features into the sentiment classifier to obtain the sentiment label corresponding to the corpus to be recognized.

[0015] In one of the embodiments, the pre-trained multi-modal sentiment recognition model is obtained by the following method:

[0016] performing feature extraction on the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus;

[0017] performing feature extraction on the primary features of the clean corpus and the primary features of the noisy corpus respectively by using a multi-modal Transformer encoder to obtain hidden features of the clean corpus and hidden features of the noisy corpus;

[0018] performing decoding processing and reconstruction processing on the hidden features of the noisy corpus in sequence to obtain reconstructed hidden features of the noisy corpus and a local feature reconstruction objective function;

[0019] calculating multi-modal high-level features corresponding to the hidden features of the clean corpus and the reconstructed hidden features of the noisy corpus respectively;

[0020] performing feature alignment processing on the multi-modal high-level features of the clean corpus and the multi-modal high-level features of the noisy corpus to obtain a feature alignment objective function;

[0021] training a preset sentiment recognition model according to the local feature reconstruction objective function and the feature alignment objective function to obtain a pre-trained multi-modal sentiment recognition model.

[0022] In one of the embodiments, the training of the preset emotion recognition model according to the local feature reconstruction objective function and the feature alignment objective function to obtain the pre-trained multi-modal emotion recognition model comprises:

[0023] constructing a sentiment classification task loss function between the output of the preset emotion recognition model and the corresponding sentiment label;

[0024] determining an overall training loss function according to the sum of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function;

[0025] training the preset emotion recognition model to reduce the overall training loss function to obtain the pre-trained multi-modal emotion recognition model.

[0026] In one of the embodiments, the clean corpus includes clean speech and clean text; before the feature extraction of the preset corpus to obtain the primary features of the clean corpus and the primary features of the noisy corpus, it further comprises:

[0027] sequentially performing random masking and noise adding processing on the clean speech to obtain noisy speech;

[0028] performing random masking processing on the clean text to obtain noisy text;

[0029] obtaining noisy corpus according to the noisy speech and the noisy text.

[0030] In a second aspect, the present application further provides a robust multi-modal emotion recognition device, comprising:

[0031] a corpus acquisition module for acquiring to-be-recognized corpus under a target scene;

[0032] a corpus processing module for performing speech recognition processing on the to-be-recognized corpus to obtain to-be-recognized text;

[0033] a corpus classification module for inputting the to-be-recognized corpus and the to-be-recognized text into a pre-trained multi-modal emotion recognition model to obtain the sentiment label corresponding to the to-be-recognized corpus; the pre-trained multi-modal emotion recognition model is trained by a preset corpus library for implicit feature reconstruction and feature alignment; the preset corpus library includes clean corpus and noisy corpus; the implicit feature reconstruction is used for completing the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used for reducing the distribution difference of the feature space of the noisy corpus and the clean corpus.

[0034] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0035] obtaining to-be-recognized corpus under a target scenario;

[0036] performing speech recognition processing on the to-be-recognized corpus to obtain to-be-recognized text;

[0037] inputting the to-be-recognized corpus and the to-be-recognized text into a pre-trained multi-modal sentiment recognition model to obtain a sentiment label corresponding to the to-be-recognized corpus; the pre-trained multi-modal sentiment recognition model is trained through a preset corpus library after implicit feature reconstruction and feature alignment; the preset corpus library comprises clean corpus and noisy corpus; the implicit feature reconstruction is used for complementing damaged data of the noisy corpus relative to the clean corpus; and the feature alignment is used for reducing distribution difference of feature spaces of the noisy corpus and the clean corpus.

[0038] In a fourth aspect, the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0039] obtaining to-be-recognized corpus under a target scenario;

[0040] performing speech recognition processing on the to-be-recognized corpus to obtain to-be-recognized text;

[0041] inputting the to-be-recognized corpus and the to-be-recognized text into a pre-trained multi-modal sentiment recognition model to obtain a sentiment label corresponding to the to-be-recognized corpus; the pre-trained multi-modal sentiment recognition model is trained through a preset corpus library after implicit feature reconstruction and feature alignment; the preset corpus library comprises clean corpus and noisy corpus; the implicit feature reconstruction is used for complementing damaged data of the noisy corpus relative to the clean corpus; and the feature alignment is used for reducing distribution difference of feature spaces of the noisy corpus and the clean corpus.

[0042] In a fifth aspect, the present application also provides a computer program product comprising a computer program, and the computer program is executed by a processor to implement the following steps:

[0043] obtaining to-be-recognized corpus under a target scenario;

[0044] performing speech recognition processing on the to-be-recognized corpus to obtain to-be-recognized text;

[0045] input the to-be-identified corpus and the to-be-identified text into a pre-trained multi-modal sentiment identification model to obtain a sentiment label corresponding to the to-be-identified corpus; the pre-trained multi-modal sentiment identification model is trained through implicit feature reconstruction and feature alignment of a preset corpus; the preset corpus includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete damaged data of the noisy corpus relative to the clean corpus; and the feature alignment is used to reduce distribution difference of feature spaces of the noisy corpus and the clean corpus.

[0046] The robust multi-modal sentiment identification method, device, computer device, computer readable storage medium, and computer program product described above obtain a to-be-identified corpus in a target scene, which includes a real scene affected by noise. The to-be-identified corpus is subjected to speech recognition processing to obtain to-be-identified text. The to-be-identified corpus and the to-be-identified text are further input into a pre-trained multi-modal sentiment identification model to obtain a sentiment label corresponding to the to-be-identified corpus. The pre-trained multi-modal sentiment identification model is trained through implicit feature reconstruction and feature alignment of a preset corpus. The preset corpus includes clean corpus and noisy corpus, and the pre-trained multi-modal sentiment identification model is trained together with the noisy corpus and the clean corpus, so that the model can identify the to-be-identified corpus with noise. The implicit feature reconstruction is used to complete damaged data of the noisy corpus relative to the clean corpus, so that the pre-trained multi-modal sentiment identification model has the ability to resist local feature loss. The feature alignment is used to reduce distribution difference of feature spaces of the noisy corpus and the clean corpus, which can improve the robustness of the pre-trained multi-modal sentiment identification model to noise, thereby improving the sentiment identification accuracy of the pre-trained multi-modal sentiment identification model. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other related drawings without creative labor based on these drawings.

[0048] Figure 1 A flowchart of a robust multi-modal sentiment identification method in an embodiment;

[0049] Figure 2 A flowchart of a training step of a multi-modal sentiment identification model in an embodiment;

[0050] Figure 3 A flowchart of a whole training loss determination step in an embodiment;

[0051] Figure 4 This is a flowchart illustrating a robust multimodal emotion recognition method in another embodiment;

[0052] Figure 5 This is a structural block diagram of a robust multimodal emotion recognition device in one embodiment;

[0053] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] In one embodiment, such as Figure 1 As shown, a robust multimodal emotion recognition method is provided. This embodiment illustrates the application of this method to a terminal. It is understood that this method can also be applied to a server, and further to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0056] Step S102: Obtain the corpus to be recognized in the target scene.

[0057] The target scenario can be a real-world application scenario, such as a scenario where the voice signal may be subject to various noise contaminations.

[0058] The corpus to be identified can be speech signals that require emotion recognition, which are usually noisy speech.

[0059] Optionally, the system acquires the corpus to be recognized for emotion recognition in the target scene, which can be obtained by collecting the voice of users in the target scene.

[0060] Step S104: Perform speech recognition processing on the corpus to be recognized to obtain the text to be recognized.

[0061] Among them, speech recognition processing can be ASR (Automatic Speech Recognition) technology, which is a technology that can convert speech audio into text.

[0062] Optionally, the system uses automatic speech recognition technology to process the speech data to be recognized and convert it into the corresponding text to be recognized.

[0063] Step S106: Input the corpus and text to be identified into the pre-trained multimodal sentiment recognition model to obtain the sentiment labels corresponding to the corpus to be identified.

[0064] The pre-trained multi-modal sentiment recognition model is trained through implicit feature reconstruction and feature alignment of a preset corpus. The preset corpus includes clean corpus and noisy corpus. The implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus. The feature alignment is used to reduce the distribution difference of the feature spaces of the noisy corpus and the clean corpus.

[0065] The sentiment label can be a sentiment classification label classified and labeled for each corpus file when the preset corpus is constructed.

[0066] Optionally, the system inputs the to-be-recognized corpus and the to-be-recognized text into the pre-trained multi-modal sentiment recognition model. The pre-trained multi-modal sentiment recognition model extracts features from the to-be-recognized corpus and the to-be-recognized text respectively, processes the extracted features in a series of ways, and outputs the sentiment label corresponding to the to-be-recognized corpus, so as to determine the sentiment recognition result of the to-be-recognized corpus.

[0067] In the above robust multi-modal sentiment recognition method, the to-be-recognized corpus in a target scene is obtained, and the target scene includes a real scene affected by noise. The to-be-recognized corpus is subjected to speech recognition processing to obtain to-be-recognized text. The to-be-recognized corpus and the to-be-recognized text are further input into a pre-trained multi-modal sentiment recognition model to obtain a sentiment label corresponding to the to-be-recognized corpus. The pre-trained multi-modal sentiment recognition model is trained through implicit feature reconstruction and feature alignment of a preset corpus. The preset corpus includes clean corpus and noisy corpus, and the pre-trained multi-modal sentiment recognition model is trained by the clean corpus and the noisy corpus, so that the to-be-recognized corpus with noise can also be recognized by the model. The implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus, so that the pre-trained multi-modal sentiment recognition model has the ability to resist local feature loss. The feature alignment is used to reduce the distribution difference of the feature spaces of the noisy corpus and the clean corpus, which can improve the robustness of the pre-trained multi-modal sentiment recognition model to noise, thereby improving the sentiment recognition accuracy of the pre-trained multi-modal sentiment recognition model.

[0068] In an exemplary embodiment, the pre-trained multi-modal sentiment recognition model includes a pre-trained speech encoder and a pre-trained text encoder, and step S106 inputs the to-be-recognized corpus and the to-be-recognized text into the pre-trained multi-modal sentiment recognition model, including:

[0069] The to-be-recognized corpus is input into the pre-trained speech encoder to obtain speech primary features, and the to-be-recognized text is input into the pre-trained text encoder to obtain text primary features.

[0070] The pre-trained speech encoder can be a HuBERT (Hidden-Unit BERT) model, which adopts a Transformer-based encoder and is pre-trained through self-supervised learning, mainly for processing speech data.

[0071] The pre-trained text encoder can be a RoBERTa (Reinforced Optimized BERT), which is an improved version of the BERT model, optimized through the use of a larger training data set and longer training time, mainly for text understanding and generation tasks.

[0072] Optionally, the system inputs the to-be-recognized corpus and the to-be-recognized text into the pre-trained speech encoder and the pre-trained text encoder respectively to obtain speech primary features and text primary features. Specifically, the to-be-recognized corpus and the to-be-recognized text are input into the HuBERT structure and the RoBERTa structure respectively to obtain the speech primary features and the text primary features, which are used for subsequent feature fusion.

[0073] In this embodiment, by extracting primary features from the to-be-recognized corpus and the to-be-recognized text respectively for subsequent feature fusion, the complementary information between speech and text can be utilized, which facilitates the model to understand the information of the speech more accurately, thereby improving the accuracy of emotion recognition.

[0074] In an exemplary embodiment, the pre-trained multi-modal emotion recognition model includes a pre-trained multi-modal Transformer encoder and an emotion classifier, and step S106 inputs the to-be-recognized corpus and the to-be-recognized text into the pre-trained multi-modal emotion recognition model to obtain the emotion label corresponding to the to-be-recognized corpus, including:

[0075] The speech primary features and the text primary features are input into the pre-trained multi-modal Transformer encoder to obtain speech hidden features and text hidden features; the speech hidden features and the text hidden features are sequentially subjected to average pooling and feature splicing operations to obtain multi-modal high-level features; and the multi-modal high-level features are input into the pre-trained emotion classifier to obtain the emotion label corresponding to the to-be-recognized corpus.

[0076] The hidden features are output by the pre-trained multi-modal Transformer encoder after jointly modeling the intra-modal and inter-modal interaction relationships.

[0077] Among them, average pooling is a commonly used downsampling technique in convolutional neural networks (CNN). It reduces the size of the feature map by calculating the average value of each local region in the input feature map. This method can effectively preserve the spatial information of the features while reducing the computational complexity and the risk of overfitting. Average pooling is usually used after the convolutional layer to reduce the dimension of the feature map.

[0078] Among them, feature concatenation is an operation that connects multiple feature vectors in a certain dimension. In deep learning, feature concatenation is often used to combine features from different layers or different models to facilitate subsequent processing or classification. By concatenating, the advantages of different features can be fully utilized to improve the performance of the model. Feature concatenation is usually performed before the fully connected layer to form a more rich feature representation.

[0079] Optionally, the system inputs the speech primary features and the text primary features into the trained multi-modal Transformer encoder respectively to obtain speech hidden features and text hidden features. For example, the speech hidden features and the text hidden features are and The specific steps for calculating the speech hidden features and the text hidden features are as follows: first, project the speech hidden features and the text hidden features respectively into Q, K, and V:

[0080] , ,

[0081] , ,

[0082] Among them, , , and , , are learnable parameters. Then model the intra-modal relationship of the Q, K, and V matrices obtained after projection:

[0083] ,

[0084] Among them, is the square root of K dimension, is the intra-modal relationship of the speech primary features after projection, is the intra-modal relationship of the text primary features after projection, is a normalization process.

[0085] Model the inter-modal relationship of the Q, K, and V matrices obtained after projection:

[0086] ,

[0087] wherein, is the inter-modal relationship between the projected speech primary features and the projected text primary features, is the inter-modal relationship between the projected text primary features and the projected speech primary features.

[0088] The speech latent features and the text latent features are calculated respectively, and the corresponding formulas are as follows:

[0089]

[0090]

[0091] wherein, represents the speech latent features, represents the text latent features, Concat represents the concatenation operation along the dimension direction, Linear represents the linear layer, and are learnable parameters, and the feature dimensions of and are projected to be consistent with and .

[0092] The system sequentially performs average pooling and feature concatenation operations on the speech latent features and the text latent features to obtain the multi-modal high-level features. The corresponding multi-modal high-level feature calculation formula is as follows:

[0093]

[0094] wherein, MeanPool represents the average pooling operation along the time direction. represents the multi-modal high-level features.

[0095] After the system calculates the multi-modal high-level features, the multi-modal high-level features are input into the sentiment classifier to obtain the sentiment label corresponding to the to-be-recognized corpus.

[0096] In this embodiment, by combining the speech latent features and the text latent features, the sentiment of the to-be-recognized corpus can be more comprehensively understood. Multi-modal learning can generally improve the accuracy of sentiment recognition, because it utilizes different types of data sources and reduces the bias that may be caused by a single modality. Speech or text may have noise or missing, and through the combination of multi-modal features, the robustness of the system to these problems can also be enhanced.

[0097] In an exemplary embodiment, as shown in Figure 2 , the pre-trained multi-modal sentiment recognition model described in the above embodiments is obtained by the following method:

[0098] In step S202, feature extraction is performed on the preset corpus to obtain primary features of the clean corpus and primary features of the noisy corpus.

[0099] The preset corpus can be constructed in advance for a plurality of clean speeches that have been classified by emotional labeling. The clean corpus includes clean speeches and clean texts.

[0100] Optionally, the system performs data enhancement processing on the clean speeches in the preset corpus to obtain noisy corpus, which also includes noisy speeches and noisy texts. Further feature extraction is performed on the clean speeches, the noisy speeches, the clean texts and the noisy texts to obtain primary features corresponding to the clean speeches, the noisy speeches, the clean texts and the noisy texts respectively.

[0101] In step S204, a multi-modal Transformer encoder is used to perform feature extraction on the primary features of the clean corpus and the primary features of the noisy corpus respectively to obtain hidden features of the clean corpus and hidden features of the noisy corpus.

[0102] Optionally, the system uses a multi-modal Transformer encoder to perform feature extraction on the primary features of the clean corpus and the primary features of the noisy corpus respectively to obtain hidden features of the clean corpus and hidden features of the noisy corpus. The extraction method is the same as the calculation method of the hidden features of the to-be-recognized corpus in the above embodiment.

[0103] In step S206, decoding processing and reconstruction processing are sequentially performed on the hidden features of the noisy corpus to obtain reconstructed hidden features of the noisy corpus and a local feature reconstruction objective function.

[0104] Optionally, the system performs decoding processing on the hidden features of the noisy corpus, and the corresponding formula is as follows:

[0105] The noisy hidden features are respectively mapped to the same dimension as the input features by a linear layer decoder:

[0106] ,

[0107] wherein, denotes the reconstructed hidden features of the noisy speeches, denotes the reconstructed hidden features of the noisy texts, and are learnable parameters.

[0108] The system further performs reconstruction processing on the reconstructed hidden features of the noisy corpus to obtain a local feature reconstruction objective function, and the corresponding formula is as follows:

[0109]

[0110] wherein, L1 represents an L1 norm loss function, denotes a loss of local reconstruction, denotes a primary feature of noisy speech, denotes a primary feature of noisy text, denotes a mask matrix of speech corpus, denotes a mask matrix of noisy text.

[0111] Step S208, the multi-modal high-level features corresponding to the hidden features of the clean corpus and the reconstructed hidden features of the noisy corpus are respectively calculated.

[0112] Optionally, the system calculates the multi-modal high-level features obtained by performing average pooling and feature splicing on the hidden features of the clean speech and the hidden features of the clean text, and calculates the multi-modal high-level features obtained by performing average pooling and feature splicing on the hidden features of the noisy speech and the hidden features of the noisy text, and the corresponding formula is the same as the formula of the above embodiment. Wherein, the multi-modal Transformer encoders used by the noisy corpus and the clean corpus respectively adopt the same weight.

[0113] Step S210, performing feature alignment processing on the multi-modal high-level features of the clean corpus and the multi-modal high-level features of the noisy corpus to obtain a feature alignment target function.

[0114] Optionally, the system performs feature alignment processing on the multi-modal high-level features of the clean corpus and the multi-modal high-level features of the noisy corpus to obtain a feature alignment target function. The corresponding formula is as follows:

[0115]

[0116] wherein, denotes a multi-modal high-level feature of noisy corpus, denotes a multi-modal high-level feature of clean corpus, JSD represents Jensen-Shannon divergence, which can measure the difference between two probability distributions, denotes a feature alignment loss.

[0117] Step S212, according to the local feature reconstruction target function and the feature alignment target function, the pre-set emotion recognition model is trained to obtain a pre-trained multi-modal emotion recognition model.

[0118] Optionally, the system trains the pre-set emotion recognition model according to the local feature reconstruction target function and the feature alignment target function, so as to minimize the loss value, and obtains a pre-trained multi-modal emotion recognition model.

[0119] In this embodiment, by means of a preset corpus, a series of feature extraction processing, and feature reconstruction of the noisy corpus, the damaged data of the noisy corpus relative to the clean corpus can be completed, so that the pre-trained multi-modal sentiment recognition model has the ability to resist local feature loss. Then, the implicit features of the noisy corpus and the clean corpus are aligned for feature, which is used to reduce the distribution difference of the feature space of the noisy corpus and the clean corpus, and can improve the robustness of the multi-modal sentiment recognition model to noise. Finally, according to the local feature reconstruction objective function and the feature alignment objective function, the pre-set sentiment recognition model is trained to obtain a pre-trained multi-modal sentiment recognition model, so that a multi-modal sentiment recognition model with stronger robustness is obtained, and the sentiment recognition accuracy of the system is further improved.

[0120] In one exemplary embodiment, as shown in Figure 3 Step S212 trains the pre-set sentiment recognition model according to the local feature reconstruction objective function and the feature alignment objective function to obtain a pre-trained multi-modal sentiment recognition model, including steps S302 to S306. Among them:

[0121] Step S302, construct the sentiment classification task loss function between the output of the pre-set sentiment recognition model and the corresponding sentiment label.

[0122] Among them, the output of the pre-set sentiment recognition model can be the classification result output by the classifier of the pre-set sentiment recognition model after inputting the multi-modal high-level features of the noisy corpus after feature alignment.

[0123] Optionally, the system constructs the sentiment classification task loss function between the output of the pre-set sentiment recognition model and the actual sentiment label corresponding to the noisy corpus, and the corresponding formula is:

[0124]

[0125] Among them, is the sentiment classification task loss, is the output of the pre-set sentiment recognition model, is the actual sentiment label corresponding to the noisy corpus.

[0126] Step S304 determines the overall training loss function according to the sum of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function.

[0127] Optionally, the system determines the overall training loss function according to the sum of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function, and the corresponding formula is:

[0128]

[0129] wherein, is the overall training loss, is the sentiment classification task loss function, is the local feature reconstruction objective function, is the feature alignment objective function, and is the hyperparameter to control the adversarial training loss weight, and the default setting is .

[0130] In step S306, the pre-set emotion recognition model is trained to reduce the overall training loss function, and a pre-trained multi-modal emotion recognition model is obtained.

[0131] Optionally, the system trains the pre-set emotion recognition model to reduce the overall training loss function until the training end condition is met, and obtains the pre-trained multi-modal emotion recognition model, wherein the training end condition can be that the overall training loss function is lower than a pre-set threshold, or that the training has reached a pre-set number of times.

[0132] In this embodiment, the overall training loss function is obtained by comprehensively considering the objective functions under different feature processing conditions, which can more comprehensively optimize the model performance and improve the accuracy of emotion recognition in the training process.

[0133] In an exemplary embodiment, the clean corpus includes clean speech and clean text; before step S202, feature extraction is performed on the pre-set corpus to obtain the primary features of the clean corpus and the primary features of the noisy corpus, the method further comprises:

[0134] The clean speech is sequentially subjected to random masking and noise addition processing to obtain noisy speech; the clean text is subjected to random masking processing to obtain noisy text; and the noisy corpus is obtained according to the noisy speech and the noisy text.

[0135] Random masking can be a data augmentation method, which is usually used to process sequence data (such as text, audio, etc.) or image data. The basic idea is to randomly select a part of the elements in the input data for masking (i.e., replacing them with a specific value such as zero or other placeholders), to simulate the situation of missing data.

[0136] Noise addition processing can be a data augmentation technique that adds random noise to input data to improve the robustness of the model. The noise can be Gaussian noise, salt and pepper noise, etc., depending on the application scenario.

[0137] Optionally, the system sequentially performs random masking and noise addition processing on clean speech to obtain noisy speech. For example, the system randomly selects a noise source from a given non-stationary noise database and randomly selects a signal-to-noise ratio within the range of 0-20dB for additive noise aliasing to simulate a real environment. After dividing the original speech waveform into frames with a frame length of 25ms and a frame shift of 10ms, 20% of the speech frames are randomly zeroed to simulate data loss. The temporal positions of the zeroing are recorded using a masking matrix to obtain noisy speech. Furthermore, the system performs random masking on clean text to obtain noisy text. For example, the system similarly masks 20% of the characters in the text sequence of clean text to simulate unrecognizable [UNK] characters and records this using a masking matrix. The system then further obtains noisy corpus based on the noisy speech and noisy text.

[0138] In this embodiment, by randomly selecting noise from a non-stationary noise database and aliasing it at different signal-to-noise ratios, the model can learn speech features under various noise environments during subsequent training. This training method enables the model to better recognize and understand speech content when faced with background noise in real-world environments. Furthermore, zeroing out 20% of the speech frames simulates data loss, allowing the model to maintain high recognition accuracy even when processing incomplete information. This method enhances the model's adaptability to missing data.

[0139] In one exemplary embodiment, such as Figure 4 As shown, another robust multimodal emotion recognition method is provided, the specific steps of which include:

[0140] Step 1: Dynamic enhancement of noisy multimodal data.

[0141] For example, during the training phase, data augmentation is performed on a given clean multimodal data in the following two steps to obtain noisy data: (1) Background noise simulation: given a non-stationary noise database (2) Data missing simulation: Randomly select one noise source and randomly select the signal-to-noise ratio within the range of 0~20dB to perform additive noise aliasing to simulate the real environment. The original speech waveform was divided into frames with a frame length of 25ms and a frame shift of 10ms. Then, 20% of the speech frames were randomly set to zero to simulate data loss. This was then processed using a masking matrix. Record the timing position of setting to zero; 20% of the characters in the text sequence are also masked to simulate unrecognizable [UNK] characters, and this is done using a masking matrix. Record.

[0142] Step 2: Primary feature extraction and multimodal high-level feature modeling based on pre-trained models

[0143] Step 2.1, primary feature extraction. Exemplarily, the speech and text pre-training models in the present application use HuBERT and RoBERTa structures respectively, and fix the weights of the pre-training encoders as the extraction module of the multi-modal primary features, obtaining the primary features of clean speech and text and , the primary features of noisy speech and text and , for subsequent feature fusion.

[0144] Step 2.1, multi-modal high-level feature modeling. Exemplarily, the multi-modal Transformer encoder in the present application jointly models the intra-modal and inter-modal interaction relationship, and outputs the noisy speech and text hidden representation. Taking the noisy speech and text input and as an example, the specific calculation steps are as follows:

[0145] Q, K, V projection:

[0146] , ,

[0147] , ,

[0148] wherein, , , and , , are learnable parameters. (2) Intra-modal relationship modeling:

[0149] ,

[0150] Inter-modal relationship modeling:

[0151] ,

[0152] Hidden feature calculation:

[0153]

[0154]

[0155] wherein, Concat represents the concatenation operation along the dimension direction, Linear represents the linear layer, and are learnable parameters, and the feature dimension of the clean speech and text are projected to the same and

[0156] Multimodal high-level feature computation:

[0157]

[0158] where MeanPool denotes the average pooling operation along the time direction. Meanwhile, the clean speech and text and The multimodal Transformer encoder with the same structure and parameters is applied to the input to obtain the corresponding hidden features and and the multimodal high-level features .

[0159] Step 3, two-stage feature reconstruction. Illustratively, based on the clean and noisy features extracted in 5.2, the present application performs feature reconstruction from the local and global perspectives to improve the robustness of the model in the inference stage.

[0160] Step 3.1, local feature decoding and reconstruction. Illustratively, the hidden features of the noisy speech and text and encode sufficient multimodal context information, the present application proposes to guide the hidden features to complete the damaged information of the noisy data relative to the clean data to encourage the model to have the ability to resist the loss of local features. The specific calculation steps are as follows:

[0161] The noisy hidden features are respectively mapped to the same dimension as the input features through a linear layer decoder:

[0162] ,

[0163] Local feature reconstruction objective function:

[0164]

[0165] where L1 represents the L1 norm loss function.

[0166] Step 3.2, global feature alignment. Illustratively, the high-level multimodal features of the noisy and clean data highly abstract the emotional semantics of the input features, and by reducing the distribution difference between the two in the feature space, the robustness of the model to noise can be directly enhanced. The objective function of global feature alignment is as follows:

[0167]

[0168] where JSD represents the Jensen-Shannon divergence, which can measure the difference between two probability distributions.​

[0169] Step 4, model training objective.

[0170] Exemplarily, the output of the sentiment classifier is denoted as , and the corresponding label is denoted as and The loss function of the sentiment classification task is defined as follows:

[0171]

[0172] According to the above analysis, the overall training loss function of the robust multi-modal sentiment recognition method can be expressed as:

[0173]

[0174] wherein and are hyperparameters for controlling the weight of the adversarial training loss, and the default setting is In the training phase, the pre-trained encoder parameters of the speech and text are fixed, and the remaining trainable modules are updated by the stochastic gradient descent algorithm. In the application inference phase, only the forward calculation of the steps involved in the noisy speech and noisy text as shown in Figure 4 is required.

[0175] In this embodiment, a multi-modal noisy data dynamic enhancement method is proposed, which dynamically generates noisy data pairs for a given multi-modal sentiment corpus. On the one hand, it expands the data volume and alleviates the problem of model overfitting; on the other hand, it provides paired clean and noisy data for subsequent feature reconstruction. A two-stage feature reconstruction method is proposed, which guides each modality to recover the missing features using context information through local feature reconstruction, and guides the model to reduce the sentiment semantic difference between clean and noisy high-level multi-modal representations through global feature alignment, combining the complementary advantages of the two to improve the robustness of the model against noisy data.

[0176] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0177] Based on the same inventive concept, the embodiments of the present application also provide a robust multi-modal emotion recognition method device for implementing the above-mentioned robust multi-modal emotion recognition method. The implementation scheme for solving problems provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more robust multi-modal emotion recognition method device embodiments provided below can refer to the limitations of the robust multi-modal emotion recognition method in the foregoing, which will not be repeated here.

[0178] In one exemplary embodiment, as shown in Figure 5 A robust multi-modal emotion recognition method device 500 is provided, comprising: a corpus acquisition module 502, a corpus processing module 504, and a corpus classification module 506, wherein:

[0179] The corpus acquisition module 502 is configured to acquire the to-be-recognized corpus in the target scene.

[0180] The corpus processing module 504 is configured to perform speech recognition processing on the to-be-recognized corpus to obtain to-be-recognized text.

[0181] The corpus classification module 506 is configured to input the to-be-recognized corpus and the to-be-recognized text into a pre-trained multi-modal emotion recognition model to obtain an emotion label corresponding to the to-be-recognized corpus; the pre-trained multi-modal emotion recognition model is trained by pre-set corpus library for implicit feature reconstruction and feature alignment; the pre-set corpus library includes clean corpus and noisy corpus; the implicit feature reconstruction is used to complete the damaged data of the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space of the noisy corpus and the clean corpus.

[0182] Further, in one embodiment, the corpus classification module 506 is further configured to input the to-be-recognized corpus into a pre-trained speech encoder to obtain speech primary features; and input the to-be-recognized text into a pre-trained text encoder to obtain text primary features.

[0183] Further, in one embodiment, the corpus classification module 506 is further configured to input the speech primary features and the text primary features into a pre-trained multi-modal Transformer encoder respectively to obtain speech implicit features and text implicit features; and perform average pooling and feature splicing operations on the speech implicit features and the text implicit features in sequence to obtain multi-modal advanced features; and input the multi-modal advanced features into a pre-trained emotion classifier to obtain an emotion label corresponding to the to-be-recognized corpus.

[0184] Further, in an embodiment, the robust multi-modal sentiment recognition method device 500 further comprises a model training module, configured to perform feature extraction on a preset corpus to obtain primary features of clean corpus and primary features of noisy corpus; perform feature extraction on the primary features of clean corpus and the primary features of noisy corpus respectively by using a multi-modal Transformer encoder to obtain implicit features of clean corpus and implicit features of noisy corpus; perform decoding processing and reconstruction processing on the implicit features of noisy corpus in sequence to obtain reconstructed implicit features of noisy corpus and a local feature reconstruction objective function; calculate multi-modal high-level features corresponding to the implicit features of clean corpus and the reconstructed implicit features of noisy corpus respectively; perform feature alignment processing on the multi-modal high-level features of clean corpus and the multi-modal high-level features of noisy corpus to obtain a feature alignment objective function; and train the preset sentiment recognition model according to the local feature reconstruction objective function and the feature alignment objective function to obtain a pre-trained multi-modal sentiment recognition model.

[0185] Further, in an embodiment, the model training module is further configured to construct a sentiment classification task loss function between an output of the preset sentiment recognition model and a corresponding sentiment label; determine an overall training loss function according to a sum value of the local feature reconstruction objective function, the feature alignment objective function and the sentiment classification task loss function; and train the preset sentiment recognition model to obtain the pre-trained multi-modal sentiment recognition model by taking reducing the overall training loss function as a target.

[0186] Further, in an embodiment, the model training module is further configured to perform random masking and noise adding processing on the clean speech in sequence to obtain noisy speech; perform random masking processing on the clean text to obtain noisy text; and obtain the noisy corpus according to the noisy speech and the noisy text.

[0187] The above-mentioned various modules in the robust multi-modal sentiment recognition device can be all or partially realized by software, hardware and combinations thereof. The above-mentioned various modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to the above-mentioned various modules.

[0188] In an exemplary embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in FIG. 1. Figure 6As shown in the figure. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as to-be-recognized corpus, to-be-recognized text and emotional label. The input / output interface of the computer device is used to exchange information between the processor and external devices. The communication interface of the computer device is used to communicate with external terminals through network connection. The computer program is executed by the processor to implement a robust multi-modal sentiment recognition method.

[0189] Those skilled in the art can understand that, Figure 6 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0190] In one embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in each of the above method embodiments.

[0191] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0192] In one embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps in each of the above method embodiments.

[0193] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0194] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0195] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0196] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A robust multimodal emotion recognition method, characterized in that, The method includes: Obtain the corpus of language to be recognized in the target scene; The corpus to be recognized is subjected to speech recognition processing to obtain the text to be recognized; The corpus and text to be identified are input into a pre-trained multimodal emotion recognition model to obtain latent speech features and latent text features. Average pooling is then performed on the latent speech features and latent text features to obtain average pooled latent speech features and average pooled latent text features. Feature concatenation is then performed on the average pooled latent speech features and average pooled latent text features to obtain multimodal high-level features. These multimodal high-level features are input into the emotion classifier of the multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be identified. The pre-trained multimodal emotion recognition model is trained by reconstructing latent features and aligning features using a preset corpus. The preset corpus includes clean corpus and noisy corpus. The latent feature reconstruction is used to complete the data in the noisy corpus that is damaged relative to the clean corpus. The feature alignment is used to reduce the distribution difference in the feature spaces of the noisy corpus and the clean corpus. The pre-trained multimodal emotion recognition model is obtained through the following method: Feature extraction is performed on the preset corpus to obtain the primary features of the clean corpus and the primary features of the noisy corpus; A multimodal Transformer encoder is used to extract features from the primary features of the clean corpus and the primary features of the noisy corpus, respectively, to obtain the latent features of the clean corpus and the latent features of the noisy corpus. The noisy corpus's latent features are sequentially decoded and reconstructed to obtain the reconstructed latent features and local feature reconstruction objective function of the noisy corpus; Calculate the multimodal high-level features corresponding to the latent features of the clean corpus and the reconstructed latent features of the noisy corpus respectively; Feature alignment processing is performed on the multimodal high-level features of the clean corpus and the multimodal high-level features of the noisy corpus to obtain the feature alignment objective function; Based on the local feature reconstruction objective function and the feature alignment objective function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model; The objective function for local feature reconstruction includes the following formulas: Where L1 represents the L1 norm loss function, This represents the loss from local reconstruction. These represent the primary features of noisy speech. M represents the primary features of noisy text. a M represents the masking matrix of the speech corpus. t D represents the masking matrix of the noisy text. a D represents the latent features of reconstructed noisy speech; t Represents the implicit features of reconstructed noisy text; The feature alignment objective function includes the following formula: Where h represents the multimodal high-level features of the noisy corpus. JSD represents the multimodal high-level features of clean corpora, and it stands for Jensen-Shannon divergence, which measures the degree of difference between two probability distributions. This represents the feature alignment loss.

2. The method according to claim 1, characterized in that, The pre-trained multimodal emotion recognition model includes a pre-trained speech encoder and a pre-trained text encoder. The step of inputting the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model includes: The corpus to be recognized is input into the pre-trained speech encoder to obtain primary speech features; The text to be recognized is input into the pre-trained text encoder to obtain the primary features of the text.

3. The method according to claim 2, characterized in that, The pre-trained multimodal emotion recognition model further includes a pre-trained multimodal Transformer encoder. The process of inputting the corpus to be recognized and the text to be recognized into the pre-trained multimodal emotion recognition model to obtain implicit speech features and implicit text features includes: The primary speech features and the primary text features are respectively input into the pre-trained multimodal Transformer encoder to obtain the speech latent features and the text latent features.

4. The method according to claim 1, characterized in that, The step of training a pre-trained multimodal emotion recognition model based on the local feature reconstruction objective function and the feature alignment objective function to obtain a pre-trained multimodal emotion recognition model includes: Construct a loss function for the emotion classification task between the output of the preset emotion recognition model and the corresponding emotion tag; The overall training loss function is determined based on the sum of the local feature reconstruction objective function, the feature alignment objective function, and the sentiment classification task loss function; With the goal of reducing the overall training loss function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model.

5. The method according to claim 1, characterized in that, The clean corpus includes clean speech and clean text; before performing feature extraction on the preset corpus to obtain the primary features of the clean corpus and the primary features of the noisy corpus, the method further includes: The clean speech is then subjected to random masking and noise addition processes in sequence to obtain noisy speech; The clean text is randomly masked to obtain noisy text; Based on the noisy speech and the noisy text, a noisy corpus is obtained.

6. A robust multimodal emotion recognition device, characterized in that, The device includes: The corpus acquisition module is used to acquire the corpus to be recognized in the target scene; The corpus processing module is used to perform speech recognition processing on the corpus to be recognized to obtain the text to be recognized; The corpus classification module is used to input the corpus to be identified and the text to be identified into a pre-trained multimodal emotion recognition model to obtain speech latent features and text latent features; to perform average pooling on the speech latent features and the text latent features respectively to obtain average pooled speech latent features and average pooled text latent features; to perform feature concatenation operation on the average pooled speech latent features and the average pooled text latent features to obtain multimodal high-level features; to input the multimodal high-level features into the emotion classifier of the multimodal emotion recognition model to obtain the emotion label corresponding to the corpus to be identified; the pre-trained multimodal emotion recognition model is trained by performing latent feature reconstruction and feature alignment on a preset corpus; the preset corpus includes clean corpus and noisy corpus; the latent feature reconstruction is used to complete the data damaged in the noisy corpus relative to the clean corpus; the feature alignment is used to reduce the distribution difference of the feature space of the noisy corpus and the clean corpus; The pre-trained multimodal emotion recognition model is obtained through the following method: Feature extraction is performed on the preset corpus to obtain the primary features of the clean corpus and the primary features of the noisy corpus; A multimodal Transformer encoder is used to extract features from the primary features of the clean corpus and the primary features of the noisy corpus, respectively, to obtain the latent features of the clean corpus and the latent features of the noisy corpus. The noisy corpus's latent features are sequentially decoded and reconstructed to obtain the reconstructed latent features and local feature reconstruction objective function of the noisy corpus; Calculate the multimodal high-level features corresponding to the latent features of the clean corpus and the reconstructed latent features of the noisy corpus respectively; Feature alignment processing is performed on the multimodal high-level features of the clean corpus and the multimodal high-level features of the noisy corpus to obtain the feature alignment objective function; Based on the local feature reconstruction objective function and the feature alignment objective function, the preset emotion recognition model is trained to obtain a pre-trained multimodal emotion recognition model; The objective function for local feature reconstruction includes the following formulas: Where L1 represents the L1 norm loss function, This represents the loss from local reconstruction. These represent the primary features of noisy speech. M represents the primary features of noisy text. a M represents the masking matrix of the speech corpus. t D represents the masking matrix of the noisy text. a D represents the latent features of reconstructed noisy speech; t Represents the implicit features of reconstructed noisy text; The feature alignment objective function includes the following formula: Where h represents the multimodal high-level features of the noisy corpus. JSD represents the multimodal high-level features of clean corpora, and it stands for Jensen-Shannon divergence, which measures the degree of difference between two probability distributions. This represents the feature alignment loss.

7. The apparatus according to claim 6, characterized in that, The corpus classification module is further configured to input the corpus to be identified into the pre-trained speech encoder to obtain primary speech features; and to input the text to be identified into the pre-trained text encoder to obtain primary text features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images