A digital interaction method and system based on emotion recognition

Abstract

The application discloses a kind of digital interaction method and system based on emotion recognition, belong to digital interaction technical field, by with the historical multimodal data after the pre-processing and its corresponding emotion label as training data, the deep learning model is trained, obtains emotion recognition model, and in with user interactive process, using the emotion recognition model is identified to the real-time multimodal data after the pre-processing, determine the real-time emotion label corresponding to the user, finally execute the preset digital interactive scene corresponding to the real-time emotion label, can effectively realize the interaction in digital cemetery and digital memorial, automatically recommend interactive memorial scene, to bring more humanized remote reflection experience.

Description

Technical Field

[0001] The present invention belongs to the field of digital interaction technology, and specifically relates to a digital interaction method and system based on emotion recognition. Background Art

[0002] Digital cemeteries are virtual graveyards created using modern technology. They display information about the deceased, commemorative items, and other items online, enabling remote memorialization. Online mourning platforms offer a variety of virtual memorial services, such as laying flowers, leaving messages, and playing commemorative videos, allowing users to express their grief anytime, anywhere. The combination of these two approaches transcends time and space, allowing families and friends to remember the deceased while conserving land resources and embracing environmental protection. The integration of emotion recognition technology further humanizes interactions, meeting users' deeper emotional needs and ushering in new trends in modern funeral services. With the advancement of technology, digital cemeteries and digital memorial services have gradually become a new way for people to commemorate the deceased and express their grief. However, existing digital cemeteries and memorial services often lack emotional interaction and fail to meet people's deeper emotional needs. Summary of the Invention

[0003] The present invention provides a digital interactive method and system based on emotion recognition, which is used to solve the problem that existing digital cemeteries and memorial methods often lack interaction.

[0004] In one aspect, the present invention provides a digital interaction method based on emotion recognition, comprising:

[0005] Collecting historical multimodal data and preprocessing the historical multimodal data to obtain preprocessed historical multimodal data; wherein the historical multimodal data includes video data and audio data;

[0006] Presenting the historical multimodal data to a staff member so that the staff member inputs an emotion label corresponding to the historical multimodal data;

[0007] Using the pre-processed historical multimodal data and its corresponding emotion labels as training data, a deep learning model is trained to obtain an emotion recognition model;

[0008] During the interaction with the user, real-time multimodal data of the user is collected in real time, and the real-time multimodal data is preprocessed to obtain the preprocessed real-time multimodal data;

[0009] The emotion recognition model is used to identify the real-time multimodal data after preprocessing, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition.

[0010] Furthermore, historical multimodal data is collected and preprocessed to obtain preprocessed historical multimodal data, including:

[0011] Based on a preset time period, the user's video data and audio data within any time period are collected to obtain historical multimodal data corresponding to a single time period;

[0012] Historical multimodal data within a plurality of time periods is collected, and the historical multimodal data is preprocessed to obtain preprocessed historical multimodal data.

[0013] Furthermore, after preprocessing the historical multimodal data, the preprocessed historical multimodal data is obtained, including:

[0014] Sampling the video data in the historical multimodal data based on a preset data sampling frequency to obtain a plurality of video frames corresponding to the historical multimodal data;

[0015] Extracting speech features from audio data in the historical multimodal data, and constructing a speech feature data matrix based on the speech features;

[0016] The multiple video frames corresponding to the historical multimodal data and the voice feature data matrix are taken together as the historical multimodal data after preprocessing.

[0017] Furthermore, extracting speech features from the audio data in the historical multimodal data and constructing a speech feature data matrix based on the speech features includes:

[0018] Extracting speech features from the audio data in the historical multimodal data, and forming a feature vector from the speech features;

[0019] Obtain the importance of each feature in the feature vector based on the trained LightGBM model, and sort the features in descending order by importance;

[0020] The average importance value corresponding to the speech features is obtained, and speech features with importance lower than the average importance value are filtered out. The optimal feature subset is selected using a sequential forward algorithm to obtain target speech features, and the target speech features are combined into a feature vector.

[0021] Furthermore, the pre-processed historical multimodal data and its corresponding emotion labels are used as training data to train a deep learning model to obtain an emotion recognition model, including:

[0022] Constructing a first deep learning model and a second deep learning model;

[0023] Initializing hyperparameters of a deep learning model to be trained to obtain multiple hyperparameter individuals; wherein the deep learning model to be trained is the first deep learning model or the second deep learning model;

[0024] Obtaining the fitness of each hyperparameter individual according to the preprocessed historical multimodal data and its corresponding emotion label, and determining the current optimal hyperparameter individual in the current training process based on the fitness of each hyperparameter individual;

[0025] Based on the current optimal hyperparameter individual, a time-adaptive local search strategy, a space-adaptive local search strategy, and a global jump search strategy are sequentially adopted to train multiple hyperparameter individuals;

[0026] After the total number of training times reaches a preset maximum number of training times, the target optimal hyperparameter individual is determined, and the target optimal hyperparameter individual is used as the final hyperparameter of the deep learning model to be trained, thereby obtaining the deep learning model to be trained after training;

[0027] After training the first deep learning model and the second deep learning model, a trained first deep learning model and a trained second deep learning model are obtained;

[0028] Removing the classification output layer of the first deep learning model after training to obtain a first feature extraction model; removing the classification output layer of the second deep learning model after training to obtain a second feature extraction model;

[0029] Constructing a third deep learning model, extracting first feature data using the first feature extraction model and extracting second feature data using the second feature extraction model based on the preprocessed historical multimodal data;

[0030] Training the third deep learning model as a deep learning model to be trained based on the first feature data, the second feature data, and the corresponding emotion labels to obtain a feature recognition model;

[0031] An emotion recognition model is obtained based on the first feature extraction model, the second feature extraction model and the feature recognition model.

[0032] Furthermore, the time-adaptive local search strategy includes:

[0033] Based on the current number of training times, the adaptive control factor of the generation time is:

[0034]

[0035] in, represents the time adaptive control factor, represents pi, Indicates the current number of training times. Express expectations, represents the variance, represents the variance control parameter, exp represents the natural constant with the natural constant e as the base;

[0036] Based on the current optimal hyperparameter individual, a time-adaptive control factor is used to perform a time-adaptive local search on the hyperparameter individual, and the first target hyperparameter individual is obtained as:

[0037]

[0038] in, represents the jth hyperparameter individual in the tth training process, and j=1,2,…,K, K represents the total number of hyperparameter individuals, represents the jth first target hyperparameter individual, represents the first learning factor, Represents the current optimal hyperparameter individual.

[0039] Furthermore, the spatially adaptive local search strategy includes:

[0040] For the i The first target hyperparameter individual, determine the i -1 first target hyperparameter individual is the first adjacent individual, determine the i +1 first target hyperparameter individual is the second adjacent individual; among them, for the first first target hyperparameter individual, its first adjacent individual is set to other random first target hyperparameter individuals; for the Kth first target hyperparameter individual, its second adjacent individual is set to other random first target hyperparameter individuals;

[0041] According to the first adjacent individual and the second adjacent individual, obtain the i The spatial adaptive control factor corresponding to the first target hyperparameter individual is:

[0042]

[0043]

[0044]

[0045] in, Indicates the i The spatial adaptive control factors corresponding to the first target hyperparameter individuals, represents the first scale factor, represents the second scale factor, Indicates the current number of training times, T indicates the preset maximum number of training times, Indicates the i The Euclidean distance between the first target hyperparameter individual and its corresponding first adjacent individual, Indicates the i The Euclidean distance between the first target hyperparameter individual and its corresponding second adjacent individual;

[0046] According to the said i The spatial adaptive control factor corresponding to the first target hyperparameter individual, i The first target hyperparameter individuals are spatially adaptively searched locally, and the second target parameter individuals are obtained as follows:

[0047]

[0048] in, Indicates the t During the training process i First target hyperparameter individuals, Indicates the i The first neighboring individuals of the first target hyperparameter individual, Indicates the i The second adjacent individuals of the first target hyperparameter individual, represents the first random number between (0,1), Indicates the i The second target parameter individual.

[0049] Furthermore, the global jump search strategy includes:

[0050] According to the current number of training times, the adaptive global jump probability is obtained as:

[0051]

[0052] in, represents the adaptive global jump probability, sin represents the sine function, Indicates the preset maximum number of training times, t indicates the current number of training times, represents pi;

[0053] Obtain the fitness corresponding to all second target parameter individuals, and arrange the second target parameter individuals in descending order of fitness. According to the arranged second target parameter individuals, obtain the information fusion position corresponding to all second target parameter individuals as follows:

[0054]

[0055]

[0056] in, Indicates the t During the training n The second target parameter individuals after permutation, Represents the second target parameter individual The weighting coefficient of Indicates the information fusion position corresponding to all second target parameter individuals, Indicates the total number of individuals corresponding to the second target parameter;

[0057] For any second target parameter individual, determining a jump action corresponding to the second target parameter individual according to the adaptive global jump probability; wherein the jump action includes whether jumping is required or not;

[0058] When the jump action corresponding to the second target parameter individual is that a jump is required, a global jump search is performed on the second target parameter individual according to the information fusion position, and the global jump search position is obtained as:

[0059]

[0060] in, Indicates the t During the training m The second target parameter individuals, Represents the second target parameter individual The corresponding global jump search position, represents the second learning factor, represents the third learning factor, represents the second random number between (0,1), represents the third random number between (0,1), Indicates that except for the second target parameter individual Random second target parameter individuals other than ;

[0061] Determine whether the fitness of the global jump search position is greater than the fitness of the second target parameter individual. If so, use the global jump search position as the third target parameter individual; otherwise, use the original second target parameter individual as the third target parameter individual; wherein the third target parameter individual is the second target parameter individual after the global jump search.

[0062] Furthermore, the emotion recognition model is used to recognize the pre-processed real-time multimodal data, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition, including:

[0063] Using the emotion recognition model to recognize the pre-processed real-time multimodal data, and determining the real-time emotion tag corresponding to the user;

[0064] Querying preset digital interactive scenes corresponding to real-time emotional tags; wherein the preset digital interactive scenes include but are not limited to customized music, generation of interactive guidance videos, or generation of interactive guidance voice;

[0065] Execute the preset digital interactive scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition.

[0066] On the other hand, the present invention provides a digital interactive system based on emotion recognition, comprising: a historical data acquisition module, a label acquisition module, a data relationship learning module, a real-time data acquisition module, and a digital interactive module;

[0067] The historical data acquisition module is used to collect historical multimodal data and preprocess the historical multimodal data to obtain preprocessed historical multimodal data; wherein the historical multimodal data includes video data and audio data;

[0068] The label acquisition module is used to display the historical multimodal data to the staff so that the staff can input the emotion label corresponding to the historical multimodal data;

[0069] The data relationship learning module is used to train the deep learning model using the pre-processed historical multimodal data and its corresponding emotion labels as training data to obtain an emotion recognition model;

[0070] The real-time data acquisition module is used to collect the real-time multimodal data of the user in real time during the interaction with the user, and pre-process the real-time multimodal data to obtain the real-time multimodal data after pre-processing;

[0071] The digital interaction module is used to use the emotion recognition model to identify the real-time multimodal data after preprocessing, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition.

[0072] The present invention provides a digital interaction method and system based on emotion recognition. By using the pre-processed historical multimodal data and its corresponding emotion labels as training data, a deep learning model is trained to obtain an emotion recognition model. In the process of interaction with the user, the emotion recognition model is used to identify the pre-processed real-time multimodal data, determine the real-time emotion label corresponding to the user, and finally execute the preset digital interaction scene corresponding to the real-time emotion label. This method can effectively realize the interaction in digital cemeteries and digital memorials, automatically recommend interactive memorial scenes, and thus bring a more humane remote memorial experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0074] Figure 1 A flowchart of a digital interaction method based on emotion recognition provided by an embodiment of the present invention.

[0075] Figure 2 A schematic structural diagram of a digital interactive system based on emotion recognition provided by an embodiment of the present invention.

[0076] The above drawings illustrate specific embodiments of the present invention, which will be described in more detail below. These drawings and the accompanying description are not intended to limit the scope of the present invention in any way, but rather to illustrate the concept of the present invention to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. In the following description, when referring to the drawings, like numbers in different figures represent like or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible embodiments consistent with the present invention. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present invention, as detailed in the appended claims.

[0078] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0079] like Figure 1 As shown, the present invention provides a digital interaction method based on emotion recognition, comprising:

[0080] S101: Collect historical multimodal data and preprocess the historical multimodal data to obtain preprocessed historical multimodal data; wherein the historical multimodal data includes video data and audio data;

[0081] The core idea of ​​multimodal emotion recognition is to analyze the overall emotional tendency of heterogeneous modal sources using a heuristic fusion strategy through learned modal feature representations. By integrating emotional information from different types of modalities, emotions can be predicted and classified more comprehensively and accurately, thereby improving the accuracy and robustness of the entire emotion recognition system. Therefore, the embodiment of the present invention uses historical multimodal data and the corresponding emotional labels corresponding to the historical multimodal data for model training, which can effectively improve the emotion recognition in the digital memorial process and ultimately enhance the interactive experience.

[0082] It is worth noting that the video data needs to include the user's face. During the preprocessing process, the face can also be tracked and cropped to obtain the user's facial area data during the digital interaction process and improve data recognition accuracy.

[0083] S102: presenting the historical multimodal data to a staff member, so that the staff member inputs an emotion label corresponding to the historical multimodal data;

[0084] Before emotion recognition, the historical multimodal data is first displayed to the staff, and more professional staff can give emotional labels. Then, in the subsequent process, the user's emotional labels can be automatically identified, and the preset digital interactive scenes corresponding to the emotional labels can be executed, helping staff to improve work efficiency and realize work automation, while also enhancing the user's more humane remote memorial experience.

[0085] S103, using the pre-processed historical multimodal data and its corresponding emotion labels as training data, training a deep learning model to obtain an emotion recognition model;

[0086] Optionally, the deep learning model provided by the embodiments of the present invention can be a single model or a combination of multiple models, depending on actual needs. After the deep learning model is trained using historical multimodal data and its corresponding emotion labels as training data, an emotion recognition model is obtained that assists personnel in identifying user emotions.

[0087] S104. During the interaction with the user (e.g., when the user is performing a digital memorial service), real-time multimodal data of the user is collected in real time, and the real-time multimodal data is pre-processed to obtain pre-processed real-time multimodal data.

[0088] In order to ensure that the data can be accurately identified, the processing process of real-time multimodal data is the same as that of historical multimodal data, so that the emotion recognition model can recognize the data.

[0089] S105. Use the emotion recognition model to identify the real-time multimodal data after preprocessing, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to achieve digital interaction based on emotion recognition.

[0090] By combining emotion recognition technology with digital cemeteries and memorial platforms, the present invention provides a new interactive mode that meets the diverse needs of modern people for funeral activities and has broad application prospects and social value.

[0091] In an embodiment of the present invention, collecting historical multimodal data and preprocessing the historical multimodal data to obtain preprocessed historical multimodal data includes:

[0092] Based on a preset time period, the user's video data and audio data within any time period are collected to obtain historical multimodal data corresponding to a single time period;

[0093] Historical multimodal data within a plurality of time periods is collected, and the historical multimodal data is preprocessed to obtain preprocessed historical multimodal data.

[0094] It is worth noting that the above two modal data are merely preferred examples of the embodiments of the present invention, and other modal data may also be used for recognition to improve the accuracy of interaction.

[0095] In an embodiment of the present invention, after preprocessing the historical multimodal data, obtaining the preprocessed historical multimodal data includes:

[0096] Sampling the video data in the historical multimodal data based on a preset data sampling frequency to obtain a plurality of video frames corresponding to the historical multimodal data;

[0097] Extracting speech features from audio data in the historical multimodal data, and constructing a speech feature data matrix based on the speech features;

[0098] The multiple video frames corresponding to the historical multimodal data and the voice feature data matrix are taken together as the historical multimodal data after preprocessing.

[0099] Optionally, the video frames may be enhanced to improve the accuracy of data recognition.

[0100] In an embodiment of the present invention, extracting speech features from audio data in the historical multimodal data and constructing a speech feature data matrix based on the speech features includes:

[0101] Extracting speech features from the audio data in the historical multimodal data, and forming a feature vector from the speech features;

[0102] Obtain the importance of each feature in the feature vector based on the trained LightGBM model, and sort the features in descending order by importance;

[0103] The average importance value corresponding to the speech features is obtained, and speech features with importance lower than the average importance value are filtered out. The optimal feature subset is selected using a sequential forward algorithm to obtain target speech features, and the target speech features are combined into a feature vector.

[0104] Optionally, the speech features may include features 1 to 809; wherein, features 1-8 are specifically: the mean, variance, maximum value and minimum value of short-time energy and its first-order difference; features 9-14 are specifically: the mean, variance and maximum value of sound intensity and its first-order difference; feature 15 is specifically: the average speaking rate; features 16-23 are specifically: the mean, variance, maximum value and minimum value of fundamental frequency and its first-order difference; features 24-53 are specifically: the mean, variance, maximum value, minimum value and median value of the first, second and third formant frequencies and their first-order differences; features 54-137 are specifically: the mean, variance, maximum value, minimum value, median value, range and sum of 1-12 order Mel cepstral coefficients MFCC; features 138-221 are specifically: 1-12 order gamma cepstral coefficients GFCC The mean, variance, maximum, minimum, median, range and sum of features 222-305 are specifically: the mean, variance, maximum, minimum, median, range and sum of 1-12 order Bark cepstral coefficients BFCC; features 306-389 are specifically: the mean, variance, maximum, minimum, median, range and sum of 1-12 order linear prediction coefficients LPC; features 390-473 are specifically: the mean, variance, maximum, minimum, median, range and sum of 1-12 order linear prediction cepstral coefficients LPCC; features 474-557 are specifically: the mean, variance, maximum, minimum, median, range and sum of 1-12 order normalized gamma chirp cepstral coefficients NGCC; features 558-641 are specifically: 1-12 order amplitude-based root spectral cepstral coefficients MSRCC The mean, variance, maximum, minimum, median, range and sum of features 642-725 are specifically: the mean, variance, maximum, minimum, median, range and sum of the phase-based root spectral cepstral coefficients PSRCC of order 1-12; the features 726-809 are specifically: the mean, variance, maximum, minimum, median, range and sum of the linear frequency cepstral coefficients LFCC of order 1-12.

[0105] It is worth noting that it is also possible to directly extract speech features from the audio data in the historical multimodal data, and directly construct the speech features into a speech feature data matrix, which can also realize the recognition of speech data.

[0106] In an embodiment of the present invention, the pre-processed historical multimodal data and its corresponding emotion labels are used as training data to train a deep learning model to obtain an emotion recognition model, including:

[0107] Constructing a first deep learning model and a second deep learning model;

[0108] Optionally, both the first deep learning model and the second deep learning model can be set as convolutional neural networks.

[0109] Initializing hyperparameters of a deep learning model to be trained to obtain multiple hyperparameter individuals; wherein the deep learning model to be trained is the first deep learning model or the second deep learning model;

[0110] Optionally, a random initialization or chaotic mapping initialization strategy can be used to initialize the hyperparameters of the deep learning model to be trained to obtain multiple hyperparameter individuals; wherein the hyperparameter individuals include all the hyperparameters to be trained of the deep learning model to be trained, and all the hyperparameters to be trained of the deep learning model to be trained can be all or part of its hyperparameters.

[0111] Obtaining the fitness of each of the hyperparameter individuals based on the preprocessed historical multimodal data and its corresponding sentiment labels, and determining the current optimal hyperparameter individual (i.e., the hyperparameter individual with the greatest fitness) in the current training process based on the fitness of each of the hyperparameter individuals;

[0112] For example, for the first deep learning model, after the hyperparameters in the hyperparameter individual are applied to the first deep learning model, the video frame can be used as input, the corresponding emotion label can be used as the expected output, the loss function value corresponding to the hyperparameter individual can be obtained, and after the loss function value is negative, the fitness corresponding to the hyperparameter individual can be obtained.

[0113] For the second deep learning model, after the hyperparameters in the hyperparameter individual are applied to the second deep learning model, the speech feature data matrix can be used as input, the corresponding emotion label can be used as the expected output, the loss function value corresponding to the hyperparameter individual can be obtained, and after the loss function value is negative, the fitness corresponding to the hyperparameter individual can be obtained.

[0114] Based on the current optimal hyperparameter individual, a time-adaptive local search strategy, a space-adaptive local search strategy, and a global jump search strategy are sequentially adopted to train multiple hyperparameter individuals;

[0115] After the total number of training times reaches the preset maximum number of training times, the target optimal hyperparameter individual is determined (i.e., the individual with the largest fitness is determined from the second target parameter individual in the last training process), and the target optimal hyperparameter individual is used as the final hyperparameter of the deep learning model to be trained, thereby obtaining the deep learning model to be trained after training;

[0116] After training the first deep learning model and the second deep learning model, a trained first deep learning model and a trained second deep learning model are obtained;

[0117] Remove the classification output layer of the first deep learning model after training to obtain the first feature extraction model (for convolutional neural networks, the Softmax classification output layer needs to be removed); remove the classification output layer of the second deep learning model after training to obtain the second feature extraction model;

[0118] Constructing a third deep learning model (e.g., using a long short-term memory network or a BP neural network to construct the third deep learning model), extracting first feature data using the first feature extraction model and extracting second feature data using the second feature extraction model based on the preprocessed historical multimodal data;

[0119] Training the third deep learning model as a deep learning model to be trained based on the first feature data, the second feature data, and the corresponding emotion labels to obtain a feature recognition model;

[0120] During the training of the third deep learning model, the first feature data and the second feature data are concatenated into a vector, which is used as input and the corresponding emotion label as the desired output to obtain fitness. It is worth noting that the first feature data and the second feature data corresponding to all video frames need to be concatenated into a single vector.

[0121] Optionally, in order to simplify the recognition process, the first deep learning model may not remove the classification output layer, and the classification labels corresponding to all video frames in the single historical multimodal data and the second feature data may be spliced ​​into a vector, and then the third deep learning model may be trained to obtain a feature recognition model.

[0122] An emotion recognition model is obtained based on the first feature extraction model, the second feature extraction model and the feature recognition model.

[0123] The existing technology is prone to falling into local optimality during the hyperparameter training process, and the training accuracy and effect are poor. Therefore, the present invention provides a time-adaptive local search strategy, a space-adaptive local search strategy and a global jump search strategy for joint search, thereby solving the problems existing in the existing technology, and ultimately being able to better realize the recognition of emotional labels and accurately complete the data recognition tasks specified by the staff.

[0124] In an embodiment of the present invention, the time-adaptive local search strategy includes:

[0125] Based on the current number of training times, the adaptive control factor of the generation time is:

[0126]

[0127] in, represents the time adaptive control factor, represents pi, Indicates the current number of training times. Indicates expectation (e.g. can be set to 0), represents the variance (which can be set to 1), represents the variance control parameter, exp represents the natural constant with the natural constant e as the base;

[0128] Based on the current optimal hyperparameter individual, a time-adaptive control factor is used to perform a time-adaptive local search on the hyperparameter individual, and the first target hyperparameter individual is obtained as:

[0129]

[0130] in, represents the jth hyperparameter individual in the tth training process, and j=1,2,…,K, K represents the total number of hyperparameter individuals, represents the jth first target hyperparameter individual, represents the first learning factor, Represents the current optimal hyperparameter individual.

[0131] The time-adaptive local search strategy provided by the embodiment of the present invention can make the hyperparameter individual produce certain fluctuations while learning the optimal position information, which can increase the ability to jump out of the local optimum to a certain extent, and at the same time will not affect the convergence of the algorithm in the later stage of the algorithm.

[0132] In an embodiment of the present invention, the spatially adaptive local search strategy includes:

[0133] For the i The first target hyperparameter individual, determine the i-1 first target hyperparameter individual is the first adjacent individual, determine the i +1 first target hyperparameter individual is the second adjacent individual; among them, for the first first target hyperparameter individual, its first adjacent individual is set to other random first target hyperparameter individuals; for the Kth first target hyperparameter individual, its second adjacent individual is set to other random first target hyperparameter individuals;

[0134] According to the first adjacent individual and the second adjacent individual, obtain the i The spatial adaptive control factor corresponding to the first target hyperparameter individual is:

[0135]

[0136]

[0137]

[0138] in, Indicates the i The spatial adaptive control factors corresponding to the first target hyperparameter individuals, represents the first scale factor, represents the second scale factor, Indicates the current number of training times, T indicates the preset maximum number of training times, Indicates the i The Euclidean distance between the first target hyperparameter individual and its corresponding first adjacent individual, Indicates the i The Euclidean distance between the first target hyperparameter individual and its corresponding second adjacent individual;

[0139] According to the said i The spatial adaptive control factor corresponding to the first target hyperparameter individual, i The first target hyperparameter individuals are spatially adaptively searched locally, and the second target parameter individuals are obtained as follows:

[0140]

[0141] in, Indicates the t During the training process i The first target hyperparameter individuals, Indicates the i The first neighboring individuals of the first target hyperparameter individual, Indicates the i The second adjacent individuals of the first target hyperparameter individual, represents the first random number between (0,1), Indicates the i The second target parameter individual.

[0142] The spatially adaptive local search strategy provided by the embodiment of the present invention can enable hyperparameter individuals to search based on their positions in space, try to search unfamiliar areas, avoid search collisions, and improve the ability to find the global optimum. As the algorithm converges, the positions of all hyperparameter individuals in the solution space gradually gather, which can increase the convergence accuracy.

[0143] In an embodiment of the present invention, the global jump search strategy includes:

[0144] According to the current number of training times, the adaptive global jump probability is obtained as:

[0145]

[0146] in, represents the adaptive global jump probability, sin represents the sine function, Indicates the preset maximum number of training times, t indicates the current number of training times, represents pi;

[0147] Obtain the fitness corresponding to all second target parameter individuals, and arrange the second target parameter individuals in descending order of fitness. According to the arranged second target parameter individuals, obtain the information fusion position corresponding to all second target parameter individuals as follows:

[0148]

[0149]

[0150] in, Indicates the t During the training n The second target parameter individuals after permutation, Represents the second target parameter individual The weighting coefficient of Indicates the information fusion position corresponding to all second target parameter individuals, Indicates the total number of individuals corresponding to the second target parameter;

[0151] For any second target parameter individual, determining a jump action corresponding to the second target parameter individual according to the adaptive global jump probability; wherein the jump action includes whether jumping is required or not;

[0152] For example, a random number between (0, 1) can be generated based on the adaptive global jump probability. When this random number is smaller than the adaptive global jump probability, a jump is required; otherwise, no jump is required.

[0153] When the jump action corresponding to the second target parameter individual is that a jump is required, a global jump search is performed on the second target parameter individual according to the information fusion position, and the global jump search position is obtained as:

[0154]

[0155] in, Indicates the t During the training m The second target parameter individuals, Represents the second target parameter individual The corresponding global jump search position, represents the second learning factor, represents the third learning factor, represents the second random number between (0,1), represents the third random number between (0,1), Indicates that except for the second target parameter individual Random second target parameter individuals other than ;

[0156] Determine whether the fitness of the global jump search position is greater than the fitness of the second target parameter individual. If so, use the global jump search position as the third target parameter individual; otherwise, use the original second target parameter individual as the third target parameter individual; wherein the third target parameter individual is the second target parameter individual after the global jump search.

[0157] The global jump search strategy provided by the embodiment of the present invention can effectively improve the global search capability of the algorithm. It has a strong global search capability in the early and middle stages of the algorithm, and can also provide a certain global search capability in the later stages, so that the hyperparameter individuals can search in areas far away from the hyperparameter individual aggregation (that is, away from the information fusion position) while searching globally. Combining the above two strategies, it ensures that the algorithm can effectively jump out of the local optimum, and introduces a greedy algorithm for search control to ensure the convergence speed of the algorithm.

[0158] Alternatively, instead of using a greedy algorithm for search control, an annealing simulation algorithm can be used for search control, which can also achieve control of the algorithm convergence speed. After each search, individuals can be processed for out-of-bounds to ensure that the hyperparameters are within the valid range.

[0159] By providing the above-mentioned hyperparameter optimization algorithm, the embodiment of the present invention can effectively improve the data learning relationship, more accurately achieve the data recognition tasks set by the staff, and ultimately improve the experience and accuracy of digital interaction.

[0160] In an embodiment of the present invention, the emotion recognition model is used to recognize the pre-processed real-time multimodal data, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition, including:

[0161] Using the emotion recognition model to recognize the pre-processed real-time multimodal data, and determining the real-time emotion tag corresponding to the user;

[0162] Querying preset digital interactive scenes corresponding to real-time emotional tags; wherein the preset digital interactive scenes include but are not limited to customized music, generation of interactive guidance videos, or generation of interactive guidance voice;

[0163] For example, pre-set digital interactive scenes can include playing customized music or generating a virtual memorial ceremony (video or audio), thereby providing a more personalized remote memorial experience. However, it is worth noting that the above-mentioned pre-set digital interactive scenes are merely examples of embodiments of the present invention, and other pre-set digital interactive scenes can also be used to enrich the digital interactive experience.

[0164] Execute the preset digital interactive scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition.

[0165] The present invention provides a digital interaction method based on emotion recognition. By using the pre-processed historical multimodal data and its corresponding emotion labels as training data, a deep learning model is trained to obtain an emotion recognition model. In the process of interaction with the user, the emotion recognition model is used to identify the pre-processed real-time multimodal data, determine the real-time emotion label corresponding to the user, and finally execute the preset digital interaction scene corresponding to the real-time emotion label. This method can effectively realize the interaction in digital cemeteries and digital memorials, automatically recommend interactive memorial scenes, and thus bring a more humane remote memorial experience.

[0166] like Figure 2 As shown, the embodiment of the present invention provides a digital interactive system based on emotion recognition, including: a historical data collection module 201, a label acquisition module 202, a data relationship learning module 203, a real-time data collection module 204 and a digital interactive module 205;

[0167] The historical data acquisition module 201 is used to collect historical multimodal data and preprocess the historical multimodal data to obtain preprocessed historical multimodal data; wherein the historical multimodal data includes video data and audio data;

[0168] The label acquisition module 202 is used to display the historical multimodal data to the staff so that the staff can input the emotion label corresponding to the historical multimodal data;

[0169] The data relationship learning module 203 is used to train a deep learning model using the pre-processed historical multimodal data and its corresponding emotion labels as training data to obtain an emotion recognition model;

[0170] The real-time data acquisition module 204 is used to collect the real-time multimodal data of the user in real time during the interaction with the user, and pre-process the real-time multimodal data to obtain the real-time multimodal data after pre-processing;

[0171] The digital interaction module 205 is used to use the emotion recognition model to identify the real-time multimodal data after preprocessing, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition.

[0172] An embodiment of the present invention provides a digital interactive system based on emotion recognition that can execute the above-mentioned method and technical solution. Its principles and beneficial effects are similar and will not be repeated here.

[0173] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art not disclosed herein. It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and variations can be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

[0174] The data involved in the present invention are all data authorized by the user or data transmitted by the user himself, and do not involve any operation of actively obtaining data.

Claims

1. A digital interaction method based on emotion recognition, characterized in that: include: Collecting historical multimodal data and preprocessing the historical multimodal data to obtain preprocessed historical multimodal data; wherein the historical multimodal data includes video data and audio data; Presenting the historical multimodal data to a staff member so that the staff member inputs an emotion label corresponding to the historical multimodal data; Using the pre-processed historical multimodal data and its corresponding emotion labels as training data, a deep learning model is trained to obtain an emotion recognition model; During the interaction with the user, real-time multimodal data of the user is collected in real time, and the real-time multimodal data is preprocessed to obtain the preprocessed real-time multimodal data; The emotion recognition model is used to identify the pre-processed real-time multimodal data, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition; The deep learning model is trained using the pre-processed historical multimodal data and its corresponding emotion labels as training data to obtain an emotion recognition model, including: Constructing a first deep learning model and a second deep learning model; Initializing hyperparameters of a deep learning model to be trained to obtain multiple hyperparameter individuals; wherein the deep learning model to be trained is the first deep learning model or the second deep learning model; Obtaining the fitness of each hyperparameter individual according to the preprocessed historical multimodal data and its corresponding emotion label, and determining the current optimal hyperparameter individual in the current training process based on the fitness of each hyperparameter individual; Based on the current optimal hyperparameter individual, a time-adaptive local search strategy, a space-adaptive local search strategy, and a global jump search strategy are sequentially adopted to train multiple hyperparameter individuals; After the total number of training times reaches a preset maximum number of training times, the target optimal hyperparameter individual is determined, and the target optimal hyperparameter individual is used as the final hyperparameter of the deep learning model to be trained, thereby obtaining the deep learning model to be trained after training; After training the first deep learning model and the second deep learning model, a trained first deep learning model and a trained second deep learning model are obtained; Removing the classification output layer of the first deep learning model after training to obtain a first feature extraction model; removing the classification output layer of the second deep learning model after training to obtain a second feature extraction model; Constructing a third deep learning model, extracting first feature data using the first feature extraction model and extracting second feature data using the second feature extraction model based on the preprocessed historical multimodal data; Training the third deep learning model as a deep learning model to be trained based on the first feature data, the second feature data, and the corresponding emotion labels to obtain a feature recognition model; An emotion recognition model is obtained based on the first feature extraction model, the second feature extraction model and the feature recognition model.

2. The digital interaction method based on emotion recognition according to claim 1, characterized in that: Collecting historical multimodal data and preprocessing the historical multimodal data to obtain preprocessed historical multimodal data includes: Based on a preset time period, the user's video data and audio data within any time period are collected to obtain historical multimodal data corresponding to a single time period; Historical multimodal data within a plurality of time periods is collected, and the historical multimodal data is preprocessed to obtain preprocessed historical multimodal data.

3. The digital interaction method based on emotion recognition according to claim 2, characterized in that: After preprocessing the historical multimodal data, the preprocessed historical multimodal data is obtained, including: Sampling the video data in the historical multimodal data based on a preset data sampling frequency to obtain a plurality of video frames corresponding to the historical multimodal data; Extracting speech features from audio data in the historical multimodal data, and constructing a speech feature data matrix based on the speech features; The multiple video frames corresponding to the historical multimodal data and the voice feature data matrix are taken together as the historical multimodal data after preprocessing.

4. The digital interaction method based on emotion recognition according to claim 3, characterized in that: Extracting speech features from audio data in the historical multimodal data and constructing a speech feature data matrix based on the speech features, including: Extracting speech features from the audio data in the historical multimodal data, and forming a feature vector from the speech features; Obtain the importance of each feature in the feature vector based on the trained LightGBM model, and sort the features in descending order by importance; The average importance value corresponding to the speech features is obtained, and speech features with importance lower than the average importance value are filtered out. The optimal feature subset is selected using a sequential forward algorithm to obtain target speech features, and the target speech features are combined into a feature vector.

5. The digital interaction method based on emotion recognition according to claim 1, characterized in that: The time-adaptive local search strategy includes: Based on the current number of training times, the adaptive control factor of the generation time is: in, represents the time adaptive control factor, represents pi, Indicates the current number of training times. Express expectations, represents the variance, represents the variance control parameter, exp represents the natural constant with the natural constant e as the base; Based on the current optimal hyperparameter individual, a time-adaptive control factor is used to perform a time-adaptive local search on the hyperparameter individual, and the first target hyperparameter individual is obtained as: in, represents the jth hyperparameter individual in the tth training process, and j=1,2,…,K, K represents the total number of hyperparameter individuals, represents the jth first target hyperparameter individual, represents the first learning factor, Represents the current optimal hyperparameter individual.

6. The digital interaction method based on emotion recognition according to claim 5, characterized in that: The spatially adaptive local search strategy includes: For the i The first target hyperparameter individual, determine the i -1 first target hyperparameter individual is the first adjacent individual, determine the i +1 first target hyperparameter individual is the second adjacent individual; among them, for the first first target hyperparameter individual, its first adjacent individual is set to other random first target hyperparameter individuals; for the Kth first target hyperparameter individual, its second adjacent individual is set to other random first target hyperparameter individuals; According to the first adjacent individual and the second adjacent individual, obtain the i The spatial adaptive control factor corresponding to the first target hyperparameter individual is: in, Indicates the i The spatial adaptive control factors corresponding to the first target hyperparameter individuals, represents the first scale factor, represents the second scale factor, Indicates the current number of training times, T indicates the preset maximum number of training times, Indicates the i The Euclidean distance between the first target hyperparameter individual and its corresponding first adjacent individual, Indicates the i The Euclidean distance between the first target hyperparameter individual and its corresponding second adjacent individual; According to the said i The spatial adaptive control factor corresponding to the first target hyperparameter individual, i The first target hyperparameter individuals are spatially adaptively searched locally, and the second target parameter individuals are obtained as follows: in, Indicates the t During the training process i First target hyperparameter individuals, Indicates the i The first neighboring individuals of the first target hyperparameter individual, Indicates the i The second adjacent individuals of the first target hyperparameter individual, represents the first random number between (0,1), Indicates the i The second target parameter individual.

7. The digital interaction method based on emotion recognition according to claim 6, characterized in that: The global jump search strategy includes: According to the current number of training times, the adaptive global jump probability is obtained as: in, represents the adaptive global jump probability, sin represents the sine function, Indicates the preset maximum number of training times, t indicates the current number of training times, represents pi; Obtain the fitness corresponding to all second target parameter individuals, and arrange the second target parameter individuals in descending order of fitness. According to the arranged second target parameter individuals, obtain the information fusion position corresponding to all second target parameter individuals as follows: in, Indicates the t During the training n The second target parameter individuals after permutation, Represents the second target parameter individual The weighting coefficient of Indicates the information fusion position corresponding to all second target parameter individuals, Indicates the total number of individuals corresponding to the second target parameter; For any second target parameter individual, determining a jump action corresponding to the second target parameter individual according to the adaptive global jump probability; wherein the jump action includes whether jumping is required or not; When the jump action corresponding to the second target parameter individual is that a jump is required, a global jump search is performed on the second target parameter individual according to the information fusion position, and the global jump search position is obtained as: in, Indicates the t During the training m The second target parameter individuals, Represents the second target parameter individual The corresponding global jump search position, represents the second learning factor, represents the third learning factor, represents the second random number between (0,1), represents the third random number between (0,1), Indicates that except for the second target parameter individual Random second target parameter individuals other than ; Determine whether the fitness of the global jump search position is greater than the fitness of the second target parameter individual. If so, use the global jump search position as the third target parameter individual; otherwise, use the original second target parameter individual as the third target parameter individual; wherein the third target parameter individual is the second target parameter individual after the global jump search.

8. The digital interaction method based on emotion recognition according to claim 7, characterized in that: The emotion recognition model is used to recognize the pre-processed real-time multimodal data, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition, including: Using the emotion recognition model to recognize the pre-processed real-time multimodal data, and determining the real-time emotion tag corresponding to the user; Querying preset digital interactive scenes corresponding to real-time emotional tags; wherein the preset digital interactive scenes include but are not limited to customized music, generation of interactive guidance videos, or generation of interactive guidance voice; Execute the preset digital interactive scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition.

9. A digital interactive system based on emotion recognition, which is capable of executing the digital interactive method based on emotion recognition according to any one of claims 1 to 8, characterized in that: include: Historical data collection module, label acquisition module, data relationship learning module, real-time data collection module and digital interaction module; The historical data acquisition module is used to collect historical multimodal data and preprocess the historical multimodal data to obtain preprocessed historical multimodal data; wherein the historical multimodal data includes video data and audio data; The label acquisition module is used to display the historical multimodal data to the staff so that the staff can input the emotion label corresponding to the historical multimodal data; The data relationship learning module is used to train the deep learning model using the pre-processed historical multimodal data and its corresponding emotion labels as training data to obtain an emotion recognition model; The real-time data acquisition module is used to collect the real-time multimodal data of the user in real time during the interaction with the user, and pre-process the real-time multimodal data to obtain the real-time multimodal data after pre-processing; The digital interaction module is used to use the emotion recognition model to identify the real-time multimodal data after preprocessing, determine the real-time emotion tag corresponding to the user, and execute the preset digital interaction scene corresponding to the real-time emotion tag to realize digital interaction based on emotion recognition.

Images