Multi-modal large model feedback emotion enhancement method and device, equipment and storage medium

Abstract

The invention discloses a multi-mode large model feedback emotion enhancement method and device, equipment and a storage medium, and can be applied to the technical field of emotion data processing. The feedback literacy score and the feedback literacy level are obtained by analyzing and quantifying the plurality of feedback literacy elements, the interaction emotion label is obtained by performing emotion recognition on the multi-mode interaction data of the current student, and the feedback historical characteristics are obtained by analyzing the feedback historical data of the current student. Inputting a student feedback emotion preference personalized model in combination with the feedback literacy score and the interaction emotion tag to obtain a feedback emotion preference tag, and inputting a multi-modal large model in combination with a to-be-generated feedback content theme, a feedback literacy level and the interaction emotion tag to perform emotion enhancement to obtain an emotion-enhanced feedback content. And the feedback content of emotion enhancement is optimized based on the man-machine symbiosis normal form, so that the efficiency of feedback accomplishment self-adaptive enhancement in the learning process of the artificial intelligence endowed students can be effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of emotion data processing, and in particular to a multimodal large model feedback emotion enhancement method and apparatus, device and storage medium. Background Art

[0002] In the relevant technology, feedback literacy is the ability and attitude of students to actively participate in feedback and use feedback to promote learning, and is an important part of learner literacy. Among them, emotional feedback can enhance students' learning motivation, improve learning satisfaction, and promote learning outcomes. However, in the current AI-enabled education, the interactive feedback between large models and students only focuses on knowledge-based questions and answers, lacks personalized emotional care for students in the feedback process, resulting in low student feedback motivation. In order to solve the problem of low feedback motivation, the existing technology increases emotional expression in feedback interaction by fine-tuning or fully adjusting the large model or simple prompt word engineering, but this method also has low continuous optimization and learning efficiency of the corresponding model, resulting in low efficiency in improving feedback literacy in the AI-enabled student learning process.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a multimodal large model feedback emotion enhancement method and device, equipment and storage medium, which can effectively improve the efficiency of artificial intelligence-enabled students' learning process feedback literacy adaptive enhancement.

[0005] To achieve the above object, an embodiment of the present application provides a multimodal large model feedback emotion enhancement method, the method comprising the following steps:

[0006] Obtaining feedback information of the current student, wherein the feedback information includes multiple feedback literacy elements;

[0007] Analyzing and quantifying the feedback information to obtain a feedback literacy score and a feedback literacy level;

[0008] Acquire first multimodal interaction data of the current student;

[0009] Performing emotion recognition according to the first multimodal interaction data to obtain an interaction emotion label;

[0010] Obtaining historical feedback data of the current student;

[0011] Analyzing the feedback history data to obtain feedback history features;

[0012] Inputting the feedback literacy score, the interactive emotion label and the feedback history feature into a student feedback emotion preference personalized model to obtain the feedback emotion preference label of the current student;

[0013] Obtain the subject of the feedback content to be generated for the current student;

[0014] Inputting the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into a multimodal large model for emotion enhancement to obtain emotion-enhanced feedback content;

[0015] The emotion-enhanced feedback content is optimized based on the human-computer symbiosis paradigm.

[0016] In some embodiments, analyzing and quantifying the feedback information to obtain a feedback literacy score and a feedback literacy level includes:

[0017] Dividing the multiple feedback literacy elements into dimensions;

[0018] quantifying the plurality of feedback literacy elements to obtain quantitative index scores of the feedback literacy elements;

[0019] Calculate the feedback literacy dimension score in each dimension according to the quantitative indicator score of the feedback literacy element;

[0020] Calculating the feedback literacy score according to the feedback literacy dimension score;

[0021] The feedback literacy level of the current student is determined according to the feedback literacy score and a preset level range.

[0022] In some embodiments, performing emotion recognition according to the first multimodal interaction data to obtain an interaction emotion label includes:

[0023] Preprocessing the first multimodal interaction data to obtain second multimodal interaction data;

[0024] Performing feature extraction on the second multimodal interaction data to obtain multimodal data features;

[0025] Emotion recognition is performed on the second multimodal interaction data according to the multimodal data features to obtain the interaction emotion label.

[0026] In some embodiments, extracting features from the second multimodal interaction data to obtain multimodal data features includes:

[0027] Extracting features of each modality in the second multimodal interaction data as single-modal features through a feature extractor; wherein the feature extractor performs optimization processing through modality similarity and contrast loss when performing feature extraction;

[0028] The fusion features of each modality are calculated according to the single modality features as the multimodal data features.

[0029] In some embodiments, performing emotion recognition on the second multimodal interaction data according to the multimodal data feature to obtain the interaction emotion label includes:

[0030] Calculating the similarity between each of the multimodal data features and the remaining multimodal data features;

[0031] Calculating a contrast loss based on the similarity;

[0032] constructing an objective function according to the contrastive loss;

[0033] Emotion recognition is performed on the second multimodal interaction data according to the objective function to obtain the interaction emotion label.

[0034] In some embodiments, the step of inputting the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level, and the interactive emotion label into a multimodal macro model for emotion enhancement to obtain emotion-enhanced feedback content includes:

[0035] Constructing a multimodal prompt template according to the subject of the feedback content to be generated, the feedback emotion preference label, the feedback literacy level and the interactive emotion label;

[0036] Generate an emotion control parameter according to the feedback emotion preference label, the feedback literacy level and the interactive emotion label;

[0037] Constructing a preset function according to the multimodal prompt template, the emotion control parameter and the multimodal macro model;

[0038] Emotion enhancement is performed according to the preset function to obtain the emotion enhanced feedback content.

[0039] In some embodiments, when inputting the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal macro model for emotion enhancement, the method further comprises the following steps:

[0040] The emotion-enhanced feedback content of different modalities is adjusted for consistency.

[0041] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a multimodal large model feedback emotion enhancement device, the device comprising:

[0042] The first module is used to obtain feedback information from the current students, and the feedback information includes multiple feedback literacy elements;

[0043] The second module is used to analyze and quantify the feedback information to obtain a feedback literacy score and a feedback literacy level;

[0044] The third module is used to obtain the first multimodal interaction data of the current student;

[0045] A fourth module is used to perform emotion recognition based on the first multimodal interaction data to obtain an interaction emotion label;

[0046] The fifth module is used to obtain the feedback history data of the current student;

[0047] The sixth module is used to analyze the feedback history data to obtain feedback history features;

[0048] The seventh module is used to input the feedback literacy score, the interactive emotion label and the feedback history feature into the student feedback emotion preference personalized model to obtain the feedback emotion preference label of the current student;

[0049] An eighth module is used to obtain the subject of the feedback content to be generated of the current student;

[0050] The ninth module is used to input the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal large model for emotion enhancement to obtain the emotion-enhanced feedback content;

[0051] The tenth module is used to optimize the emotion-enhanced feedback content based on the human-computer symbiosis paradigm.

[0052] To achieve the above object, another aspect of an embodiment of the present application provides an electronic device, including:

[0053] at least one processor;

[0054] at least one memory for storing at least one program;

[0055] When the at least one program is executed by the at least one processor, the at least one processor implements the above method.

[0056] To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program implements the above method when executed by a processor.

[0057] The embodiments of the present application include at least the following beneficial effects: The present application provides a multimodal large model feedback emotion enhancement method and apparatus, equipment and storage medium, which obtains feedback literacy scores and feedback literacy levels by analyzing and quantifying feedback information including multiple feedback literacy elements, and simultaneously performs emotion recognition on the current student's multimodal interaction data to obtain an interactive emotion label, and analyzes the current student's feedback history data to obtain feedback history features, and then inputs the feedback literacy score, interactive emotion label and feedback history features into the student feedback emotion preference personalized model to obtain the current student's feedback emotion preference label, and then inputs the feedback emotion preference label, the topic of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal large model for emotion enhancement to obtain emotion-enhanced feedback content, and optimizes the emotion-enhanced feedback content based on the human-computer symbiosis paradigm, thereby effectively improving the efficiency of adaptive enhancement of feedback literacy in the learning process of students empowered by artificial intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flow chart of the multimodal large model feedback emotion enhancement method provided in the embodiment of the present application;

[0059] Figure 2 It is a structural schematic diagram of a multimodal large model feedback emotion enhancement device provided in an embodiment of the present application;

[0060] Figure 3 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application, they are only examples of devices and methods consistent with some aspects of the embodiments of the present application.

[0062] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0063] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0064] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0065] Before describing the embodiments of the present application in detail, some nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0066] The symbiotic paradigm is a system design and operation model that emphasizes collaboration, mutual benefit, and co-evolution between humans and artificial intelligence systems. In this paradigm, humans and AI are not simply in a user-user relationship, but form an ecosystem of mutual benefit and common growth.

[0067] The embodiment of the present application provides a multimodal large model feedback emotion enhancement method and apparatus, equipment and storage medium. The present application obtains a feedback literacy score and a feedback literacy level by analyzing and quantifying feedback information including multiple feedback literacy elements, and at the same time performs emotion recognition on the current student's multimodal interaction data to obtain an interactive emotion label, and analyzes the current student's feedback history data to obtain feedback history features, and then inputs the feedback literacy score, interactive emotion label and feedback history features into the student feedback emotion preference personalized model to obtain the current student's feedback emotion preference label, and then inputs the feedback emotion preference label, the theme of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal large model for emotion enhancement to obtain emotion-enhanced feedback content, and optimizes the emotion-enhanced feedback content based on the human-computer symbiosis paradigm, thereby effectively improving the efficiency of adaptive enhancement of feedback literacy in the student learning process enabled by artificial intelligence.

[0068] The multimodal large model feedback emotion enhancement method provided in the embodiment of the present application relates to the field of emotional data processing technology. The multimodal large model feedback emotion enhancement method provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The server can also be a node server in the blockchain network; the software can be an application that implements the multimodal large model feedback emotion enhancement method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0070] Figure 1 is an optional flow chart of the multimodal large model feedback emotion enhancement method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S00 to S90.

[0071] Step S100, obtaining feedback information of the current student, the feedback information including multiple feedback literacy elements;

[0072] Step S110, analyzing and quantifying the feedback information to obtain a feedback quality score and a feedback quality level;

[0073] Step S120, obtaining first multimodal interaction data of the current student;

[0074] Step S130: performing emotion recognition according to the first multimodal interaction data to obtain an interaction emotion label;

[0075] Step S140, obtaining the current student's feedback history data;

[0076] Step S150: Analyze the feedback history data to obtain feedback history features;

[0077] Step S160: input the feedback literacy score, interactive emotion label and feedback history features into the student feedback emotion preference personalized model to obtain the current student's feedback emotion preference label;

[0078] Step S170, obtaining the subject of the feedback content to be generated for the current student;

[0079] Step S180: input the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal large model for emotion enhancement to obtain the emotion-enhanced feedback content;

[0080] Step S190: Optimizing the emotion-enhanced feedback content based on the human-machine symbiosis paradigm.

[0081] In the embodiment of the present application, in order to monitor the student's feedback literacy level in real time and realize the subsequent adaptive enhancement of emotional feedback, the present embodiment automatically analyzes and quantifies the feedback information after obtaining the current student's feedback information. The feedback information refers to the feedback data of the student's activity behavior data. It is understandable that the process of automatically analyzing and quantifying the feedback information in the present embodiment includes but is not limited to the following steps:

[0082] Step S210, dividing the multiple feedback literacy elements into dimensions;

[0083] Step S220, quantifying multiple feedback literacy factors to obtain quantitative index scores of the feedback literacy factors;

[0084] Step S230, calculating the feedback literacy dimension score in each dimension according to the quantitative indicator score of the feedback literacy element;

[0085] Step S240, calculating the feedback literacy score according to the feedback literacy dimension score;

[0086] Step S250: Determine the current student's feedback literacy level according to the feedback literacy score and a preset level range.

[0087] In this embodiment, feedback information can be divided into four dimensions: feedback awareness, feedback strategy, feedback effect, and feedback emotion. Feedback awareness refers to students' understanding of the importance and value of feedback, feedback strategy refers to the methods and techniques used by students in the feedback process, feedback effect refers to the impact of feedback on learning goals and grades, and feedback emotion refers to the positive or negative emotional response generated by students in the feedback process. Each dimension contains several specific elements, as shown in Table 1:

[0088] Table 1

[0089]

[0090]

[0091] Furthermore, in order to convert the feedback literacy elements into computable data, the present embodiment quantifies each feedback literacy element by adopting a variety of data sources and data types. Specifically, the various data sources include, but are not limited to, students' learning behavior data, learning performance data, learning emotion data, and learning feedback data. Learning behavior data refers to various operations and activities of students on the learning platform, such as number of logins, learning time, learning progress, learning frequency, learning interaction, etc.; learning performance data refers to the results of various assessments and tests of students on the learning platform, such as homework grades, test grades, exam grades, etc.; learning emotion data refers to various emotional expressions and feedback of students on the learning platform, such as emoticons, emotional words, emotional evaluations, etc.; learning feedback data refers to various feedback information received and given by students on the learning platform, such as feedback content, feedback form, feedback frequency, feedback quality, etc. Based on the above data, the present embodiment combines a series of quantitative indicators to measure the quantitative indicator scores of each feedback literacy element, as shown in Table 2:

[0092] Table 2

[0093]

[0094] It can be understood that this embodiment uses the weighted average method to calculate the score y of each feedback literacy dimension based on the quantitative index of each feedback literacy factor. i , as shown in Formula 1:

[0095]

[0096] In the formula, w i is the weight of the i-th feedback literacy factor, x i is the quantitative index score of the i-th feedback literacy factor, and n is the number of factors under the feedback literacy dimension. i The determination can be made based on expert scoring or data analysis.

[0097] After obtaining the feedback literacy score for each dimension, the feedback literacy score z is calculated using the AHP method using Formula 2:

[0098]

[0099] In the formula, w j is the weight of the jth feedback literacy dimension, y j is the score of the jth feedback literacy dimension, and m is the number of dimensions of feedback literacy. Weight v j The determination can also be made based on expert scoring or data analysis methods.

[0100] Then, the current student's feedback literacy level L is determined according to the feedback literacy score and the preset level range. Specifically, this embodiment uses a five-level evaluation method to divide the feedback literacy level into excellent, good, medium, poor and insufficient, as shown in Table 3:

[0101] Table 3

[0102]

[0103] As shown in Table 3, the level of feedback literacy reflects students' abilities and attitudes in terms of feedback needs, trust, acceptance, seeking, utilization, reflection, satisfaction, improvement, achievement and emotion, and has an important impact on the personalized generation of students' emotional feedback content.

[0104] In the embodiment of the present application, the process of performing emotion recognition according to the first multimodal interaction data to obtain an interaction emotion label includes but is not limited to the following steps:

[0105] Step S310: preprocessing the first multimodal interaction data to obtain second multimodal interaction data;

[0106] Step S320: extracting features from the second multimodal interaction data to obtain multimodal data features;

[0107] Step S330: Perform emotion recognition on the second multimodal interaction data according to the multimodal data features to obtain the interaction emotion label.

[0108] It is understandable that the first multimodal interaction data can be a multimodal data collected in real time by using a variety of sensors and devices, such as cameras, microphones, keyboards, mice, touch screens, etc., during the interactive feedback process of students, including but not limited to visual data, voice data, text data, behavioral data, etc. Visual data refers to students' facial expressions, eye movements, head postures, etc.; voice data refers to students' voice signals, intonation, speech speed, etc.; text data refers to students' input content, language style, emotional vocabulary, etc.; behavioral data refers to students' operation frequency, operation duration, operation type, etc.

[0109] After obtaining the first multimodal interaction data, the first multimodal interaction data is subjected to data cleaning, data alignment, and data standardization. Data cleaning refers to removing invalid data, noise data, abnormal data, etc.; data alignment refers to synchronizing data of different modes according to timestamps; and data standardization refers to converting data of different modes into a unified format and scale.

[0110] After completing the preprocessing of the data, feature extraction is performed on the second multimodal interaction data to obtain multimodal data features. Specifically, the feature extractor can extract the features of each modality in the second multimodal interaction data as unimodal features, and then calculate the fusion features of each modality according to the unimodal features as multimodal data features. Among them, the feature extractor performs optimization processing through modal similarity and contrast loss when performing feature extraction. It can be understood that this embodiment can extract features that are helpful for emotion recognition from multimodal interaction data by utilizing a highly semantically centralized cross-sample fusion network. Specifically, a unimodal feature extractor is first used to extract features from each modality as unimodal features, and then unimodal contrastive learning with high semantic centralization is used to retain modality-specific information, and finally cross-sample fusion with high semantic centralization is used to fuse features of different modalities.

[0111] Exemplarily, the processing of this embodiment may be performed in the following manner:

[0112] First, extract unimodal features: For each modality m, use the feature extractor f m From the second multimodal interaction data x m Extract feature z m , that is, z m =f m (x m ).

[0113] Second, single-modal contrastive learning with high semantic centralization: For each modality m, calculate its similarity s with the modality with high semantic centralization (such as text, numerical value with clear meaning) mt , and use the contrastive loss L m To optimize the feature extractor f m ,Right now and Where d is the feature dimension, and τ is the temperature parameter. This embodiment focuses on how to use text information to guide feature extraction and alignment of other modalities to retain modality-specific information.

[0114] Third, cross-sample fusion with high semantic centralization: Calculate the fusion feature h of each modality m m ,Right now in is the attention weight and N is the number of samples.

[0115] This embodiment extracts features that are helpful for emotion recognition from multimodal interaction data while retaining modality-specific information, thereby improving the accuracy and robustness of emotion feature representation learning.

[0116] After extracting the multimodal data features, emotion recognition is performed on the second multimodal interaction data based on the multimodal data features. It can be understood that this embodiment can use a cross-sample fusion network with high semantic modality centralization to perform emotion recognition on multimodal interaction data, and effectively integrate the features of different modalities to construct a multimodal emotion recognition model, so as to achieve real-time recognition of students' interactive emotions. Specifically, first use two-way contrast learning of the fused modality to optimize the fusion features, and then use the objective function and prediction to achieve emotion recognition. Exemplarily, the processing process of this embodiment can be adopted as follows:

[0117] First, two-way contrastive learning of fused modalities: Calculate the fusion feature h of each modality m m The fusion feature h of the remaining modalities m′ The similarity of mm′ , and use the contrastive loss L mm′ To optimize the fusion features, and Where d is the feature dimension and τ is the temperature parameter. This embodiment can improve the accuracy and robustness of emotion recognition by effectively integrating features of different modalities.

[0118] Step 2: Objective function and prediction: This embodiment constructs an objective function JJ based on contrast loss, and its objective function J includes sentiment classification loss L c , contrast loss L mm′ And the regularization term R, that is, J = L c +λL mm′ +γR, where λ and γ are weight parameters. In this embodiment, the objective function is optimized by the gradient descent method to obtain the optimal model parameters, and then the multimodal data is subjected to emotion recognition based on the objective function to obtain the interactive emotion label E.

[0119] Specifically, the interactive emotion label E is a multivariate vector that represents the emotional state of the student during the interactive feedback process, such as happiness, sadness, anger, fear, surprise, disgust, etc. The interactive emotion label E reflects the emotional response of the student during the interactive feedback process, and has an important impact on the personalized generation of the student's emotional feedback content. This embodiment further enhances the complementarity and synergy between modalities through a two-way contrast learning method of fusion modalities, and can realize real-time recognition of students' interactive emotions.

[0120] When obtaining the interactive emotion label E, this embodiment also analyzes the student's feedback history data to obtain each student's feedback history feature H as one of the inputs of the student's feedback emotion preference personalized model. Feedback history data refers to various feedback information received and given by students on the learning platform, such as feedback content, feedback form, feedback frequency, feedback quality, etc. The feedback history feature H is a multivariate vector that represents the student's preference, habit, style, etc. for feedback content, such as the theme, length, tone, language, emotion, etc. of the feedback content. The feedback history feature H reflects the student's personalized needs for feedback content and has an important impact on the personalized generation of students' emotional feedback content.

[0121] It can be understood that the student feedback sentiment preference personalized model can use the multi-layer perceptron (MLP) to construct the student feedback sentiment preference personalized model P, which takes the feedback literacy score z, the interactive sentiment label E and the feedback history feature H as input and outputs the student feedback sentiment preference label S = {s1, s2, ..., s k}, k is the number of emotional dimensions. Among them, the feedback emotional preference label S is used as the input of multimodal AIGC to provide guidance for the generation of multimodal content enhanced by emotional feedback. Specifically, the feedback emotional preference label S is a multivariate vector that represents the student's emotional preference for feedback content, such as positive, negative, neutral, motivational, encouraging, comforting, praising, criticizing, etc. The feedback emotional preference label S is the core output of the personalized model of student feedback emotional preference and is the key to realizing the personalized generation of student emotional feedback content.

[0122] In the embodiment of the present application, after obtaining the feedback emotion preference label S, the present embodiment can generate multimodal feedback content with emotion enhancement effect by utilizing the generation capability of the multimodal large model and combining the feedback emotion preference of the students. It is understandable that the present embodiment includes but is not limited to the following steps:

[0123] Step S410: construct a multimodal prompt template according to the feedback content theme T to be generated, the feedback emotion preference label S, the feedback literacy level L and the interactive emotion label E; L∈{excellent, good, medium, poor, insufficient};

[0124] Step S420: generating emotion control parameters according to the feedback emotion preference label, the feedback literacy level and the interaction emotion label;

[0125] Step S430, constructing a preset function according to the multimodal prompt template, the emotion control parameter and the multimodal large model;

[0126] Step S440: Perform emotion enhancement according to a preset function to obtain emotion enhanced feedback content.

[0127] Specifically, the process of constructing the multimodal prompt template can be by constructing a function ConstructEmotionalPrompt(T, S, L, E), which integrates the input information into a multimodal prompt template Ψ. The specific implementation is as follows: Ψ=Concat(W t ·T,W s ·S,W l ·L,W e ·E,Ψ base ), where W t ,W s ,W l ,W e are the embedding matrices of the corresponding inputs, Ψ base It is a basic multimodal prompt template, which can ensure that emotional factors are effectively integrated into the prompt template.

[0128] The process of constructing the emotion control parameters can be achieved by designing a function GenerateEmotionControlParams(S,L,E) to generate the parameter Λ for the emotion intensity in the control generation process. This function maps the emotion preference, feedback literacy level and interaction emotion to the control parameters of each mode: Λ={λ text ,λ image ,λ audio}=f(S,L,E), where f is a mapping function that generates appropriate emotion control parameters based on the input.

[0129] Then, according to the multimodal prompt template Ψ, the emotional control parameter Λ and the multimodal large model M, a preset function EmotionalMultiModalGenerate(M,Ψ,Λ) is constructed. This function uses the multimodal large model M to generate emotionally enhanced feedback content C based on the prompt template Ψ and the emotional control parameter Λ. This function generates text, image and audio content respectively, and applies the corresponding emotional control parameters in the generation process to ensure that the generated content meets the required emotional characteristics.

[0130] In an embodiment of the present application, when emotion enhancement is performed through a multimodal large model, this embodiment will also adjust the consistency of the emotion-enhanced feedback content of different modalities. Specifically, this embodiment designs a function AdjustModalConsistency(C) to ensure that the content between different modalities is consistent. This function mainly performs the following operations: a) ensures that the generated image content is semantically consistent with the text description; b) ensures that the generated audio content is semantically consistent with the text content. This function can ensure the unity of multimodal feedback content in semantics and emotional expression.

[0131] This embodiment fully utilizes the capabilities of the multimodal large model, designs prompts and control parameters, and incorporates the required emotional features during the generation process, thereby achieving more accurate and efficient emotional feedback enhancement, improving the quality and pertinence of feedback content, and providing new technical support for personalized learning and emotional intelligence education.

[0132] In the embodiment of the present application, after obtaining the emotion-enhanced feedback content, the emotion-enhanced feedback content can also be continuously optimized based on the human-computer symbiosis paradigm. Among them, this embodiment can continuously evolve through external knowledge base updates, prompt engineering optimization and in-context learning, thereby continuously improving students' feedback literacy. It is understandable that the process of continuous optimization includes but is not limited to the following steps:

[0133] Step S510, collecting feedback literacy data: collecting key data after students use emotion-enhanced feedback:

[0134] The student's feedback literacy score R = {r1, r2, ..., r n}, corresponding to the elements of feedback literacy defined in 4.1;

[0135] Students’ emotional response to the generated content E = {e1, e2, ..., e m};

[0136] Student feedback behavior data B = {b1, b2, ..., b k}, such as feedback frequency, feedback quality, etc.;

[0137] Step S520, update the external knowledge base: based on the collected data, update the external knowledge base K: K t+1 =f K (K t ,R,E,B), where f K is the knowledge base update function, t represents the current iteration number;

[0138] Step S530, prompt engineering optimization: optimize the above multimodal prompt template Ψ according to the updated knowledge base: Ψ t+1 =f Ψ (Ψ f ,K t+1 ), where f Ψ It is the prompt template optimization function;

[0139] Step S540, update In-context learning: maintain a personalized context memory M for each student i i , and update it: where f M is the context memory update function;

[0140] Step S550, adaptive feedback generation: when generating new emotion enhancement feedback, the optimized prompt template, the updated knowledge base and the personalized context memory are combined: where f F is the adaptive feedback generation function, where T, S, L, and E represent feedback topic, emotional preference, literacy level, and interactive emotion, respectively;

[0141] Step S560, evaluate the effect of improving feedback literacy: evaluate the effect of emotion enhancement feedback on improving students' feedback literacy: ΔR = f Δ (R before ,R after ), where f Δ is the feedback literacy improvement evaluation function;

[0142] Step S570, loop optimization: continue to execute step 510 to step S560 to form a closed-loop optimization process. Specifically, the iterative formula can be expressed as: (K, Ψ, M i ,F) t+1 =f opt ((K,Ψ,M i ,F) t ,R,E,B), where f opt Represents the entire optimization cycle;

[0143] Step S580: Tracking long-term effects: Tracking and recording the long-term changing trend of students' feedback literacy: T R =f T ({R t}), where f T is the long-term progress tracking function, R t Represents the feedback literacy score at the tth time point.

[0144] In this embodiment, a dynamically evolving external system (K, Ψ, M i ) is used to enhance the emotional feedback capability of the multimodal large model, thereby achieving continuous optimization of the content generated by the emotion-enhanced feedback and continuous improvement of students' feedback literacy, thereby improving the adaptability and personalization of the system to which the present application method is applied, and also providing a new paradigm for sustainable development for artificial intelligence-assisted education. Through human-machine symbiosis, it can learn and evolve with students, continuously provide more accurate and effective emotion-enhanced feedback, and thus continuously improve students' feedback literacy.

[0145] In the embodiment of the present application, the application of the human-machine symbiosis paradigm in the present embodiment is mainly reflected in the following aspects:

[0146] Two-way interaction: The system used in this application method can not only provide students with emotion-enhancing feedback, but also learn and improve from students' responses.

[0147] Co-evolution: As students’ feedback literacy improves, the system used in this application method is also constantly optimizing its feedback generation capabilities.

[0148] Personalized Adaptation: The system used in this application method adapts to the unique needs and preferences of each student through continuous learning.

[0149] Knowledge sharing: The external knowledge base of the system applied by this application method continuously accumulates new knowledge from student interactions, forming a dynamically updated knowledge ecosystem.

[0150] Complementary advantages: The system used in this application method utilizes the computing power of AI and human creative thinking to complement each other and achieve the effect of 1+1>2.

[0151] Continuous optimization: This application method is applied through a circular feedback mechanism, and the system and students applied by this application method are constantly improving and growing in this process.

[0152] Common goal: The system used in this application method and the students are committed to improving learning outcomes and feedback literacy, forming a symbiotic relationship with consistent goals.

[0153] This symbiotic paradigm makes the AI-assisted education system no longer a static tool, but an intelligent partner that can grow and evolve with learners. This provides a new idea for the application of artificial intelligence in the field of education, and is expected to bring a more personalized, intelligent and sustainable learning experience.

[0154] Reference Figure 2 The embodiment of the present application provides a multi-modal large model feedback emotion enhancement device, the device comprising:

[0155] The first module 600 is used to obtain feedback information of the current student, and the feedback information includes multiple feedback literacy elements;

[0156] The second module 610 is used to analyze and quantify the feedback information to obtain a feedback literacy score and a feedback literacy level;

[0157] The third module 620 is used to obtain the first multimodal interaction data of the current student;

[0158] The fourth module 630 is used to perform emotion recognition according to the first multimodal interaction data to obtain an interaction emotion label;

[0159] The fifth module 640 is used to obtain the feedback history data of the current student;

[0160] The sixth module 650 is used to analyze the feedback history data to obtain feedback history features;

[0161] The seventh module 660 is used to input the feedback literacy score, the interactive emotion label and the feedback history feature into the student feedback emotion preference personalized model to obtain the current student's feedback emotion preference label;

[0162] The eighth module 670 is used to obtain the subject of the feedback content to be generated for the current student;

[0163] The ninth module 680 is used to input the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal large model for emotion enhancement to obtain the emotion-enhanced feedback content;

[0164] The tenth module 690 is used to optimize the feedback content of emotion enhancement based on the human-computer symbiosis paradigm.

[0165] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0166] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above multi-modal large model feedback emotion enhancement method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0167] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0168] See also Figure 3 , Figure 3 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0169] The processor 710 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0170] The memory 720 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 720 can store an operating system and other applications. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 720, and the processor 710 calls and executes the multi-modal large model feedback emotion enhancement method of the embodiment of this application;

[0171] Input / output interface 730, used to implement information input and output;

[0172] Communication interface 740, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0173] bus 750 , which transmits information between the various components of the device (e.g., processor 710 , memory 720 , input / output interface 730 , and communication interface 740 );

[0174] The processor 710 , the memory 720 , the input / output interface 730 , and the communication interface 740 are connected to each other in communication within the device via the bus 750 .

[0175] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned multimodal large model feedback emotion enhancement method.

[0176] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0177] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0179] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0180] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0181] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0182] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0183] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0184] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0185] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0188] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A multimodal large model feedback emotion enhancement method, characterized in that: The method comprises the following steps: Obtaining feedback information of the current student, wherein the feedback information includes multiple feedback literacy elements; Analyzing and quantifying the feedback information to obtain a feedback literacy score and a feedback literacy level; Acquire first multimodal interaction data of the current student; Performing emotion recognition according to the first multimodal interaction data to obtain an interaction emotion label; Obtaining historical feedback data of the current student; Analyzing the feedback history data to obtain feedback history features; Inputting the feedback literacy score, the interactive emotion label and the feedback history feature into a student feedback emotion preference personalized model to obtain the feedback emotion preference label of the current student; Obtain the subject of the feedback content to be generated for the current student; Inputting the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into a multimodal large model for emotion enhancement to obtain emotion-enhanced feedback content; The emotion-enhanced feedback content is optimized based on the human-computer symbiosis paradigm.

2. The method according to claim 1, characterized in that The analyzing and quantifying the feedback information to obtain a feedback quality score and a feedback quality level includes: Dividing the multiple feedback literacy elements into dimensions; quantifying the plurality of feedback literacy elements to obtain quantitative index scores of the feedback literacy elements; Calculate the feedback literacy dimension score in each dimension according to the quantitative indicator score of the feedback literacy element; Calculating the feedback literacy score according to the feedback literacy dimension score; The feedback literacy level of the current student is determined according to the feedback literacy score and a preset level range.

3. The method according to claim 1, characterized in that The performing emotion recognition according to the first multimodal interaction data to obtain an interaction emotion label includes: Preprocessing the first multimodal interaction data to obtain second multimodal interaction data; Performing feature extraction on the second multimodal interaction data to obtain multimodal data features; Emotion recognition is performed on the second multimodal interaction data according to the multimodal data features to obtain the interaction emotion label.

4. The method according to claim 3, characterized in that The extracting features of the second multimodal interaction data to obtain multimodal data features includes: Extracting features of each modality in the second multimodal interaction data as single-modal features through a feature extractor; wherein the feature extractor performs optimization processing through modality similarity and contrast loss when performing feature extraction; The fusion features of each modality are calculated according to the single modality features as the multimodal data features.

5. The method according to claim 3, characterized in that: The performing emotion recognition on the second multimodal interaction data according to the multimodal data feature to obtain the interaction emotion label includes: Calculating the similarity between each of the multimodal data features and the remaining multimodal data features; Calculating a contrast loss based on the similarity; constructing an objective function according to the contrastive loss; Emotion recognition is performed on the second multimodal interaction data according to the objective function to obtain the interaction emotion label.

6. The method according to claim 1, characterized in that The step of inputting the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal macro model for emotion enhancement to obtain the emotion-enhanced feedback content includes: Constructing a multimodal prompt template according to the subject of the feedback content to be generated, the feedback emotion preference label, the feedback literacy level and the interactive emotion label; Generate an emotion control parameter according to the feedback emotion preference label, the feedback literacy level and the interactive emotion label; Constructing a preset function according to the multimodal prompt template, the emotion control parameter and the multimodal macro model; Emotion enhancement is performed according to the preset function to obtain the emotion enhanced feedback content.

7. The method according to claim 6, characterized in that When the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label are input into the multimodal macro model for emotion enhancement, the method further comprises the following steps: The emotion-enhanced feedback content of different modalities is adjusted for consistency.

8. A multi-modal large model feedback emotion enhancement device, characterized in that: The device comprises: The first module is used to obtain feedback information from the current students, and the feedback information includes multiple feedback literacy elements; The second module is used to analyze and quantify the feedback information to obtain a feedback literacy score and a feedback literacy level; The third module is used to obtain the first multimodal interaction data of the current student; A fourth module is used to perform emotion recognition based on the first multimodal interaction data to obtain an interaction emotion label; The fifth module is used to obtain the feedback history data of the current student; The sixth module is used to analyze the feedback history data to obtain feedback history features; The seventh module is used to input the feedback literacy score, the interactive emotion label and the feedback history feature into the student feedback emotion preference personalized model to obtain the feedback emotion preference label of the current student; An eighth module is used to obtain the subject of the feedback content to be generated of the current student; The ninth module is used to input the feedback emotion preference label, the subject of the feedback content to be generated, the feedback literacy level and the interactive emotion label into the multimodal large model for emotion enhancement to obtain the emotion-enhanced feedback content; The tenth module is used to optimize the emotion-enhanced feedback content based on the human-computer symbiosis paradigm.

9. An electronic device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images