Emotional text determination method and device and electronic equipment

Abstract

The invention discloses an emotion text determination method and device and electronic equipment. The method comprises the steps that target audio and video data to be subjected to sentiment analysis are acquired, and the video data are video data corresponding to the target audio; determining key image information corresponding to the key time information in the target audio from the video data; according to the expression change characteristics corresponding to the key image information, determining expression change scores, the expression change scores being used for determining the degree of expression change; according to the expression change score and the keyword information of the target audio, different types of emotion texts are generated; testing according to different types of emotion texts to obtain feedback scores; and determining a target sentiment text from the different types of sentiment texts according to the feedback score. The technical problem of poor accuracy of the emotion text generated according to the audio and video data in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and in particular, to a method, device and electronic device for determining emotional text. Background Art

[0002] Sentiment analysis refers to the use of natural language processing, text analysis, computational linguistics, and biometrics to systematically identify, extract, quantify, and study emotional states and subjective information. Sentiment analysis is widely used to analyze the voice of the customer, such as reviews and survey responses, online and on social media, and in healthcare materials from customer service to clinical medicine.

[0003] In the related art, the emotional text generated based on audio and video data is not subjected to secondary analysis, resulting in poor accuracy. Therefore, the emotional text is one-sided. When the emotional text is subsequently put into use, there will be certain errors in the analysis of the emotional fluctuations of the tester.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, device and electronic device for determining an emotional text, so as to at least solve the technical problem of poor accuracy of the emotional text generated based on audio and video data in the related art.

[0006] According to one aspect of an embodiment of the present application, a method for determining an emotional text is provided, comprising: obtaining target audio and video data to be subjected to emotional analysis, wherein the video data is video data corresponding to the target audio; determining key image information corresponding to key time information in the target audio from the video data; determining an expression change score based on expression change features corresponding to the key image information, wherein the expression change score is used to determine the degree of expression change; generating different types of emotional texts based on the expression change scores and keyword information of the target audio; performing tests based on different types of emotional texts to obtain feedback scores; and determining a target emotional text from different types of emotional texts based on the feedback scores.

[0007] Optionally, after obtaining the target audio and video data to be subjected to emotion analysis, the method further includes: sampling the target audio to obtain a discrete audio signal; quantizing the discrete audio signal to obtain a quantized audio signal; encoding the quantized audio signal to obtain a digital audio signal; converting the digital audio signal from the time domain to the frequency domain through a fast Fourier transform algorithm, filtering the digital audio signal in the frequency domain, and converting the processed digital audio signal from the frequency domain to the time domain through an inverse Fourier transform algorithm to obtain pure target audio.

[0008] Optionally, after obtaining the pure target audio, the method also includes: analyzing and processing the pure target audio to obtain audio decibel data; determining key decibel data that meets a preset threshold from the audio decibel data; and determining key time information and keyword information corresponding to the key decibel data from the pure target audio.

[0009] Optionally, the expression change feature is determined by: obtaining preset normal image information; comparing the normal image information and the key image information to determine a changed image in the key image information, wherein the changed image is used to represent an image in the key image information that is inconsistent with the normal image information; extracting features of the changed image to obtain the expression change feature.

[0010] Optionally, determining an expression change score based on expression change features corresponding to key image information includes: determining normal expression features corresponding to the changed image from normal image information; comparing normal expression features with expression change features to obtain a comparison result; when the comparison result indicates that the degree of overlap between the expression change features and the normal expression features is greater than or equal to a first threshold, determining the expression change score to be a first expression change score; when the comparison result indicates that the degree of overlap between the expression change features and the normal expression features is less than the first threshold and greater than or equal to a second threshold, determining the expression change score to be a second expression change score, wherein the first threshold is greater than the second threshold; when the comparison result indicates that the degree of overlap between the expression change features and the normal expression features is less than the second threshold and greater than or equal to a third threshold, determining the expression change score to be a third expression change score, wherein the second threshold is greater than the third threshold.

[0011] Optionally, different types of emotional texts are generated based on the expression change score and the keyword information of the target audio, including: generating a first text outline based on the first expression change score and the first keyword information in the keyword information, wherein the first keyword information is the keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is greater than or equal to the first threshold; filling in the first text outline to obtain a first emotional text; generating a second text outline based on the second expression change score and the second keyword information in the keyword information, wherein the second keyword information is the keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is less than the first threshold and greater than or equal to the second threshold; filling in the second text outline to obtain a second emotional text; generating a third text outline based on the third expression change score and the third keyword information in the keyword information, wherein the third keyword information is the keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is less than the second threshold and greater than or equal to the third threshold; filling in the third text outline to obtain a third emotional text; determining different types of emotional texts based on the first emotional text, the second emotional text and the third emotional text.

[0012] Optionally, testing is performed based on different types of emotional texts to obtain feedback scores, including: obtaining first feedback expression change characteristics and first feedback decibel data obtained by the tester based on the first emotional text; obtaining second feedback expression change characteristics and second feedback decibel data obtained by the tester based on the second emotional text; obtaining third feedback expression change characteristics and third feedback decibel data obtained by the tester based on the third emotional text; determining a first feedback score based on the first feedback expression change characteristics and the first feedback decibel data, determining a second feedback score based on the second feedback expression change characteristics and the second feedback decibel data, and determining a third feedback score based on the third feedback expression change characteristics and the third feedback decibel data.

[0013] Optionally, a target emotional text is determined from different types of emotional texts based on feedback scores, including: respectively evaluating the first emotional text, the second emotional text, and the third emotional text to obtain a first evaluation score, a second evaluation score, and a third evaluation score; comparing the first evaluation score and the first feedback score, the second evaluation score and the second feedback score, and the third evaluation score and the third feedback score to obtain a first text quality comprehensive score for the first emotional text, a second text quality comprehensive score corresponding to the second emotional text, and a third text quality comprehensive score corresponding to the third emotional text; and determining the emotional text corresponding to the highest score among the first text quality comprehensive score, the second text quality comprehensive score, and the third text quality comprehensive score as the target emotional text.

[0014] According to another aspect of an embodiment of the present application, a device for determining an emotional text is also provided, including: an acquisition module, used to acquire target audio and video data to be subjected to emotional analysis, wherein the video data is video data corresponding to the target audio; a first determination module, used to determine key image information corresponding to key time information in the target audio from the video data; a second determination module, used to determine an expression change score based on expression change features corresponding to the key image information, wherein the expression change score is used to determine the degree of expression change; a generation module, used to generate different types of emotional texts based on the expression change scores and keyword information of the target audio; a testing module, used to perform tests based on different types of emotional texts to obtain feedback scores; and a third determination module, used to determine a target emotional text from different types of emotional texts based on the feedback scores.

[0015] According to another aspect of the embodiment of the present application, there is also provided an electronic device, including: a memory and a processor, the memory being used to store program instructions; the processor being connected to the memory and being used to execute the method for determining the above-mentioned emotional text.

[0016] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method for determining the emotional text by running the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is also provided, including computer instructions, which implement the above-mentioned method for determining emotional text when executed by a processor.

[0018] In an embodiment of the present application, target audio and video data to be subjected to sentiment analysis are obtained, wherein the video data is video data corresponding to the target audio; key image information corresponding to key time information in the target audio is determined from the video data; an expression change score is determined based on the expression change features corresponding to the key image information, wherein the expression change score is used to determine the degree of expression change; different types of sentiment texts are generated based on the expression change score and keyword information of the target audio; different types of sentiment texts are tested to obtain feedback scores; and target sentiment texts are determined from different types of sentiment texts based on the feedback scores, thereby achieving the purpose of improving the accuracy of sentiment text generation, thereby realizing the technical effect of enhancing the multi-dimensionality of sentiment recognition, and further solving the technical problem of poor accuracy of sentiment text generated based on audio and video data in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of a computer terminal for implementing a method for determining an emotional text according to an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for determining an emotional text according to an embodiment of the present application;

[0022] Figure 3 It is a structural diagram of a device for determining emotional text according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] In order to solve the problems existing in the related art, the present application embodiment provides a method for determining emotional text, which can be run on Figure 1 In the computer terminal shown, the computer terminal is explained below.

[0026] The emotional text determination method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for determining emotional text. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0027] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0028] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for determining the emotional text in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the above-mentioned method for determining the emotional text is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0029] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0031] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0032] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for determining emotional text. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0033] Figure 2 is a flow chart of a method for determining an emotional text according to an embodiment of the present application, such as Figure 2 As shown, the method comprises the following steps:

[0034] Step S202, obtaining target audio and video data to be subjected to sentiment analysis, wherein the video data is video data corresponding to the target audio.

[0035] In the above step S202, the above target audio may include, for example, human voice (such as speeches, conversations, interviews, etc.), and may also include other sound elements (such as background music, environmental noise, etc.), but the main focus is on the human voice part because it usually carries rich emotional information. The video data corresponding to the target audio refers to those video files that are recorded at the same time as the audio or are closely related to the audio. This correspondence is usually based on identifiers such as timestamps or file numbers to ensure the synchronization of audio and video content. Video data not only contains the sound information in the audio, but also provides additional context and emotional clues through visual elements such as images, actions, and expressions.

[0036] Step S204: determining key image information corresponding to key time information in the target audio from the video data.

[0037] In the above step S204, in the target audio, the key time information is a time point with specific meaning or significant features. After the key time information is identified, it needs to be time-marked for synchronization and correspondence in the subsequent video processing process, so that the key image information can be obtained.

[0038] Step S206, determining an expression change score according to the expression change features corresponding to the key image information, wherein the expression change score is used to determine the degree of expression change.

[0039] In the above step S206, corresponding expression change features are extracted from the key image information. These expression change features may include, for example, deformation of facial muscles, shape and position of eyes, curvature of eyebrows, opening and closing state of mouth, etc., which can be automatically extracted by computer vision technology (such as feature point detection, texture analysis, deep learning, etc.). According to different expression change features, the corresponding expression change degree can be obtained, that is, the above expression change score can be obtained.

[0040] Step S208, generating different types of emotional texts based on the expression change scores and the keyword information of the target audio.

[0041] In the above-mentioned step S208, it is necessary to integrate the expression change score and the keyword information of the target audio. The expression change score reflects the degree of change and emotional tendency of the character's expression in the video data, while the audio keyword information provides the key semantic elements in the audio content. The two data sources need to be accurately aligned in time and space to ensure that they can jointly reflect the information at the same time or under the same emotional state. The integrated data is mapped to the emotional space, for example, by establishing an emotional model or an emotional dictionary, which contains different types of emotions (such as happiness, sadness, anger, surprise, etc.) and their corresponding expression features and keywords. By comparing the expression change score and the audio keyword information with the entries in the emotional model, the current emotional type or emotional intensity can be determined.

[0042] Based on the results of sentiment mapping, a strategy for text generation is formulated. For example, it may include determining the type of sentiment text to be generated (such as descriptive text, evaluative text, suggestive text, etc.), the language style of the text (such as formal, informal, humorous, etc.), and the specific content of the text. According to the generated strategy, natural language processing technology (such as template filling, sentence generation, language model, etc.) is used to generate different types of sentiment text. During the generation process, it is necessary to ensure that the text content can accurately reflect the sentiment information conveyed by the expression change score and the audio keyword information, while maintaining the fluency and naturalness of the language. The generated sentiment text is optimized and evaluated. Specifically, the optimization may include modifying grammatical errors in the text, adjusting the text structure, increasing or decreasing the details of the sentiment expression, etc. The evaluation involves checking whether the text meets the expected sentiment type, sentiment intensity, and language style, and whether it matches the expectations of the target audience. If the evaluation results are not satisfactory, you can return to the previous steps for adjustment and improvement.

[0043] Step S210, testing is performed based on different types of emotional texts to obtain feedback scores.

[0044] In the above step S210, different types of emotional texts are presented to the target audience according to the test plan, and their feedback is collected. Feedback may include subjective evaluations (such as emotional response descriptions, text preference ratings, etc.) and objective data (such as changes in physiological indicators, eye tracking data, etc.). Based on the collected feedback, the feedback score is calculated. For example, subjective evaluations and objective data can be converted into quantifiable indicators, such as emotional response intensity ratings, preference ratings, resonance ratings, etc.

[0045] Step S212, determining a target emotional text from different types of emotional texts according to the feedback scores.

[0046] In the above step S212, for example, texts that meet or are close to the expected target can be screened out from all tested emotional texts, and the screened texts are sorted according to the feedback scores, and the best performing texts are preferentially selected to determine the target emotional text.

[0047] In the embodiment of the present application, when a person's emotions fluctuate, the decibel size of the sound and the facial expression will change to a certain extent. Therefore, the target audio is analyzed to determine the interval where the decibel data is greater than the set value, and the audio is feature extracted according to the interval where the decibel data is greater than the set value, and the keyword information that may cause emotional fluctuations is determined. In addition, the time corresponding to the interval where the decibel data is greater than the set value (i.e., the above-mentioned key time information) is extracted, and the video is feature extracted according to the time using the convolutional neural network algorithm to obtain facial expression changes (features), and the degree of facial expression changes is scored to obtain expression change scores. Different types of emotional texts are generated according to the expression change scores and keyword information, and then certain testers are selected to perform secondary tests on the emotional texts, and the optimal emotional texts (i.e., target emotional texts) are selected in combination with the text aesthetic evaluation algorithm, so that the generated text can arouse the emotional resonance of the tester, and the detection results are more accurate. Through the above steps S202 to S212, the purpose of improving the accuracy of emotional text generation is achieved, thereby achieving the technical effect of enhancing the multidimensionality of emotion recognition, and then solving the technical problem of poor accuracy of emotional texts generated according to audio and video data in the related technology. The following is explained.

[0048] In step S202 of the above-mentioned method for determining emotional text, after obtaining the target audio and video data to be subjected to emotional analysis, the method further includes: sampling the target audio to obtain a discrete audio signal; quantizing the discrete audio signal to obtain a quantized audio signal; encoding the quantized audio signal to obtain a digital audio signal; converting the digital audio signal from the time domain to the frequency domain through a fast Fourier transform algorithm, filtering the digital audio signal in the frequency domain, and converting the processed digital audio signal from the frequency domain to the time domain through an inverse Fourier transform algorithm to obtain pure target audio.

[0049] In the embodiment of the present application, sampling is the process of converting a continuous analog audio signal into a discrete digital signal. In this process, the audio signal is divided into a series of equally spaced points in time, and its amplitude value is measured at each point. Through sampling processing, the above-mentioned discrete audio signal is obtained. Quantization processing is the process of mapping the discrete signal amplitude value obtained by sampling to a finite number of discrete values ​​(or "levels"). This is because digital systems cannot directly process numerical values ​​with infinite precision. The quantization process will cause a certain amount of information loss (called quantization error), but it is a necessary step to convert analog signals into digital signals. The number of quantization bits (such as 16 bits, 24 bits) determines the number of quantization levels, which in turn affects the dynamic range and signal-to-noise ratio of the audio signal. Through quantization processing, the above-mentioned quantized audio signal is obtained. Coding is the process of representing the quantized audio signal in digital form. For example, the quantized audio sample sequence is converted into a binary data stream for storage or transmission. Compression technology may also be applied in the encoding process to reduce the size of the data while maintaining the audio quality as much as possible. The Fast Fourier Transform (FFT) algorithm is an efficient algorithm for converting signals from the time domain (i.e., the domain with time as the independent variable) to the frequency domain (i.e., the domain with frequency as the independent variable). In the frequency domain, audio signals are represented as a combination of different frequency components, which makes it easier to process signals within a specific frequency range. Since there will be noise and silent segments in the target audio, the target audio is pre-processed by the Fast Fourier Transform algorithm to make the subsequent analysis of the audio more convenient and quick. Filtering is an important step in signal processing, which is used to remove or attenuate unwanted components (such as noise, interference, etc.) in the audio signal. In the frequency domain, filtering can be achieved by modifying the amplitude or phase of specific frequency components in the signal spectrum. For example, a low-pass filter allows low-frequency signals to pass through while attenuating high-frequency signals, while a high-pass filter does the opposite. After completing the processing in the frequency domain, the signal needs to be converted back to the time domain for further analysis or playback. The Inverse Fourier Transform (IFT) is the inverse process of the FFT, which accepts a signal in the frequency domain and outputs a signal in the time domain. Through IFT, a pure target audio signal can be obtained after filtering, which is more suitable for sentiment analysis.

[0050] In the above steps, after obtaining the pure target audio, the method also includes: analyzing and processing the pure target audio to obtain audio decibel data; determining key decibel data that meets a preset threshold from the audio decibel data; and determining key time information and keyword information corresponding to the key decibel data from the pure target audio.

[0051] In an embodiment of the present application, audio decibel (Decibel, dB) is a unit for measuring the intensity or power of an audio signal. In audio processing, decibel data is often used to evaluate the loudness of an audio signal. Obtaining audio decibel data generally involves calculating the energy or power of the audio signal and converting it into decibel units. For example, this can be achieved by sampling the audio signal, then calculating the energy or power of each sampling point, and converting it into a decibel value. After obtaining the audio decibel data, the decibel data is compared with a preset threshold value, which can be set based on specific application requirements, such as focusing on high decibel values ​​(indicating strong emotional expression) or low decibel values ​​(indicating soft or calm emotional expression) in emotional analysis.

[0052] Since emotional fluctuations can cause changes in voice and expression at the same time, when the emotion is anger or joy, the decibel of the sound will be greater than the decibel of the sound during normal speech, and when the emotion is frustrated and lost, the decibel of the sound will be less than the decibel of the sound during normal speech. Therefore, two different threshold decibels are set to extract the audio. When the decibel data exceeds the high preset threshold or is lower than the low preset threshold, these data points are considered to be key decibel data. Since the audio signal is sampled in time sequence, each decibel data point has a corresponding timestamp. By finding the position of the key decibel data in the audio signal, the exact time when they occur can be determined, that is, the above-mentioned key time information. In addition, while determining the key decibel data, the pure target audio can be transcribed using speech recognition technology to obtain the audio content in text form. Then, these text contents can be analyzed to find keywords or phrases that match the time points of the key decibel data to obtain the above-mentioned keyword information, which helps to understand the emotions and event content in the audio signal more deeply.

[0053] In step S206 of the above-mentioned method for determining emotional text, the expression change characteristics are determined in the following manner: obtaining preset normal image information; comparing the normal image information and the key image information, and determining the changed image in the key image information, wherein the changed image is used to represent the image in the key image information that is inconsistent with the normal image information; extracting the characteristics of the changed image to obtain the expression change characteristics.

[0054] In the embodiment of the present application, a preset "normal" or "reference" image information is required as a reference. This normal image usually represents the facial state of a person without a specific emotion or expression. This state can be neutral, calm, or the facial expression that the individual usually shows in a specific situation. This preset normal image information can be static (such as a photo) or dynamic (such as the average or typical expression in a series of video frames). Compare the normal image information with the key image information to be analyzed. Through the comparison process, it is possible to identify which areas or features in the key image have changed significantly. These changed image parts are called "change images". Change images are parts of the key image that are significantly different from the normal image, and they directly reflect the changes in expression. For example, the raising of eyebrows, the raising or drooping of the corners of the mouth, the widening or squinting of eyes, etc. may all be part of the change image. Specific features are extracted from the change image, and these features are expression change features. These features can be based on the pixel level (such as color, texture changes) or at a higher level (such as shape, contour, position relationship, etc.). Methods for extracting features may include image processing techniques (such as edge detection, corner detection), machine learning algorithms (such as feature extractors, convolutional neural networks), etc. The extracted expression change features will be used for subsequent analysis and recognition tasks, such as emotion classification, expression recognition, etc.

[0055] In step S206 of the above-mentioned method for determining emotional text, an expression change score is determined based on the expression change characteristics corresponding to the key image information, including: determining the normal expression characteristics corresponding to the changed image from the normal image information; comparing the normal expression characteristics and the expression change characteristics to obtain a comparison result; when the comparison result indicates that the degree of overlap between the expression change characteristics and the normal expression characteristics is greater than or equal to a first threshold, determining the expression change score to be a first expression change score; when the comparison result indicates that the degree of overlap between the expression change characteristics and the normal expression characteristics is less than the first threshold and greater than or equal to a second threshold, determining the expression change score to be a second expression change score, wherein the first threshold is greater than the second threshold; when the comparison result indicates that the degree of overlap between the expression change characteristics and the normal expression characteristics is less than the second threshold and greater than or equal to a third threshold, determining the expression change score to be a third expression change score, wherein the second threshold is greater than the third threshold.

[0056] In an embodiment of the present application, normal expression features corresponding to the changed image are extracted from the preset normal image information. These features represent the typical state of an individual's face when there is no specific emotion or expression change. Normal expression features may include information on the position of eyebrows, the degree of eye opening and closing, the shape of the corners of the mouth, and other aspects. The extracted normal expression features are compared with the expression change features obtained from the key image information. The purpose of the comparison is to evaluate the similarity or overlap between the two, that is, how much in common there are between the expression change features and the normal expression features, and how much difference there is. In order to quantify the degree of expression change, multiple thresholds (such as a first threshold, a second threshold, and a third threshold) are set, and the expression change score is determined based on the relationship between the overlap in the comparison result and these thresholds. Specifically:

[0057] When the comparison result indicates that the degree of overlap between the expression change features and the normal expression features is greater than or equal to the first threshold, it means that the expression change is small and the similarity with the normal expression is high. Therefore, a lower expression change score is assigned (indicating "slight change" or "no significant change"), that is, the above-mentioned first expression change score; when the degree of overlap is less than the first threshold but greater than or equal to the second threshold, it indicates that the expression has changed to a certain extent, but still has a certain similarity with the normal expression. Therefore, a moderate expression change score is assigned (indicating "moderate change" or "more significant change"), that is, the above-mentioned second expression change score; when the degree of overlap is less than the second threshold but greater than or equal to the third threshold, it means that the expression has changed significantly and is significantly different from the normal expression. Therefore, a higher expression change score is assigned (indicating "significant change" or "strong change"), that is, the above-mentioned third expression change score.

[0058] In step S208 of the above-mentioned method for determining the emotional text, different types of emotional texts are generated based on the expression change score and the keyword information of the target audio, including: generating a first text outline based on the first expression change score and the first keyword information in the keyword information, wherein the first keyword information is the keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is greater than or equal to the first threshold; filling the first text outline to obtain the first emotional text; generating a second text outline based on the second expression change score and the second keyword information in the keyword information, wherein the second keyword information is the keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is less than the first threshold and greater than or equal to the second threshold; filling the second text outline to obtain the second emotional text; generating a third text outline based on the third expression change score and the third keyword information in the keyword information, wherein the third keyword information is the keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is less than the second threshold and greater than or equal to the third threshold; filling the third text outline to obtain the third emotional text; determining different types of emotional texts based on the first emotional text, the second emotional text and the third emotional text.

[0059] In an embodiment of the present application, when the overlap between the expression change feature and the normal expression feature is greater than or equal to the first threshold, it indicates that the expression change is small, that is, it corresponds to a relatively calm or neutral emotional state. At this time, the first keyword information extracted from the target audio (i.e., the keyword corresponding to this slight expression change) will be used to generate the first text outline. Based on the first text outline, the content is filled by adding appropriate words, phrases and sentences to form a complete first emotional text, which can reflect the emotional color matching the first expression change score, which may be a peaceful, neutral or slightly positive / negative emotion. When the overlap between the expression change feature and the normal expression feature is less than the first threshold but greater than or equal to the second threshold, the expression has changed to a certain extent, that is, it corresponds to a more obvious emotional fluctuation. At this time, the second keyword information extracted from the target audio will be used to generate the second text outline. Similar to the generation of the first emotional text, the second text outline is filled to form a complete second emotional text, which can more strongly express the emotional color matching the second expression change score, such as more obvious joy, sadness, anger, etc. When the overlap between the expression change feature and the normal expression feature is less than the second threshold but greater than or equal to the third threshold, the expression has changed significantly, which corresponds to an extreme emotional state. At this time, the third keyword information extracted from the target audio will be used to generate a third text outline. The third text outline is filled to form a complete third emotional text, which can most strongly express the emotional color that matches the third expression change score, such as extreme joy, despair, anger, etc. After the first, second and third emotional texts are generated, the type of emotional text that needs to be used can be determined according to the specific application scenarios and needs. These text types can represent different emotional intensities or emotional categories, such as calm, pleasure, sadness, anger, etc.

[0060] In step S210 of the above-mentioned method for determining emotional text, testing is performed based on different types of emotional text to obtain feedback scores, including: obtaining first feedback expression change characteristics and first feedback decibel data obtained by the tester based on the first emotional text; obtaining second feedback expression change characteristics and second feedback decibel data obtained by the tester based on the second emotional text; obtaining third feedback expression change characteristics and third feedback decibel data obtained by the tester based on the third emotional text; determining the first feedback score based on the first feedback expression change characteristics and the first feedback decibel data, determining the second feedback score based on the second feedback expression change characteristics and the second feedback decibel data, and determining the third feedback score based on the third feedback expression change characteristics and the third feedback decibel data.

[0061] In the embodiment of the present application, after reading or listening to the first emotional text, the tester will show a corresponding emotional response. Through facial expression capture technology (such as camera recording) and sound analysis technology (such as microphone recording), the first feedback expression change characteristics and the first feedback decibel data of the tester can be obtained. Expression change characteristics may include the raising of eyebrows, the rising or falling of the corners of the mouth, etc., and the decibel data reflects the size of the tester's voice, which can indirectly reflect the intensity of its emotion. Similarly, after reading or listening to the second emotional text, the tester will show expression changes and sound reactions that match the text emotion. Through the same technical means, the second feedback expression change characteristics and the second feedback decibel data can be obtained. For the third emotional text, the tester's feedback will also be recorded, including the third feedback expression change characteristics and the third feedback decibel data. These data will be used to evaluate the tester's reaction to extreme or significant emotional texts. Based on the first feedback expression change characteristics and the first feedback decibel data, a certain evaluation standard or algorithm can be used to determine the first feedback score. This score can reflect the tester's perception and reaction intensity of the emotion conveyed by the first emotional text. The evaluation criteria may include the amplitude, duration, and decibel change of expression changes, etc. For example, the first score is determined based on the degree of overlap between the first feedback expression change feature and the corresponding expression change feature, the second score is determined based on the similarity between the first feedback decibel data and the corresponding key decibel data, and the first feedback score is determined based on the weighted sum of the first score and the second score.

[0062] Similarly, the second feedback score can be determined based on the second feedback expression change feature and the second feedback decibel data. This score will reflect the intensity and accuracy of the tester's emotional response to the second emotional text. For the third emotional text, the third feedback score is determined based on the third feedback expression change feature and the third feedback decibel data.

[0063] In step S212 of the above-mentioned method for determining emotional text, a target emotional text is determined from different types of emotional texts based on feedback scores, including: respectively evaluating the first emotional text, the second emotional text and the third emotional text to obtain a first evaluation score, a second evaluation score and a third evaluation score; comparing the first evaluation score and the first feedback score, the second evaluation score and the second feedback score, and the third evaluation score and the third feedback score to obtain a first text quality comprehensive score of the first emotional text, a second text quality comprehensive score corresponding to the second emotional text, and a third text quality comprehensive score corresponding to the third emotional text; and determining the emotional text corresponding to the highest score among the first text quality comprehensive score, the second text quality comprehensive score and the third text quality comprehensive score as the target emotional text.

[0064] In the embodiment of the present application, the first emotional text, the second emotional text and the third emotional text are evaluated respectively. For example, the text aesthetics evaluation algorithm can be used to evaluate the accuracy of the text content, the richness of emotional expression, the degree of fit with the target emotion, complexity, clarity, emotional tendency and other aspects. The evaluation results are quantified into a first evaluation score, a second evaluation score and a third evaluation score, which reflect the emotional communication effect that the text should achieve in theory. The above evaluation scores are compared with the actual feedback scores. Specifically, the first evaluation score is compared with the first feedback score, the second evaluation score is compared with the second feedback score, and the third evaluation score is compared with the third feedback score. This comparison is intended to reveal the gap between the theoretical performance of the text and the actual communication effect. In order to more comprehensively reflect the quality of the text, the text quality comprehensive score of each emotional text can be calculated based on the comparison results of the evaluation score and the feedback score. The first text quality comprehensive score, the second text quality comprehensive score and the third text quality comprehensive score are compared to find the highest score. The emotional text corresponding to this highest score is determined as the target emotional text. The target emotional text is the text that can most accurately convey the target emotion, resonate with the tester and perform best in practical applications among all candidate texts.

[0065] The method for determining the emotional text provided in the embodiment of the present application analyzes the audio to determine the expression change characteristics and audio decibel data, extracts the keyword information that causes emotional fluctuations based on the expression change characteristics and audio decibel data, generates the emotional text based on the keyword information, performs secondary emotional analysis on the text, and selects the optimal emotional text, i.e., the above-mentioned target emotional text, in combination with the text aesthetic evaluation algorithm. The text after secondary analysis is more comprehensive and the analysis of the emotional fluctuations of the tester is more accurate.

[0066] Figure 3 is a structural diagram of a device for determining an emotional text according to an embodiment of the present application, such as Figure 3 As shown, the device comprises:

[0067] An acquisition module 30 is used to acquire target audio and video data to be subjected to sentiment analysis, wherein the video data is video data corresponding to the target audio;

[0068] A first determination module 32, configured to determine key image information corresponding to key time information in the target audio from the video data;

[0069] A second determination module 34 is used to determine an expression change score according to the expression change feature corresponding to the key image information, wherein the expression change score is used to determine the degree of expression change;

[0070] A generating module 36, for generating different types of emotional texts according to the expression change scores and the keyword information of the target audio;

[0071] A testing module 38 is used to test different types of emotional texts and obtain feedback scores;

[0072] The third determination module 40 is used to determine the target emotional text from different types of emotional texts according to the feedback scores.

[0073] In the acquisition module in the above-mentioned emotional text determination device, the acquisition module is also used to sample the target audio to obtain a discrete audio signal; quantize the discrete audio signal to obtain a quantized audio signal; encode the quantized audio signal to obtain a digital audio signal; convert the digital audio signal from the time domain to the frequency domain through the fast Fourier transform algorithm, filter the digital audio signal in the frequency domain, and then convert the processed digital audio signal from the frequency domain to the time domain through the inverse Fourier transform algorithm to obtain pure target audio.

[0074] In the acquisition module in the above-mentioned emotional text determination device, the acquisition module is also used to analyze and process the pure target audio to obtain audio decibel data; determine the key decibel data that meets the preset threshold from the audio decibel data; determine the key time information and keyword information corresponding to the key decibel data from the pure target audio.

[0075] In the second determination module in the above-mentioned emotional text determination device, the second determination module is also used to determine the expression change characteristics. Specifically, the expression change characteristics are determined in the following manner: obtaining preset normal image information; comparing the normal image information and the key image information, and determining the changed image in the key image information, wherein the changed image is used to represent the image in the key image information that is inconsistent with the normal image information; extracting the characteristics of the changed image to obtain the expression change characteristics.

[0076] In the second determination module in the above-mentioned emotional text determination device, the second determination module is also used to determine the normal expression features corresponding to the changed image from the normal image information; compare the normal expression features and the expression change features to obtain a comparison result; when the comparison result indicates that the overlap between the expression change features and the normal expression features is greater than or equal to the first threshold, determine the expression change score as the first expression change score; when the comparison result indicates that the overlap between the expression change features and the normal expression features is less than the first threshold and greater than or equal to the second threshold, determine the expression change score as the second expression change score, wherein the first threshold is greater than the second threshold; when the comparison result indicates that the overlap between the expression change features and the normal expression features is less than the second threshold and greater than or equal to the third threshold, determine the expression change score as the third expression change score, wherein the second threshold is greater than the third threshold.

[0077] In the generation module in the above-mentioned device for determining emotional text, the generation module is also used to generate a first text outline based on the first expression change score and the first keyword information in the keyword information, wherein the first keyword information is the keyword information corresponding to the case where the overlap between the expression change feature and the normal expression feature is greater than or equal to the first threshold; fill in the first text outline to obtain the first emotional text; generate a second text outline based on the second expression change score and the second keyword information in the keyword information, wherein the second keyword information is the keyword information corresponding to the case where the overlap between the expression change feature and the normal expression feature is less than the first threshold and greater than or equal to the second threshold; fill in the second text outline to obtain the second emotional text; generate a third text outline based on the third expression change score and the third keyword information in the keyword information, wherein the third keyword information is the keyword information corresponding to the case where the overlap between the expression change feature and the normal expression feature is less than the second threshold and greater than or equal to the third threshold; fill in the third text outline to obtain the third emotional text; determine different types of emotional texts based on the first emotional text, the second emotional text and the third emotional text.

[0078] In the test module in the above-mentioned emotional text determination device, the test module is also used to obtain the first feedback expression change characteristics and first feedback decibel data obtained by the tester based on the first emotional text; obtain the second feedback expression change characteristics and second feedback decibel data obtained by the tester based on the second emotional text; obtain the third feedback expression change characteristics and third feedback decibel data obtained by the tester based on the third emotional text; determine the first feedback score based on the first feedback expression change characteristics and the first feedback decibel data, determine the second feedback score based on the second feedback expression change characteristics and the second feedback decibel data, and determine the third feedback score based on the third feedback expression change characteristics and the third feedback decibel data.

[0079] In the third determination module in the above-mentioned emotional text determination device, the third determination module is also used to evaluate the first emotional text, the second emotional text and the third emotional text respectively to obtain a first evaluation score, a second evaluation score and a third evaluation score; compare the first evaluation score and the first feedback score, the second evaluation score and the second feedback score, and the third evaluation score and the third feedback score to obtain a first text quality comprehensive score for the first emotional text, a second text quality comprehensive score corresponding to the second emotional text, and a third text quality comprehensive score corresponding to the third emotional text; and determine the emotional text corresponding to the highest score among the first text quality comprehensive score, the second text quality comprehensive score and the third text quality comprehensive score as the target emotional text.

[0080] It should be noted that Figure 3 The device for determining the emotional text is used to perform Figure 2 The method for determining the emotional text shown in the figure, therefore the relevant explanations in the above-mentioned method for determining the emotional text are also applicable to the device for determining the emotional text, and will not be repeated here.

[0081] An embodiment of the present application also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute the method for determining the above-mentioned emotional text.

[0082] An embodiment of the present application also provides a non-volatile storage medium, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method for determining the emotional text by running the computer program.

[0083] An embodiment of the present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the method for determining emotional text in each embodiment of the present application.

[0084] The embodiments of the present application also provide a computer program, which, when executed by a processor, implements the steps of the method for determining emotional text in each embodiment of the present application.

[0085] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0086] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0087] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0090] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0091] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining emotional text, characterized in that: include: Acquire target audio and video data to be subjected to sentiment analysis, wherein the video data is video data corresponding to the target audio; determining, from the video data, key image information corresponding to key time information in the target audio; Determining an expression change score according to the expression change feature corresponding to the key image information, wherein the expression change score is used to determine the degree of expression change; Generate different types of emotional texts according to the expression change score and the keyword information of the target audio; Conducting tests based on the different types of emotional texts to obtain feedback scores; A target emotional text is determined from the different types of emotional texts according to the feedback scores.

2. The method according to claim 1, characterized in that After obtaining the target audio and video data to be subjected to sentiment analysis, the method further includes: Sampling the target audio to obtain a discrete audio signal; Quantizing the discrete audio signal to obtain a quantized audio signal; Encoding the quantized audio signal to obtain a digital audio signal; The digital audio signal is converted from the time domain to the frequency domain by a fast Fourier transform algorithm. After filtering the digital audio signal in the frequency domain, the processed digital audio signal is converted from the frequency domain to the time domain by an inverse Fourier transform algorithm to obtain pure target audio.

3. The method according to claim 2, characterized in that After obtaining the pure target audio, the method further includes: Analyzing and processing the pure target audio to obtain audio decibel data; Determining key decibel data that meets a preset threshold from the audio decibel data; Key time information and keyword information corresponding to the key decibel data are determined from the clean target audio.

4. The method according to claim 1, characterized in that: The expression change feature is determined by: Obtain preset normal image information; Comparing the normal image information with the key image information, and determining a changed image in the key image information, wherein the changed image is used to represent an image in the key image information that is inconsistent with the normal image information; The features of the changed image are extracted to obtain the expression change features.

5. The method according to claim 4, characterized in that Determining an expression change score according to the expression change features corresponding to the key image information includes: Determining a normal expression feature corresponding to the changed image from the normal image information; Comparing the normal expression feature with the expression change feature to obtain a comparison result; In a case where the comparison result indicates that the degree of overlap between the expression change feature and the normal expression feature is greater than or equal to a first threshold, determining the expression change score to be a first expression change score; When the comparison result indicates that the overlap between the expression change feature and the normal expression feature is less than the first threshold and greater than or equal to a second threshold, determining the expression change score to be a second expression change score, wherein the first threshold is greater than the second threshold; When the comparison result indicates that the degree of overlap between the expression change feature and the normal expression feature is less than the second threshold and greater than or equal to a third threshold, the expression change score is determined to be a third expression change score, wherein the second threshold is greater than the third threshold.

6. The method according to claim 5, characterized in that Different types of emotional texts are generated according to the expression change score and the keyword information of the target audio, including: Generate a first text outline according to the first expression change score and first keyword information in the keyword information, wherein the first keyword information is keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is greater than or equal to a first threshold; Filling the first text outline to obtain a first emotional text; generating a second text outline according to the second expression change score and second keyword information in the keyword information, wherein the second keyword information is keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is less than the first threshold and greater than or equal to the second threshold; Filling the second text outline to obtain a second emotional text; generating a third text outline according to the third expression change score and the third keyword information in the keyword information, wherein the third keyword information is the keyword information corresponding to when the overlap between the expression change feature and the normal expression feature is less than the second threshold and greater than or equal to the third threshold; Filling the third text outline to obtain a third emotional text; Different types of emotional texts are determined according to the first emotional text, the second emotional text, and the third emotional text.

7. The method according to claim 6, characterized in that The test is conducted based on the different types of emotional texts to obtain feedback scores, including: Obtaining first feedback expression change characteristics and first feedback decibel data obtained by the tester based on the first emotional text; Acquire the second feedback expression change feature and the second feedback decibel data obtained by the tester according to the second emotional text; Obtaining a third feedback expression change feature and a third feedback decibel data obtained by the tester according to the third emotional text; A first feedback score is determined based on the first feedback expression change characteristics and the first feedback decibel data, a second feedback score is determined based on the second feedback expression change characteristics and the second feedback decibel data, and a third feedback score is determined based on the third feedback expression change characteristics and the third feedback decibel data.

8. The method according to claim 7, characterized in that Determining a target emotional text from the different types of emotional texts according to the feedback scores includes: Evaluate the first emotion text, the second emotion text, and the third emotion text respectively to obtain a first evaluation score, a second evaluation score, and a third evaluation score; Compare the first evaluation score and the first feedback score, the second evaluation score and the second feedback score, and the third evaluation score and the third feedback score to obtain a first text quality comprehensive score of the first emotional text, a second text quality comprehensive score corresponding to the second emotional text, and a third text quality comprehensive score corresponding to the third emotional text; The emotional text corresponding to the highest score among the first text quality comprehensive score, the second text quality comprehensive score and the third text quality comprehensive score is determined as the target emotional text.

9. A device for determining emotional text, characterized in that: include: An acquisition module, used to acquire target audio and video data to be subjected to sentiment analysis, wherein the video data is video data corresponding to the target audio; A first determination module, configured to determine key image information corresponding to key time information in the target audio from the video data; A second determination module is used to determine an expression change score according to the expression change feature corresponding to the key image information, wherein the expression change score is used to determine the degree of expression change; A generation module, used to generate different types of emotional texts according to the expression change score and the keyword information of the target audio; A testing module, used to perform tests based on the different types of emotional texts to obtain feedback scores; The third determination module is used to determine the target emotional text from the different types of emotional texts according to the feedback score.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and is used to execute the method for determining the emotional text as described in any one of claims 1 to 8.

11. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the method for determining the emotional text described in any one of claims 1 to 8 by running the computer program.

12. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method for determining the emotional text described in any one of claims 1 to 8 is implemented.

Images