A method, device, electronic device and storage medium for identifying emotion intensity

Abstract

The present application provides a method, device, electronic device and storage medium for identifying emotion intensity, wherein the identification method includes: processing the surveillance video corresponding to the target object, selecting the target video corresponding to the target object; inputting each frame of the target image in the target video into a pre-trained emotion classification model, and obtaining the emotion category of the frame of the target image output by the emotion classification model; for multiple frames of first target images corresponding to the first emotion category, determining the baseline speech intensity according to the speech intensity of the audio signals respectively corresponding to the multiple frames of the first target images; for multiple frames of second target images corresponding to each second emotion category, determining the change information of the emotion intensity corresponding to the target object under the second emotion category according to the audio signals respectively corresponding to the multiple frames of the second target images and the baseline speech intensity. The present application improves the accuracy of emotion intensity recognition.

Description

Technical Field

[0001] The present application relates to the technical field of multimedia information emotion recognition, and specifically, to an emotion intensity recognition method, device, electronic device and storage medium. Background Art

[0002] At this stage, with the rapid development of Internet technology, various industries have gradually realized online and intelligent business processing, such as online processing of insurance business.

[0003] Usually, when users handle online business, it is necessary to collect the user's facial image and archive the collected facial image in the business system, or to verify the user's identity based on the collected facial image. In the process of user interaction with the business system, it is often necessary to collect the user's voice information and convert the voice information into interactive instructions for the business system to realize the final processing of the business.

[0004] However, when users conduct business online, their facial images and voice information often contain the user's emotional tendencies. Obtaining the user's emotional tendencies can improve the efficiency of business processing. Therefore, emotion recognition in audio and video is a technical problem that needs to be solved urgently at this stage. Summary of the invention

[0005] In view of this, the purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for identifying emotion intensity, which can identify the emotion category and emotion intensity of the target object in the target video based on each frame of the target image in the target video and the speech intensity of the audio signal corresponding to each frame of the target image, thereby improving the accuracy of emotion intensity recognition.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying emotion intensity, the method comprising:

[0007] Processing the surveillance video corresponding to the target object, and selecting the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal;

[0008] Input each frame of the target image in the target video into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model;

[0009] For a plurality of frames of first target images corresponding to a first emotion category, determining a reference speech intensity according to speech intensities of audio signals respectively corresponding to the plurality of frames of first target images;

[0010] For multiple frames of second target images corresponding to each second emotion category, the change information of the emotion intensity corresponding to the target object under the second emotion category is determined according to the audio signals corresponding to the multiple frames of second target images and the reference speech intensity.

[0011] In a possible implementation, for a plurality of frames of second target images corresponding to each second emotion category, determining, according to the audio signals respectively corresponding to the plurality of frames of second target images and the reference speech intensity, change information of the emotion intensity corresponding to the target object under the second emotion category includes:

[0012] Determine the emotion intensity corresponding to the target object in the second target image frame under the second emotion category according to the speech intensity of the audio signal corresponding to each frame of the second target image under each second emotion category and the reference speech intensity;

[0013] Based on the emotion intensity corresponding to the target object in each frame of the second target image under the second emotion category, change information of the emotion intensity corresponding to the target object under the second emotion category is determined.

[0014] In a possible implementation, the identification method further includes:

[0015] According to the emotion intensity corresponding to the target object in each frame of the second target image, the emotion intensity corresponding to the target object in other frames of the second target image adjacent to the frame of the second target image, the first weight corresponding to the frame of the second target image, and the second weights corresponding to the other frames of the second target image, the corresponding smoothed emotion intensity in each frame of the second target image is generated.

[0016] In a possible implementation, the processing of the surveillance video corresponding to the target object and selecting the target video corresponding to the target object includes:

[0017] According to the short-time energy and the number of zero crossings of the audio signal corresponding to each frame of the surveillance video, intercepting the surveillance video corresponding to the voice from the surveillance video;

[0018] Input each frame of the surveillance video into a pre-trained face detection model to obtain a detection result output by the face detection model, and based on the detection result, intercept a surveillance video including a face from the surveillance video;

[0019] The target video is generated based on the surveillance video corresponding to the voice and the surveillance video including the face.

[0020] In a possible implementation, the speech intensity of the audio signal corresponding to each frame of the target image is determined in the following manner:

[0021] The speech intensity of the audio signal corresponding to each frame of the target image is determined according to the initial speech intensity of the audio signal corresponding to each frame of the target image and the initial speech intensity of the audio signals corresponding to multiple frames of the target image adjacent to the frame of the target image.

[0022] In a possible implementation, the identification method further includes:

[0023] Generate an emotion label that matches the emotion intensity and the emotion category according to the emotion intensity and the emotion category corresponding to the target object in each frame of the second target image;

[0024] The emotion label is marked at the second target image of the frame.

[0025] In a possible implementation, the identification method further includes:

[0026] According to the change information of the emotion intensity corresponding to the target object in each second emotion category, business prompt information matching the change information of the emotion intensity is generated and displayed.

[0027] In a second aspect, an embodiment of the present application provides a device for identifying emotion intensity, the device comprising:

[0028] A selection module is used to process the surveillance video corresponding to the target object and select the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal;

[0029] An input module, used to input each frame of the target image in the target video into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model;

[0030] A first determination module is used to determine a reference speech intensity according to speech intensities of audio signals respectively corresponding to the multiple frames of first target images corresponding to the first emotion category;

[0031] The second determination module is used to determine the change information of the emotion intensity corresponding to the target object under the second emotion category according to the audio signals corresponding to the multiple frames of second target images and the reference speech intensity for each second emotion category.

[0032] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to perform the steps of the method for identifying emotion intensity described in any one of the first aspects.

[0033] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for identifying emotion intensity described in any one of the first aspects are executed.

[0034] The emotion intensity recognition method, device, electronic device and storage medium provided in the embodiment of the present application process the surveillance video corresponding to the target object and select the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal; each frame of the target image in the target video is input into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model; for multiple frames of the first target image corresponding to the first emotion category, the reference speech intensity is determined according to the speech intensity of the audio signals respectively corresponding to the multiple frames of the first target image; for multiple frames of the second target image corresponding to each second emotion category, the change information of the emotion intensity corresponding to the target object under the second emotion category is determined according to the audio signals respectively corresponding to the multiple frames of the second target image and the reference speech intensity. The embodiment of the present application improves the accuracy of emotion intensity recognition.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A flowchart of a method for identifying emotion intensity provided by an embodiment of the present application is shown;

[0038] Figure 2 A flowchart showing another method for identifying emotion intensity provided in an embodiment of the present application is shown;

[0039] Figure 3 A flowchart showing another method for identifying emotion intensity provided in an embodiment of the present application is shown;

[0040] Figure 4 A flowchart showing another method for identifying emotion intensity provided in an embodiment of the present application is shown;

[0041] Figure 5 A schematic diagram of the structure of a device for identifying emotion intensity provided in an embodiment of the present application is shown;

[0042] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0044] At present, with the rapid development of Internet technology, various industries are gradually realizing online and intelligent business processing, such as online insurance business processing. Usually, when users handle online business, it is necessary to collect the user's facial image and archive the collected facial image in the business system, or verify the user's identity based on the collected facial image. In the process of user interaction with the business system, it is often necessary to collect the user's voice information and convert the voice information into interactive instructions for the business system to realize the final processing of the business.

[0045] However, when users conduct business online, their facial images and voice information often contain the user's emotional tendencies. Obtaining the user's emotional tendencies can improve the efficiency of business processing. Therefore, emotion recognition in audio and video is a technical problem that needs to be solved urgently at this stage.

[0046] Based on the above problems, the embodiments of the present application provide a method, device, electronic device and storage medium for identifying emotion intensity, which processes the surveillance video corresponding to the target object and selects the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal; each frame of the target image in the target video is input into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model; for multiple frames of the first target image corresponding to the first emotion category, a reference speech intensity is determined according to the speech intensity of the audio signals respectively corresponding to the multiple frames of the first target image; for multiple frames of the second target image corresponding to each second emotion category, the change information of the emotion intensity corresponding to the target object under the second emotion category is determined according to the audio signals respectively corresponding to the multiple frames of the second target image and the reference speech intensity. The embodiments of the present application improve the accuracy of emotion intensity recognition.

[0047] The defects existing in the above solutions are the results obtained by the inventor after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by this application for the above problems below should be the contributions made by the inventor to this application during the application process.

[0048] The technical solutions in the present application will be clearly and completely described below in conjunction with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0050] To facilitate understanding of this embodiment, a method for identifying emotion intensity disclosed in an embodiment of the present application is first introduced in detail.

[0051] See also Figure 1 As shown, Figure 1 A flowchart of a method for identifying emotion intensity provided in an embodiment of the present application, the method comprising:

[0052] S101. Process the surveillance video corresponding to the target object and select the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal.

[0053] In an embodiment of the present application, the surveillance video is composed of multiple frames of images, and each frame of the image corresponds to an audio signal. Here, each frame of the surveillance video may include the target object, or may not include the target object. The audio signal corresponding to each frame of the surveillance video may be voice, silence, or noise. In the present application, the emotion category and emotion intensity of the target object are identified based on the target image including the target object and the voice information of the target object. Therefore, it is necessary to preprocess the surveillance video corresponding to the target object and select the target video in which the target object is included in the image and the audio signal corresponding to the image is voice.

[0054] Optionally, the monitoring video is acquired by using a camera, a smart phone or other acquisition device to acquire the monitoring video of the target object during the process of the target object handling business. Here, the target object is a specific user.

[0055] S102: input each frame of the target image in the target video into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model.

[0056] In an embodiment of the present application, the emotion category of the target object is determined according to the facial expression of the target object, a pre-trained emotion classification model is used to determine the emotion category of the target object based on the facial expression of the target object, and each frame of the target image in the target video is input into the pre-trained emotion classification model frame by frame to obtain the emotion category output by the above emotion classification model, which is the emotion category of the target object in the current frame target image, wherein the emotion categories include anger, fear, disgust, happiness, sadness, surprise and neutrality, based on this, the emotion category of the target object in each frame of the target video is obtained.

[0057] Optionally, the emotion classification model is a facial expression recognition model (Self-Cure Network, SCN), which is used to determine the category to which the facial expression in the target image belongs, that is, the emotion category.

[0058] S103 . For a plurality of frames of first target images corresponding to a first emotion category, determine a reference speech intensity according to speech intensities of audio signals respectively corresponding to the plurality of frames of first target images.

[0059] In an embodiment of the present application, the emotion intensity of the target object is determined based on the speech intensity of the target object. However, the speech intensity of different target objects at the same emotion intensity is greatly different. Therefore, it is necessary to determine a baseline speech intensity of each target object. The emotion category includes a first emotion category and a second emotion category, wherein the first emotion category is an emotion category with no obvious emotion fluctuations, such as neutrality, and the second emotion category is an emotion category with relatively obvious emotion fluctuations, such as anger, fear, disgust, happiness, sadness, and surprise. A target image with an emotion category of the first emotion category is determined as the first target image, and a target image with an emotion category of the second emotion category is determined as the second target image. The speech intensity of the audio signal of each frame of the first target image including the same target object is obtained, and the average value of the above multiple speech intensities is determined as the baseline speech intensity of the target object.

[0060] Optionally, in order to simplify calculation, part of the first target images are selected from multiple frames of first target images, and the reference speech intensity is determined based on the speech intensity of the audio signals of the part of the first target images.

[0061] S104. For a plurality of frames of second target images corresponding to each second emotion category, determine the change information of the emotion intensity corresponding to the target object under the second emotion category according to the audio signals respectively corresponding to the plurality of frames of second target images and the reference speech intensity.

[0062] In an embodiment of the present application, each frame of the second target image in the target video corresponds to an emotion category and an emotion intensity. For each frame of the second target image, the emotion intensity corresponding to the target object in each frame of the second target image is determined according to the emotion category to which the second target image belongs, the speech intensity of the audio signal corresponding to the second target image, and the baseline speech intensity of the target object in the target video, and multiple frames of second target images corresponding to each second emotion category are obtained. According to the emotion intensity corresponding to each frame of the second target image under the second emotion category, the change information of the emotion intensity corresponding to the target object under the second emotion category is determined.

[0063] Optionally, an emotion intensity variation curve corresponding to each second emotion category is generated according to the time point and emotion intensity corresponding to each frame of the second target image under each second emotion category.

[0064] The method for identifying emotion intensity provided in the embodiment of the present application determines the emotion category of the target object in each frame of the target video based on the expression of the target object in each frame of the target image of the target video, and further determines the emotion intensity of the target user in each frame of the target video based on the voice intensity reflected in the audio signal of the target object in each frame of the target image. This method for identifying emotion category and emotion intensity has high accuracy and efficiency.

[0065] For further information, see Figure 2 As shown, in the method for identifying emotion intensity provided in the embodiment of the present application, for multiple frames of second target images corresponding to each second emotion category, according to the audio signals respectively corresponding to the multiple frames of second target images, and the reference speech intensity, determining the change information of the emotion intensity corresponding to the target object under the second emotion category, includes:

[0066] S201, determining the emotion intensity corresponding to the target object in the frame of the second target image under each second emotion category according to the speech intensity of the audio signal corresponding to each frame of the second target image under the second emotion category and the reference speech intensity.

[0067] In an embodiment of the present application, the second emotion category is further divided into two categories, a proportional emotion category and an inversely proportional emotion category. In the proportional emotion category, the stronger the voice intensity of the target object, the stronger its emotion intensity, such as anger, happiness, and surprise. In the inversely proportional emotion category, the stronger the voice intensity of the target object, the weaker its emotion intensity, such as fear, disgust, and sadness. For each of the multiple frames of second target images under the second emotion category, it is first determined whether the corresponding second emotion category is a proportional emotion category or an inversely proportional emotion category, and then the emotion intensity judgment rule is determined. Then, according to the speech intensity of the audio signal of each frame of the second target image under the second emotion category, the baseline speech intensity, and the determined emotion intensity judgment rule, the emotion intensity corresponding to the target object in each frame of the second target image under the second emotion category is determined.

[0068] Optionally, the emotion intensity judgment rule for the proportional emotion category is: Pi = a×Vi / (b+Vn); the emotion intensity judgment rule for the inverse emotion category is: Pi = c×Vn / (d+Vi), where Pi is the emotion intensity, Vi is the voice intensity, Vn is the benchmark voice intensity, and a, b, c, and d are empirical constants that can be set according to the actual needs of the user.

[0069] S202: Determine change information of the emotion intensity corresponding to the target object in the second emotion category based on the emotion intensity corresponding to the target object in each frame of the second target image in the second emotion category.

[0070] In an embodiment of the present application, multiple frames of second target images corresponding to each second emotion category are obtained, and based on the emotion intensity corresponding to each frame of the second target image under the second emotion category, change information of the emotion intensity corresponding to the target object under the second emotion category is determined.

[0071] Optionally, an emotion intensity variation curve corresponding to each second emotion category is generated according to the time point and emotion intensity corresponding to each frame of the second target image under each second emotion category.

[0072] Furthermore, in the method for identifying emotion intensity provided in the embodiment of the present application, the method for identifying emotion intensity further includes:

[0073] According to the emotion intensity corresponding to the target object in each frame of the second target image, the emotion intensity corresponding to the target object in other frames of the second target image adjacent to the frame of the second target image, the first weight corresponding to the frame of the second target image, and the second weights corresponding to the other frames of the second target image, the corresponding smoothed emotion intensity in each frame of the second target image is generated.

[0074] In an embodiment of the present application, taking into account the continuity of changes in emotion intensity, in order to eliminate the impact of accidental changes in speech intensity in the audio signal, the present application performs Gaussian filtering and smoothing on the emotion intensity corresponding to each frame of the second target image to obtain the smoothed emotion intensity, and based on the smoothed emotion intensity, determines the change information of the emotion intensity corresponding to the target object under each second emotion category.

[0075] When smoothing the emotion intensity corresponding to the target object in each frame of the second target image, the first product of the emotion intensity corresponding to the frame of the second target image and the first weight is calculated, and the second product of the emotion intensity corresponding to the target object in other frames of the second target image adjacent to the frame of the second target image (for example, the first two frames of the second target image and the second two frames of the second target image) and the second weight is calculated, and the sum of the first product and the second product is determined as the emotion intensity corresponding to the target object in the second target image after smoothing. Here, the first weight and the second weight are determined based on a one-dimensional Gaussian filter.

[0076] For further information, see Figure 3 As shown, in the emotion intensity recognition method provided in the embodiment of the present application, the monitoring video corresponding to the target object is processed, and the target video corresponding to the target object is selected, including:

[0077] S301. According to the short-time energy and the number of zero crossings of the audio signal corresponding to each frame image in the surveillance video, intercept the surveillance video corresponding to the voice from the surveillance video.

[0078] In the embodiment of the present application, endpoint detection is performed on the surveillance video based on voice activity detection VAD (Voice Activity Detection). Endpoint detection is to accurately locate the starting point and the ending point of the speech from the noisy speech, remove the silent part, remove the noisy part, distinguish between the speech and non-speech areas, find the truly effective part of a speech, and select multiple time periods T = (T1...Tn) that can be marked in the entire surveillance video.

[0079] Specifically, multiple time periods T = (T1 ... Tn) are marked by short time energy (STE) and zero crosscounter (ZCC). STE is the energy of a frame of speech signal, audio signal is a time domain signal, and ZCC is the number of times a frame of speech time domain signal crosses 0 (time axis). When the signal-to-noise ratio is not very low, the STE of the speech segment is relatively large, while the ZCC is relatively small; and the STE of the non-speech segment is relatively small, but the ZCC is relatively large, because most of the speech signal energy is contained in the low frequency band, while the noise signal usually has less energy and contains information in the higher frequency band. Therefore, the short-time energy and zero-crossing times of the audio signal corresponding to each frame of the surveillance video are measured. If the short-time energy is greater than a preset first threshold value, and the zero-crossing times are less than a preset second threshold value, the frame of the image is determined as a speech frame, and then a continuous speech segment is obtained, and the start and end points of multiple speech segments are marked, that is, multiple time periods T = (T1 ... Tn) are marked, and the surveillance video corresponding to the speech is intercepted from the surveillance video.

[0080] S302: input each frame image of the surveillance video into a pre-trained face detection model to obtain a detection result output by the face detection model, and based on the detection result, extract a surveillance video including a face from the surveillance video.

[0081] In the embodiment of the present application, the facial expression classification result is used as the emotion classification, so the face must first be detected in the surveillance video, and then the facial expression is identified and classified. Each frame of the surveillance video is input into a pre-trained face detection model to obtain the detection result output by the face detection model, that is, whether there is a face in the video and the position of the face. Through face detection of the surveillance video, the time period when the face is detected is selected as the effective time period S=(S1...Sm) in the surveillance video, that is, the surveillance video including the face is intercepted from the surveillance video.

[0082] S303: Generate the target video based on the surveillance video corresponding to the voice and the surveillance video including the face.

[0083] In the embodiment of the present application, it is necessary to select a surveillance video clip with both human face and voice as the target video. Based on the effective time period T = (T1...Tn) obtained by voice VAD detection and the effective time period S = (S1...Sm) obtained by face detection, the overlapping time period of the two is taken as the final effective time period F (F = T∩S) to obtain the target video.

[0084] Furthermore, in the emotion intensity recognition method provided in the embodiment of the present application, the speech intensity of the audio signal corresponding to each frame of the target image is determined in the following manner:

[0085] The speech intensity of the audio signal corresponding to each frame of the target image is determined according to the initial speech intensity of the audio signal corresponding to each frame of the target image and the initial speech intensity of the audio signals corresponding to multiple frames of the target image adjacent to the frame of the target image.

[0086] In an embodiment of the present application, different speaking contents have a certain influence on the speech intensity. In order to reduce the influence of different texts in the speech on the speech intensity, a median filter is used as a smoothing function to perform smoothing preprocessing on the speech intensity. Among them, the median filter is a nonlinear smoothing technology. The present application adopts a one-dimensional median filter. Specifically, the initial speech intensity of the audio signal corresponding to each frame of the target image and the initial speech intensity of the audio signal corresponding to the multiple frames of target images adjacent to the frame of the target image are obtained, wherein the multiple frames of target images adjacent to the frame of the target image are even-numbered frame target images, and the multiple initial speech intensities obtained are sorted, and the middle value of the above multiple initial speech intensities is used as the output value of the median filter, that is, the speech intensity of the audio signal corresponding to the frame of the target image.

[0087] For further information, see Figure 4 As shown, in the method for identifying emotion intensity provided in the embodiment of the present application, the method for identifying emotion intensity further includes:

[0088] S401. Generate an emotion label that matches the emotion intensity and the emotion category according to the emotion intensity and the emotion category corresponding to the target object in each frame of the second target image.

[0089] S402: Mark the emotion label at the second target image of the frame.

[0090] Combining step 401 and step 402, the emotion label includes an emotion category label and an emotion intensity label. In the present application, there is an emotion category label corresponding to each emotion category, and an emotion intensity label corresponding to each emotion intensity. After obtaining the corresponding emotion intensity and emotion category in each frame of the second target image, the emotion intensity label and emotion category label that respectively match the corresponding emotion intensity and emotion category in the second target image of the frame are selected, and the selected labels are marked on the corresponding second target image.

[0091] Furthermore, in the method for identifying emotion intensity provided in the embodiment of the present application, the method for identifying emotion intensity further includes:

[0092] According to the change information of the emotion intensity corresponding to the target object in each second emotion category, business prompt information matching the change information of the emotion intensity is generated and displayed.

[0093] In an embodiment of the present application, a surveillance video of the target object is obtained during the process of the target object handling related business. After determining the change information of the target object's emotion intensity, the target object can be guided to handle other business according to the change of the target object's emotion category and emotion intensity. Specifically, business prompt information matching the change information of the emotion intensity is generated, and the generated business prompt information is displayed. For example, when it is determined that the target object's happiness level is gradually increasing, other business handling prompt information related to the business currently handled by the target object is generated; or, when it is determined that the target object's anger level is gradually increasing, preferential prompt information related to the business currently handled by the target object is generated, so as to improve the success rate of the target object in handling related business.

[0094] Based on the same inventive concept, an emotion intensity recognition device corresponding to the emotion intensity recognition method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned emotion intensity recognition method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0095] See also Figure 5 As shown, Figure 5 A schematic diagram of a structure of an emotion intensity recognition device provided in an embodiment of the present application, the recognition device comprising:

[0096] A selection module 501 is used to process the surveillance video corresponding to the target object and select the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal;

[0097] An input module 502 is used to input each frame of the target image in the target video into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model;

[0098] A first determination module 503 is used to determine a reference speech intensity for a plurality of frames of first target images corresponding to a first emotion category according to speech intensities of audio signals respectively corresponding to the plurality of frames of first target images;

[0099] The second determination module 504 is used to determine the change information of the emotion intensity corresponding to the target object under the second emotion category according to the audio signals corresponding to the multiple frames of second target images and the reference speech intensity for each second emotion category.

[0100] In a possible implementation manner, the second determination module 504, when determining, for a plurality of frames of second target images corresponding to each second emotion category, according to the audio signals respectively corresponding to the plurality of frames of second target images and the reference speech intensity, the change information of the emotion intensity corresponding to the target object under the second emotion category, includes:

[0101] Determine the emotion intensity corresponding to the target object in the second target image frame under the second emotion category according to the speech intensity of the audio signal corresponding to each frame of the second target image under each second emotion category and the reference speech intensity;

[0102] Based on the emotion intensity corresponding to the target object in each frame of the second target image under the second emotion category, change information of the emotion intensity corresponding to the target object under the second emotion category is determined.

[0103] In a possible implementation manner, the identification device further includes:

[0104] The first generating module is used to generate the corresponding smoothed emotion intensity in each frame of the second target image according to the emotion intensity corresponding to the target object in each frame of the second target image, the emotion intensity corresponding to the target object in other frames of the second target image adjacent to the frame of the second target image, the first weight corresponding to the frame of the second target image, and the second weights corresponding to the other frames of the second target image.

[0105] In a possible implementation manner, the selection module 501, when processing the surveillance video corresponding to the target object and selecting the target video corresponding to the target object, includes:

[0106] According to the short-time energy and the number of zero crossings of the audio signal corresponding to each frame of the surveillance video, intercepting the surveillance video corresponding to the voice from the surveillance video;

[0107] Input each frame of the surveillance video into a pre-trained face detection model to obtain a detection result output by the face detection model, and based on the detection result, intercept a surveillance video including a face from the surveillance video;

[0108] The target video is generated based on the surveillance video corresponding to the voice and the surveillance video including the face.

[0109] In a possible implementation manner, the identification device further includes:

[0110] The third determination module is used to determine the speech intensity of the audio signal corresponding to each frame of the target image according to the initial speech intensity of the audio signal corresponding to each frame of the target image and the initial speech intensity of the audio signals corresponding to multiple frames of target images adjacent to the frame of the target image.

[0111] In a possible implementation manner, the identification device further includes:

[0112] A second generating module is used to generate an emotion label matching the emotion intensity and the emotion category according to the emotion intensity and emotion category corresponding to the target object in each frame of the second target image;

[0113] The marking module is used to mark the emotion label at the second target image of the frame.

[0114] In a possible implementation manner, the identification device further includes:

[0115] The third generating module is used to generate and display business prompt information matching the change information of the emotion intensity according to the change information of the emotion intensity corresponding to the target object in each second emotion category.

[0116] The emotion intensity recognition device provided in the embodiment of the present application determines the emotion category of the target object in each frame of the target video based on the expression of the target object in each frame of the target image of the target video, and further determines the emotion intensity of the target user in each frame of the target video based on the voice intensity reflected in the audio signal of the target object in each frame of the target image. This method of recognizing the emotion category and emotion intensity has high accuracy and efficiency.

[0117] See also Figure 6 Show, Figure 6 An embodiment of the present application provides an electronic device 600, which includes: a processor 601, a memory 602 and a bus, wherein the memory 602 stores machine-readable instructions executable by the processor 601. When the electronic device is running, the processor 601 communicates with the memory 602 via the bus, and the processor 601 executes the machine-readable instructions to perform the steps of the above-mentioned emotion intensity recognition method.

[0118] Specifically, the above-mentioned memory 602 and processor 601 can be general-purpose memory and processor, which are not specifically limited here. When the processor 601 runs the computer program stored in the memory 602, the above-mentioned emotion intensity recognition method can be executed.

[0119] Corresponding to the above-mentioned method for identifying emotion intensity, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for identifying emotion intensity are executed.

[0120] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0121] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0122] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0123] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that can be executed by a processor. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for identifying emotional intensity, characterized in that: The identification method comprises: Processing the surveillance video corresponding to the target object, and selecting the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal; Input each frame of the target image in the target video into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model; For a plurality of frames of first target images corresponding to a first emotion category, determining a reference speech intensity according to speech intensities of audio signals respectively corresponding to the plurality of frames of first target images; For multiple frames of second target images corresponding to each second emotion category, according to the audio signals respectively corresponding to the multiple frames of second target images, and the reference speech intensity, determine the change information of the emotion intensity corresponding to the target object under the second emotion category, including: determining the emotion intensity corresponding to the target object in the frame of the second target image under the second emotion category according to the speech intensity of the audio signal corresponding to each frame of the second target image under each second emotion category, and the reference speech intensity; determining the change information of the emotion intensity corresponding to the target object under the second emotion category based on the emotion intensity corresponding to the target object in each frame of the second target image under the second emotion category.

2. The method for identifying emotion intensity according to claim 1, characterized in that: The identification method further comprises: According to the emotion intensity corresponding to the target object in each frame of the second target image, the emotion intensity corresponding to the target object in other frames of the second target image adjacent to the frame of the second target image, the first weight corresponding to the frame of the second target image, and the second weights corresponding to the other frames of the second target image, the corresponding smoothed emotion intensity in each frame of the second target image is generated.

3. The method for identifying emotion intensity according to claim 1, characterized in that: The step of processing the surveillance video corresponding to the target object and selecting the target video corresponding to the target object includes: According to the short-time energy and the number of zero crossings of the audio signal corresponding to each frame of the surveillance video, intercepting the surveillance video corresponding to the voice from the surveillance video; Input each frame of the surveillance video into a pre-trained face detection model to obtain a detection result output by the face detection model, and based on the detection result, intercept a surveillance video including a face from the surveillance video; The target video is generated based on the surveillance video corresponding to the voice and the surveillance video including the face.

4. The method for identifying emotion intensity according to claim 1, characterized in that: The speech intensity of the audio signal corresponding to each frame of the target image is determined in the following way: The speech intensity of the audio signal corresponding to each frame of the target image is determined according to the initial speech intensity of the audio signal corresponding to each frame of the target image and the initial speech intensity of the audio signals corresponding to multiple frames of the target image adjacent to the frame of the target image.

5. The method for identifying emotion intensity according to claim 1, characterized in that: The identification method further comprises: Generate an emotion label that matches the emotion intensity and the emotion category according to the emotion intensity and the emotion category corresponding to the target object in each frame of the second target image; The emotion label is marked at the second target image of the frame.

6. The method for identifying emotion intensity according to claim 1, characterized in that: The identification method further comprises: According to the change information of the emotion intensity corresponding to the target object in each second emotion category, business prompt information matching the change information of the emotion intensity is generated and displayed.

7. A device for identifying emotional intensity, characterized in that: The identification device comprises: A selection module is used to process the surveillance video corresponding to the target object and select the target video corresponding to the target object; wherein each frame of the target image in the target video includes the facial information of the target object and corresponds to a matching audio signal; An input module, used to input each frame of the target image in the target video into a pre-trained emotion classification model to obtain the emotion category of the frame of the target image output by the emotion classification model; A first determination module is used to determine a reference speech intensity according to speech intensities of audio signals respectively corresponding to the multiple frames of first target images corresponding to the first emotion category; The second determination module is used to determine the change information of the emotion intensity corresponding to the target object under the second emotion category for multiple frames of second target images corresponding to each second emotion category according to the audio signals respectively corresponding to the multiple frames of second target images and the reference speech intensity, including: determining the emotion intensity corresponding to the target object in the frame of the second target image under the second emotion category according to the speech intensity of the audio signal corresponding to each frame of the second target image under each second emotion category and the reference speech intensity; determining the change information of the emotion intensity corresponding to the target object under the second emotion category based on the emotion intensity corresponding to the target object in each frame of the second target image under the second emotion category.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for identifying the intensity of emotion as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method for identifying the intensity of emotion as claimed in any one of claims 1 to 6.

Images