Video evaluation method and device, equipment and storage medium

Abstract

The invention discloses a video evaluation method and device, equipment and a storage medium. Performing audio-video separation on the target video to obtain an audio sequence and a video sequence of the target video; performing audio feature extraction on audio signals in the audio sequence, and screening target audio features from the extracted audio features; identifying video content in the video sequence to obtain expression feature information and action feature information corresponding to the video sequence; and determining video evaluation information of the target video according to the target audio feature, the expression feature information and the action feature information. According to the method, the target audio features, the expression feature information and the action feature information are fused, the states and behaviors of the participants are interpreted more comprehensively and accurately by means of mutual complementation of the data sources, the accuracy, reliability and efficiency of video evaluation information are improved, and the user experience is improved. Therefore, powerful support is provided for promotion of follow-up related services and improvement of service quality and efficiency.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a video evaluation method, device, equipment and storage medium. Background Art

[0002] In the field of financial services, especially the banking industry, the quality of customer service directly affects customer satisfaction and loyalty. At the same time, accurate understanding of customer attitudes can help to adjust service strategies in a timely and effective manner, thereby improving service efficiency and quality.

[0003] However, traditional methods of evaluating service levels and customer attitudes rely on questionnaires and post-event feedback. These methods have problems such as poor timeliness, information lag, difficulty in obtaining information, and inability to trace and verify information. They cannot meet the needs of the banking and other financial services industries, making it difficult for the banking and other financial services industries to provide targeted training and guidance in a timely manner. At the same time, they are unable to adjust service strategies in a timely manner according to customer sentiment, which limits the improvement of service efficiency and quality. Summary of the invention

[0004] The present invention provides a video evaluation method, device, equipment and storage medium to solve the problem that the service efficiency and quality improvement of financial service industries such as banking are limited due to the lack of timely and reliable emotional feedback and performance evaluation.

[0005] In a first aspect, an embodiment of the present invention provides a video evaluation method, the method comprising:

[0006] Separating the target video from audio and video to obtain an audio sequence and a video sequence of the target video;

[0007] Extracting audio features from the audio signal in the audio sequence, and screening target audio features from the extracted audio features;

[0008] Identify the video content in the video sequence to obtain expression feature information and action feature information corresponding to the video sequence;

[0009] The video evaluation information of the target video is determined according to the target audio feature, the expression feature information and the action feature information.

[0010] In a second aspect, an embodiment of the present invention provides a video evaluation device, the device comprising:

[0011] An audio and video separation module is used to separate the audio and video of the target video to obtain the audio sequence and video sequence of the target video;

[0012] An audio feature extraction module, used to extract audio features from the audio signal in the audio sequence, and filter target audio features from the extracted audio features;

[0013] A video content recognition module is used to recognize the video content in the video sequence and obtain expression feature information and action feature information corresponding to the video sequence;

[0014] The evaluation information determination module is used to determine the video evaluation information of the target video according to the target audio feature and the expression feature information and the action feature information.

[0015] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0016] at least one processor;

[0017] and a memory communicatively coupled to the at least one processor;

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video evaluation method described in any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video evaluation method described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention obtains the audio sequence and video sequence of the target video by separating the audio and video of the target video; extracts the audio features of the audio signal in the audio sequence, and selects the target audio features from the extracted audio features; identifies the video content in the video sequence, obtains the expression feature information and action feature information corresponding to the video sequence; and determines the video evaluation information of the target video according to the target audio features, expression feature information and action feature information. The method uses the mutual complementation of various data sources based on the target audio features, expression feature information and action feature information to provide a more comprehensive and accurate interpretation of the participant's state and behavior, and improves the accuracy, reliability and efficiency of the video evaluation information. At the same time, a video evaluation method provided by the embodiment of the present invention effectively solves the problems of poor timeliness, information lag, difficulty in obtaining and inability to trace and verify in traditional evaluation methods that rely on questionnaires and post-event feedback, and can determine the video evaluation information in a timely, objective and convenient manner, and repeatedly view and analyze it, thereby providing strong support for the optimization, promotion and improvement of service quality and efficiency of subsequent related businesses.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flowchart of a video evaluation method provided by an embodiment of the present invention;

[0024] Figure 2 A schematic diagram of the structure of a video evaluation device provided by an embodiment of the present invention;

[0025] Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device that can be used to implement an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "target", "original", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] It should be noted that in the field of financial services, especially in the banking industry, service levels and customer attitudes are often evaluated through questionnaires and post-event feedback (such as telephone follow-up). These methods have problems such as poor timeliness (such as feeling satisfied at the time, but becoming average after a period of time), information lag (such as follow-up calls not made in a timely manner or filling out questionnaires when handling business again), difficulty in obtaining (such as follow-up calls not being answered) and inability to trace back and verify (such as leaving after filling out the questionnaire and being unable to confirm whether it is filled out correctly; service personnel lack experience and perceive customer attitudes incorrectly, and are unable to review and help judge). As a result, service efficiency and quality improvements are limited, and customer experience is affected.

[0029] Based on this, an embodiment of the present invention provides a video evaluation method. Figure 1 A flowchart of a video evaluation method provided in an embodiment of the present invention. The embodiment of the present invention can be applied to scenarios in which the emotional state (such as enthusiasm, satisfaction, etc.) of participants in a communication process is evaluated. The method can be executed by a video evaluation device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which is preferably a mobile terminal, a desktop computer, a laptop computer, a server, etc.

[0030] like Figure 1 As shown, the video evaluation method provided by the embodiment of the present invention may specifically include:

[0031] S101, separating the audio and video of a target video to obtain an audio sequence and a video sequence of the target video.

[0032] The target video can be understood as a video clip that meets the video evaluation requirements and only contains a single conversation party.

[0033] It is understandable that in order to ensure the focus and effectiveness of the evaluation, each video evaluation should only be targeted at a single participant who is talking in the video. Therefore, the video clip selected as the target video should contain video content of only one participant to avoid confusion in the evaluation process.

[0034] In this embodiment, a video clip that needs to be evaluated is selected as the target video, and then the audio and video in the segmented target video are separated by using a codec, channel separation, etc., so as to obtain the audio sequence and video sequence of the target video.

[0035] S102: extracting audio features from audio signals in the audio sequence, and screening target audio features from the extracted audio features.

[0036] Among them, the target audio feature can be understood as the most useful feature for the video evaluation task selected from the extracted audio features, that is, as the feature of the entire audio sequence.

[0037] In this embodiment, audio features are extracted from the audio signals in the audio sequence. It can be understood that the audio features can reflect the emotional state and tone of the participants, etc. The audio feature extraction can include extracting pitch, rhythm, volume, spectral features (such as Mel-frequency cepstral coefficients), spectrogram, etc., and filtering out the audio features that are most useful for the video evaluation task from the extracted audio features, and using the filtered audio features as the target audio features of the entire audio sequence.

[0038] S103: Identify the video content in the video sequence to obtain expression feature information and action feature information corresponding to the video sequence.

[0039] In this embodiment, the video content in the video sequence is identified frame by frame or continuously frame by frame through feature extraction algorithms or deep learning models, so as to obtain expression feature information and action feature information corresponding to the video sequence. Among them, the expression feature information is used to reflect the emotional state and interest level of the participants, and the expression feature information can be determined based on information such as the category of expression, the number of each expression category and / or the proportion of each expression category; the action feature information is used to reflect the emotional state, intention and involvement of the participants, and the action feature information can be determined based on information such as the category of action, the number of each action category and / or the proportion of each action category.

[0040] Exemplarily, facial expression recognition can be performed on the video content through face detection, feature point positioning (such as eyes, eyebrows, corners of the mouth, etc.) and expression classification (such as happiness, disappointment, anger, etc.) to obtain expression feature information corresponding to the video sequence.

[0041] Exemplarily, the video content in the video sequence can be identified through feature extraction algorithms, target recognition algorithms, target detection models, etc., to obtain expression feature information and action feature information corresponding to the video sequence.

[0042] It should be noted that by recognizing the facial expressions of participants in the video content, it is possible to identify the emotions expressed by the participants, such as happiness, anger, and surprise, etc., and by recognizing the body movements of participants in the video content, it is possible to identify the actions taken by the participants, such as shaking hands, crossing arms, etc. Compared with the existing method of judging emotions only by facial expressions, by adding the recognition of body movements, the accuracy of video evaluation can be effectively improved.

[0043] S104: Determine video evaluation information of the target video according to the target audio feature, expression feature information, and action feature information.

[0044] The video evaluation information may be understood as quantitative information reflecting the emotional state of the participants in the target video, and the video evaluation information may be a score value or an evaluation level.

[0045] In this embodiment, the video evaluation information of the target video is comprehensively determined based on the target audio features, expression feature information, and action feature information, so as to evaluate the emotional state of the participants in the target video from multiple dimensions.

[0046] For example, the target audio features, expression feature information, and action feature information can be analyzed by an artificial intelligence algorithm to determine the video evaluation information of the target video, so as to accurately evaluate the performance of the participants in the video. The artificial intelligence algorithm can be a machine learning algorithm, a deep learning algorithm, a neural network model, etc., which is not limited in the embodiments of the present invention.

[0047] Exemplarily, a method for determining video evaluation information may be to use an artificial intelligence algorithm to respectively determine evaluation values ​​of target audio features, expression feature information, and action feature information, and use the determined three evaluation values ​​as video evaluation information of the target video.

[0048] Exemplarily, following the above description, after the three evaluation values ​​are determined, an evaluation value may be determined as the video evaluation information of the target video by accumulating, weighted summing, averaging, etc. the three evaluation values.

[0049] Exemplarily, a method for determining video evaluation information may also be to use an artificial intelligence algorithm to comprehensively analyze target audio features, expression feature information, and action feature information, and directly determine an evaluation value as the video evaluation information of the target video.

[0050] A video evaluation method provided by an embodiment of the present invention obtains an audio sequence and a video sequence of a target video by performing audio and video separation on a target video; extracts audio features from an audio signal in an audio sequence, and selects target audio features from the extracted audio features; identifies video content in a video sequence, obtains expression feature information and action feature information corresponding to the video sequence; and determines video evaluation information of the target video according to the target audio features, expression feature information, and action feature information. The method uses the mutual complementation of various data sources based on the target audio features, expression feature information, and action feature information to provide a more comprehensive and accurate interpretation of the participant's state and behavior, thereby improving the accuracy, reliability, and efficiency of the video evaluation information. At the same time, a video evaluation method provided by an embodiment of the present invention effectively solves the problems of poor timeliness, information lag, difficulty in obtaining, and inability to trace and verify in traditional evaluation methods that rely on questionnaires and post-event feedback, and can timely, objectively, and conveniently determine video evaluation information, and repeatedly view and analyze it, thereby providing strong support for the optimization, promotion, and improvement of service quality and efficiency of subsequent related businesses.

[0051] As a first optional embodiment of this embodiment, based on the above embodiment, the audio and video separation of the target video can be performed to obtain the audio sequence and video sequence of the target video, which can be specifically implemented as the following steps:

[0052] a1) Dividing the target video into segments according to a set time interval to obtain at least one video segment.

[0053] In this embodiment, when the target video is long or there is a need for a more detailed and accurate analysis of the target video, the target video can be divided into at least one video segment according to a set time interval, so that subsequent processing is more efficient and the obtained video evaluation information is more accurate.

[0054] Exemplarily, a one-minute target video can be segmented using a video editing tool or editing library (such as FFmpeg, OpenCV, etc.), and the segmentation time interval can be fixed (for example, every 5 seconds is divided into a video segment) or dynamically determined according to the logical structure of the video content.

[0055] b1) For each video segment, separate the audio signal from the video segment to obtain an audio sequence and a video sequence corresponding to the video segment.

[0056] In this embodiment, the audio signal in each obtained video segment is separated so as to perform separate audio analysis, thereby obtaining an audio sequence and a video sequence corresponding to the video segment.

[0057] It should be noted that the audio sequence can be used to analyze the speaking speed and tone changes of the participants, while the video sequence can be used to analyze the facial expressions and body language of the participants.

[0058] The above technical scheme of this embodiment improves the efficiency of subsequent processing by dividing the target video into multiple video segments according to set time intervals, and separating audio signals from the video segments to obtain audio sequences and video sequences corresponding to the video segments, and enables more detailed analysis and evaluation of the target video. For example, for a 1-minute target video, if no division is performed, an evaluation is given to the entire video. If 12 video segments are obtained by division at a time interval of 5 seconds, and the audio sequence and video sequence corresponding to each video segment are obtained, then each video segment can be processed in parallel and evaluated, and the evaluation of each video segment can be summarized as the video evaluation information of the target video, further improving the accuracy and efficiency of the video evaluation information.

[0059] As a second optional embodiment of this embodiment, the step of extracting audio features from the audio signal in the audio sequence and screening target audio features from the extracted audio features may be specifically optimized into the following steps:

[0060] a2) for each video segment formed by dividing the target video, performing speech signal enhancement processing on the audio sequence separated from the video segment to obtain an enhanced audio sequence after the enhancement processing.

[0061] In this embodiment, for each video segment formed by dividing the target video, the speech signal in the audio sequence can be enhanced, such as by using pre-emphasis, short-time average amplitude, bilinear transformation, wavelet method, spectral subtraction method and the like, to improve the quality of the speech signal and reduce noise and other interference, thereby obtaining an enhanced audio sequence after enhancement processing.

[0062] As one implementation manner, this optional embodiment may perform speech signal enhancement processing on the audio sequence separated from the video segment to obtain an enhanced audio sequence after the enhancement processing, which is further embodied as follows:

[0063] a21) obtaining an audio signal of an audio sequence separated from the video segment, and performing signal emphasis processing on a high-frequency signal in the audio signal to obtain an emphasized audio signal.

[0064] In this embodiment, an audio signal of an audio sequence separated from a video segment is obtained. Since the high-frequency signal in the audio signal usually contains important speech information, such as unvoiced consonants, the high-frequency signal in the audio signal is emphasized to obtain an emphasized audio signal, so as to help eliminate the influence of lip radiation and improve the high-frequency resolution of speech, so as to facilitate subsequent feature extraction and analysis.

[0065] a22) dividing the emphasized audio signal into at least one short-time frame according to the frame length, wherein the audio signal in the previous short-time frame overlaps in the next short-time frame.

[0066] In this embodiment, the emphasized audio signal is divided into at least one short-time frame according to a certain frame length (usually 20-30 milliseconds). In order to ensure the continuity of the signal, the end of the previous short-time frame will overlap with the beginning of the next short-time frame, and this overlap is usually 50% of the frame length.

[0067] a23) multiplying each of the short-time frames by a Hamming window respectively, and combining and smoothing the multiplication results by a window function to obtain an audio combined signal.

[0068] In this embodiment, each of the short-time frames is multiplied by a Hamming window, and the multiplication results are merged (superimposed) through a window function, and can be smoothed by an algorithm such as weighted averaging, so as to obtain a more continuous and smooth audio merged signal.

[0069] a24) performing direct current component removal processing on the audio combined signal in the frequency domain, and converting the obtained removal result into the time domain to determine the enhanced audio sequence corresponding to the audio sequence.

[0070] It should be noted that, since the DC component usually includes the DC offset of the audio signal and does not include useful information about the speech characteristics, removing the DC component can reduce the DC offset of the audio signal and improve the quality of the audio signal.

[0071] In this embodiment, Fourier transform is used to transform the audio merged signal into the frequency domain, and the DC component of the audio merged signal is removed in the frequency domain, that is, the element with index 0 in the result after Fourier transform is set to 0 to obtain the removal result, and then an inverse Fourier transform is performed to convert the obtained removal result into the time domain, and the removal result converted to the time domain is determined as the enhanced audio sequence corresponding to the audio sequence.

[0072] The above technical scheme of this embodiment performs signal emphasis processing on the high-frequency signal in the audio signal, then divides it into at least one short-time frame according to the frame length, and multiplies each of the short-time frames with a Hamming window respectively, and then merges and smoothes the multiplication results through a window function, and finally performs DC component removal processing to obtain an enhanced audio sequence corresponding to the audio sequence, thereby improving the quality of the audio signal. Compared with the audio signal after enhancement processing, it has a better signal-to-noise ratio and clearer speech features, and is more suitable for subsequent audio feature extraction tasks to improve the quality of the extracted audio features.

[0073] b2) extracting features of the audio signal in the enhanced audio sequence through an audio feature extraction model to obtain audio features of the enhanced audio sequence.

[0074] In this embodiment, an audio feature extraction model is used to perform feature extraction on the enhanced audio sequence to obtain audio features of the enhanced audio sequence, and the audio features may include features extracted from the audio signal in the enhanced audio sequence in the time domain and features extracted in the frequency domain after converting the time domain audio signal to the frequency domain.

[0075] Exemplarily, the characteristics of the audio signal in the time domain, such as short-time energy, short-time zero-crossing rate, short-time average amplitude, etc., can be directly extracted from the time series of the audio signal; the time domain audio signal can also be converted to the frequency domain through Fourier transform and other methods, and the characteristics of the audio signal in the frequency domain, such as Mel-frequency cepstral coefficients, resonance peak frequency, spectrum centroid and spectrum distribution, can be obtained by analyzing the frequency components and energy distribution of the signal.

[0076] c2) selecting feature values ​​of a set feature dimension from the audio features as target audio features corresponding to the audio sequence associated with the video segment.

[0077] In this embodiment, feature values ​​are screened from the audio features according to set feature dimensions based on video evaluation requirements, and the screened feature vectors of the set feature dimensions are used as target audio features corresponding to the audio sequence associated with the video segment, for subsequent video evaluation information determination.

[0078] Exemplarily, the feature dimension can be set to four dimensions, that is, four eigenvalues ​​are screened. The screened eigenvalues ​​can be the value of the Mel-frequency cepstral coefficient feature (used to extract the features most relevant to speech emotion recognition), the value of the short-time energy feature (used to reflect the energy of the audio signal in a short period of time, that is, the intensity and change of the sound, thereby reflecting the emotional state of the speech), the value of the short-time zero-crossing rate feature (used to reflect the rate of change of the audio signal, indirectly reflecting the emotional state of the speech) and the value of the resonance peak frequency feature (used to reflect whether the emotion is excited or not). That is, the vector composed of the values ​​of the above four features is the target audio feature.

[0079] The above technical scheme of this embodiment obtains the features most relevant to emotion recognition in the audio sequence associated with the video segment by performing enhancement processing, feature extraction and feature value screening on the audio signal, thereby avoiding the curse of dimensionality and feature redundancy, thereby improving the expressive power and computational efficiency of the features, and providing strong support for the subsequent determination of accurate video evaluation information.

[0080] As a third optional embodiment of this embodiment, based on the above embodiment, the step of identifying the video frames in the video sequence to obtain the expression feature information and action feature information corresponding to the video sequence can be specifically implemented as the following steps:

[0081] a3) for each video segment formed by dividing the target video, performing target recognition on a video sequence formed by audio separation of the video segment to obtain an action category and an expression category contained in each video frame in the video sequence.

[0082] In this embodiment, a machine learning algorithm, a deep learning model (such as a convolutional neural network model), and a target detection model (such as a YOLO model) can be used to perform target recognition on a video sequence formed after audio separation of each video segment divided from the target video, so as to respectively identify the action category and expression category contained in each video frame in the video sequence.

[0083] For example, a target detection model, such as a YOLO model, can be used for target recognition. First, the YOLO model is trained according to the selected actions, such as handshake, head shaking, gesture expression, standing or sitting, arms crossed, and body twisting. After the training is completed, each video segment is processed frame by frame, and each frame is input into the YOLO model for target detection. For each video frame, the YOLO model will identify the action category of the participant, that is, one or more actions selected for training.

[0084] Continuing with the above description, if you want to perform facial expression category recognition, you can select expressions such as happy, natural, indifferent, sad, angry, disgusted, and disappointed to train the YOLO model. The use after training is similar to the above method. Finally, the YOLO model will recognize the expression category of the participant for each video frame.

[0085] b3) Summarize the first category quantity of the action category and the second category quantity of the expression category in each video frame.

[0086] The first category quantity can be understood as the total number of all action categories, and the second category quantity can be understood as the total number of all expression categories. It should be noted that both the first category quantity and the second category quantity are not less than the number of video frames.

[0087] In this embodiment, the number of action categories recognized in each video frame is accumulated to obtain a first category quantity, and the number of expression categories recognized in each video frame is accumulated to obtain a second category quantity of the expression category.

[0088] Exemplarily, there is a video sequence with 3 video frames, where the action category identified in the first video frame is handshake, the action category identified in the second video frame is handshake and head shaking, and the action category identified in the third video frame is gesture expression. Then the first category quantity is the sum of the number of occurrences of handshake, the number of occurrences of head shaking, and the number of occurrences of gesture expression, that is, the first category quantity is 4.

[0089] c3) determining a first ratio of the amount of action categories included in the video frame to the amount of the first category, and constructing action feature information of the video sequence based on the first ratios.

[0090] The action category quantity can be understood as the number of each action category appearing. The first proportion can be understood as the ratio of the action category quantity of each action category to the first category quantity, that is, the proportion of the number of each action category appearing to the number of all action categories appearing.

[0091] In this embodiment, the action category quantity of each action category included in the video frame is summarized, and then each action category quantity is divided by the first category quantity, and each result obtained is determined as the first proportion corresponding to each action category, and the action feature information of the video sequence is formed based on each first proportion, and the action feature information can reflect the distribution and change of the action category in the video sequence. Exemplarily, each first proportion can be used as an element to form a vector, and the vector is used as the action feature information.

[0092] Exemplarily, if the first category quantity is 4, wherein the action category quantity of the handshake action category is 2, the first proportion corresponding to the handshake action category is 0.5 or 50%.

[0093] d3) determining a second ratio of the amount of expression categories included in the video frame to the amount of the second categories, and constructing expression feature information of the video sequence based on the second ratios.

[0094] The expression category quantity can be understood as the number of each expression category appearing. The second proportion can be understood as the ratio of the expression category quantity of each expression category to the second category quantity, that is, the proportion of the number of each expression category appearing to the number of all expression categories appearing.

[0095] In this embodiment, the expression category quantity of each expression category included in the video frame is summarized, and then the quotient of each expression category quantity and the second category quantity is taken respectively, and each obtained result is determined as the second proportion corresponding to each expression category, and the expression feature information of the video sequence is constructed based on each second proportion. The expression feature information can reflect the distribution and change of expressions in the video sequence.

[0096] The above technical solution of this embodiment provides rich feature data for the analysis of video content by determining and quantifying the actions and expressions in the video, thereby providing strong support for realizing automated and reliable video content analysis and evaluation.

[0097] As a fourth optional embodiment of this embodiment, the step of determining the video evaluation information of the target video according to the target audio feature and the expression feature information and the action feature information may be specifically implemented as the following steps:

[0098] a4) for each video segment formed by dividing the target video, feature fusion is performed on the target audio feature, expression feature information and action feature information corresponding to the video segment, and a target feature matrix is ​​determined based on the feature fusion result.

[0099] The target feature matrix can be understood as a more comprehensive and accurate feature representation corresponding to each video segment, which is used to provide feature information for evaluating the video segment.

[0100] In this embodiment, the target audio features, expression feature information, and action feature information corresponding to each video segment formed by dividing the target video are respectively fused through simple feature splicing, weighted averaging, or using more complex machine learning methods (such as autoencoders, etc.), and the target feature matrix is ​​determined based on the feature fusion results.

[0101] As one implementation method, this optional embodiment may perform feature fusion on the target audio features, expression feature information, and action feature information corresponding to the video segment, and determine the target feature matrix based on the fused feature fusion matrix, which is further concretized as follows:

[0102] a41) performing standardization processing on the target audio features, expression feature information and action feature information respectively, and merging the respectively formed standardized features to obtain a feature fusion matrix.

[0103] In this embodiment, the target audio features, expression feature information, and action feature information are standardized respectively so that the scales of different features are consistent for subsequent merging. After obtaining the standardized features formed after the standardization process, the standardized features are merged into a feature matrix, i.e., a feature fusion matrix. Exemplarily, the standardization process can adopt a Z-score method so that the features have the same scale and variance.

[0104] a42) Using principal component analysis to perform dimensionality reduction processing on the feature fusion matrix, and determining the dimensionality reduction processing result as the target feature matrix.

[0105] In this embodiment, the principal component analysis method is used to reduce the dimension of the feature fusion matrix, while retaining the main information of the feature fusion matrix as much as possible, to obtain the dimensionality reduction result and determine the dimensionality reduction result as the target feature matrix. The target feature matrix contains the main information of the feature fusion matrix, but has a lower dimension and is more suitable for subsequent analysis and processing.

[0106] In this embodiment, the method of using the principal component analysis method to perform dimensionality reduction processing on the feature fusion matrix can be to apply the principal component analysis method to convert the feature fusion matrix into a set of linearly unrelated variables, called principal components, through orthogonal transformation. These principal components are sorted according to the size of the variance, and the first principal component has the largest variance, that is, it contains the most information, and so on. Then, according to the pre-set number of principal components selected and retained, the principal component selection is started from the principal component with the largest variance, and the selected principal component is used as a new feature to form a new feature matrix. The new feature matrix is ​​the feature representation result obtained after the principal component analysis method is used to perform dimensionality reduction processing on the feature fusion matrix, that is, the dimensionality reduction processing result.

[0107] The above technical scheme of this embodiment merges the standardized features formed after standardizing the target audio features, expression feature information and action feature information, and maps the feature fusion matrix obtained after the merger from the high-dimensional feature space to the low-dimensional feature space, while retaining the main information, improving the representation and generalization capabilities of the features, thereby improving the efficiency and accuracy of subsequent video evaluation.

[0108] b4) inputting the target feature matrix into a trained feature evaluation model, and determining the feature evaluation value output by the feature evaluation model as the video evaluation value of the video segment.

[0109] Among them, the feature evaluation value can be understood as the quantitative evaluation of each video segment output by the feature evaluation model according to the target feature matrix, which is used to reflect the emotional state of the participants in the video segment, such as enthusiasm, satisfaction or involvement, etc., which is equivalent to the video evaluation value.

[0110] In this embodiment, each target feature matrix is ​​input into a trained feature evaluation model, which may be a support vector machine, a neural network model, etc. The feature evaluation model evaluates each corresponding video segment according to the input target feature matrix, and outputs a feature evaluation value corresponding to each video segment, and determines the feature evaluation value as the video evaluation value of the video segment to reflect the emotional state of the participants in the video segment.

[0111] It should be noted that the emotional state of the participants in the video segment corresponding to the video evaluation value can be determined according to the preset emotion corresponding interval. For example, the preset emotion corresponding interval can be that the video evaluation value is in [0, 20) for unenthusiastic, (20, 40) for moderately enthusiastic, (40, 60) for enthusiastic, and (60, 100) for very enthusiastic. That is, when the video evaluation value corresponding to the video segment is 50, it is determined that the emotional state of the participant is enthusiastic.

[0112] c4) forming a video rating curve based on the video rating values ​​of each of the video segments, and determining the video rating curve as the video rating information of the target video.

[0113] In this embodiment, the video rating values ​​of each video segment are summarized to form a video rating curve, which shows the change of the video rating value of the entire target video over time, that is, it can reflect the emotional change of the entire target video. Then, the video rating curve is determined as the video rating information of the target video, which shows the overall evaluation and change trend of the video content in the target video in an intuitive view, and the change of the video rating value of the participants can be viewed in segments.

[0114] It should be noted that the video rating curve can be formed according to each video rating value in a coordinate system composed of time and video rating value. Therefore, the video rating curve can be formed by connecting the video rating values ​​of each time point with a smooth curve, or it can be formed by connecting the video rating values ​​of each time point with a straight line.

[0115] The above technical scheme of this embodiment improves the accuracy and robustness of each video evaluation value by comprehensively utilizing multiple features of each video segment, and at the same time forms a video evaluation curve (i.e., video evaluation information) based on the video evaluation values ​​of each video segment, which displays the emotional changes and change trends of the participants in the entire target video in an intuitive view, and can flexibly view the changes in the video evaluation values ​​of the participants in segments and correspond to the corresponding video content, providing strong support for subsequent business promotion and service quality improvement.

[0116] Figure 2 FIG. 1 is a schematic diagram of the structure of a video evaluation device provided by an embodiment of the present invention. Figure 2 As shown, the device includes: an audio and video separation module 21, an audio feature extraction module 22, a video content recognition module 23 and an evaluation information determination module 24, wherein:

[0117] The audio and video separation module 21 is used to separate the audio and video of the target video to obtain the audio sequence and video sequence of the target video;

[0118] An audio feature extraction module 22, configured to extract audio features from the audio signal in the audio sequence and select target audio features from the extracted audio features;

[0119] A video content recognition module 23 is used to recognize the video content in the video sequence and obtain expression feature information and action feature information corresponding to the video sequence;

[0120] The evaluation information determination module 24 is used to determine the video evaluation information of the target video according to the target audio feature and the expression feature information and the action feature information.

[0121] The embodiment of the present invention provides a video evaluation device, which obtains the audio sequence and video sequence of the target video by separating the audio and video of the target video; extracts the audio features of the audio signal in the audio sequence, and selects the target audio features from the extracted audio features; identifies the video content in the video sequence, obtains the expression feature information and action feature information corresponding to the video sequence; and determines the video evaluation information of the target video according to the target audio features, expression feature information and action feature information. The method uses the mutual complementation of various data sources based on the target audio features, expression feature information and action feature information to provide a more comprehensive and accurate interpretation of the participant's state and behavior, and improves the accuracy, reliability and efficiency of the video evaluation information. At the same time, the video evaluation method provided by the embodiment of the present invention effectively solves the problems of poor timeliness, information lag, difficulty in obtaining and inability to trace and verify in the traditional evaluation method that relies on questionnaire surveys and post-event feedback, and can timely, objectively and conveniently determine the video evaluation information, and repeatedly view and analyze it, thereby providing strong support for the optimization, promotion and improvement of service quality and efficiency of subsequent related businesses.

[0122] Furthermore, the audio and video separation module 21 can be specifically used for:

[0123] Dividing the target video according to a set time interval to obtain at least one video segment;

[0124] For each video segment, an audio signal is separated from the video segment to obtain an audio sequence and a video sequence corresponding to the video segment.

[0125] Furthermore, the audio feature extraction module 22 may specifically include:

[0126] A signal enhancement processing unit, configured to perform speech signal enhancement processing on the audio sequence separated from the video segment to obtain an enhanced audio sequence after the enhancement processing;

[0127] An audio feature extraction unit, configured to extract features of the audio signal in the enhanced audio sequence through an audio feature extraction model to obtain audio features of the enhanced audio sequence;

[0128] The target audio feature determination unit is used to select feature values ​​of a set feature dimension from the frequency domain audio features as target audio features corresponding to the audio sequence associated with the video segment.

[0129] Furthermore, the signal enhancement processing unit can be specifically used for:

[0130] Acquire an audio signal of an audio sequence separated from the video segment, and perform signal emphasis processing on a high-frequency signal in the audio signal to obtain an emphasized audio signal;

[0131] Dividing the emphasized audio signal into at least one short-time frame according to the frame length, wherein the audio signal in the previous short-time frame overlaps in the next short-time frame;

[0132] Multiplying each of the short-time frames with a Hamming window respectively, and combining and smoothing the multiplication results through a window function to obtain an audio combined signal;

[0133] The audio combined signal is subjected to direct current component removal processing in the frequency domain, and the obtained removal result is converted into the time domain to be determined as an enhanced audio sequence corresponding to the audio sequence.

[0134] Furthermore, the video content identification module 23 can be specifically used for:

[0135] For each video segment formed by dividing the target video, target recognition is performed on a video sequence formed by audio separation of the video segment to obtain an action category and an expression category contained in each video frame in the video sequence;

[0136] Summarize the first category quantity of the action category and the second category quantity of the expression category in each video frame;

[0137] Determine a first ratio of the amount of action categories included in the video frame to the amount of the first category, and construct action feature information of the video sequence based on the first ratios;

[0138] A second ratio of the amount of the expression category included in the video frame to the amount of the second category is determined, and expression feature information of the video sequence is constructed based on each of the second ratios.

[0139] Furthermore, the evaluation information determination module 24 may specifically include:

[0140] A feature fusion unit is used to perform feature fusion on target audio features, expression feature information and action feature information corresponding to each video segment formed by dividing the target video, and determine a target feature matrix based on the feature fusion result;

[0141] A video evaluation value determination unit, used for inputting the target feature matrix into a trained feature evaluation model, and determining the feature evaluation value output by the feature evaluation model as the video evaluation value of the video segment;

[0142] The video rating information determining unit is used to form a video rating curve based on the video rating values ​​of each of the video segments, and determine the video rating curve as the video rating information of the target video.

[0143] Furthermore, the feature fusion unit can be specifically used for:

[0144] The target audio features, expression feature information, and action feature information are respectively standardized, and the standardized features formed are merged to obtain a feature fusion matrix;

[0145] The principal component analysis method is used to perform dimensionality reduction processing on the feature fusion matrix, and the dimensionality reduction processing result is determined as the target feature matrix.

[0146] The video evaluation device provided in the embodiment of the present invention can execute the video evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0147] Figure 3 A schematic diagram of the structure of an electronic device 30 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0148] like Figure 3As shown, the electronic device 30 includes at least one processor 31, and a memory connected to the at least one processor 31, such as a read-only memory (ROM) 32, a random access memory (RAM) 33, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 31 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 32 or the computer program loaded from the storage unit 38 to the random access memory (RAM) 33. In the RAM 33, various programs and data required for the operation of the electronic device 30 can also be stored. The processor 31, the ROM 32, and the RAM 33 are connected to each other through a bus 34. An input / output (I / O) interface 35 is also connected to the bus 34.

[0149] A number of components in the electronic device 30 are connected to the I / O interface 35, including: an input unit 36, such as a keyboard, a mouse, etc.; an output unit 37, such as various types of displays, speakers, etc.; a storage unit 38, such as a disk, an optical disk, etc.; and a communication unit 39, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 39 allows the electronic device 30 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0150] The processor 31 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 31 performs the various methods and processes described above, such as the video evaluation method.

[0151] In some embodiments, the video evaluation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 38. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 30 via the ROM 32 and / or the communication unit 39. When the computer program is loaded into the RAM 33 and executed by the processor 31, one or more steps of the video evaluation method described above may be performed. Alternatively, in other embodiments, the processor 31 may be configured to perform the video evaluation method in any other appropriate manner (e.g., by means of firmware).

[0152] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0154] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0156] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0157] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0158] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0159] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video evaluation method, characterized in that: include: Separating the target video from audio and video to obtain an audio sequence and a video sequence of the target video; Extracting audio features from the audio signal in the audio sequence, and screening target audio features from the extracted audio features; Identify the video content in the video sequence to obtain expression feature information and action feature information corresponding to the video sequence; The video evaluation information of the target video is determined according to the target audio feature, the expression feature information and the action feature information.

2. The method according to claim 1, characterized in that The step of separating the audio and video of the target video to obtain the audio sequence and the video sequence of the target video includes: Dividing the target video according to a set time interval to obtain at least one video segment; For each video segment, an audio signal is separated from the video segment to obtain an audio sequence and a video sequence corresponding to the video segment.

3. The method according to claim 1, characterized in that The step of extracting audio features from the audio signal in the audio sequence and selecting target audio features from the extracted audio features includes: For each video segment formed by dividing the target video, performing speech signal enhancement processing on an audio sequence separated from the video segment to obtain an enhanced audio sequence after the enhancement processing; Extracting features of the audio signal in the enhanced audio sequence using an audio feature extraction model to obtain audio features of the enhanced audio sequence; Feature values ​​of a set feature dimension are selected from the audio features to serve as target audio features corresponding to the audio sequence associated with the video segment.

4. The method according to claim 3, characterized in that: The performing speech signal enhancement processing on the audio sequence separated from the video segment to obtain an enhanced audio sequence after the enhancement processing comprises: Acquire an audio signal of an audio sequence separated from the video segment, and perform signal emphasis processing on a high-frequency signal in the audio signal to obtain an emphasized audio signal; Dividing the emphasized audio signal into at least one short-time frame according to the frame length, wherein the audio signal in the previous short-time frame overlaps in the next short-time frame; Multiplying each of the short-time frames with a Hamming window respectively, and combining and smoothing the multiplication results through a window function to obtain an audio combined signal; The audio combined signal is subjected to direct current component removal processing in the frequency domain, and the obtained removal result is converted into the time domain to be determined as an enhanced audio sequence corresponding to the audio sequence.

5. The method according to claim 1, characterized in that The step of identifying the video frames in the video sequence to obtain expression feature information and action feature information corresponding to the video sequence includes: For each video segment formed by dividing the target video, target recognition is performed on a video sequence formed by audio separation of the video segment to obtain an action category and an expression category contained in each video frame in the video sequence; Summarize the first category quantity of the action category and the second category quantity of the expression category in each video frame; Determine a first ratio of the amount of action categories included in the video frame to the amount of the first category, and construct action feature information of the video sequence based on the first ratios; A second ratio of the amount of the expression category included in the video frame to the amount of the second category is determined, and expression feature information of the video sequence is constructed based on each of the second ratios.

6. The method according to claim 1, characterized in that The step of determining the video evaluation information of the target video according to the target audio feature, the expression feature information, and the action feature information includes: For each video segment formed by dividing the target video, feature fusion is performed on the target audio feature, expression feature information, and action feature information corresponding to the video segment, and a target feature matrix is ​​determined based on the feature fusion result; Inputting the target feature matrix into a trained feature evaluation model, and determining the feature evaluation value output by the feature evaluation model as the video evaluation value of the video segment; A video rating curve is formed based on the video rating values ​​of each of the video segments, and the video rating curve is determined as the video rating information of the target video.

7. The method according to claim 6, characterized in that The step of fusing the target audio features, expression feature information, and action feature information corresponding to the video segment, and determining a target feature matrix based on a feature fusion matrix formed by the fusion, includes: The target audio features, expression feature information, and action feature information are respectively standardized, and the standardized features formed are merged to obtain a feature fusion matrix; The principal component analysis method is used to perform dimensionality reduction processing on the feature fusion matrix, and the dimensionality reduction processing result is determined as the target feature matrix.

8. A video evaluation device, characterized in that: include: An audio and video separation module is used to separate the audio and video of the target video to obtain the audio sequence and video sequence of the target video; An audio feature extraction module, used to extract audio features from the audio signal in the audio sequence, and filter target audio features from the extracted audio features; A video content recognition module is used to recognize the video content in the video sequence and obtain expression feature information and action feature information corresponding to the video sequence; The evaluation information determination module is used to determine the video evaluation information of the target video according to the target audio feature and the expression feature information and the action feature information.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively coupled to the at least one processor; The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the video evaluation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the video evaluation method according to any one of claims 1 to 7 when executed.

Images