Intelligent video content analysis and interaction system

Abstract

The invention relates to the technical field of video interaction, and discloses an intelligent video content analysis and interaction system, which comprises a video extraction module, an interaction action module, an analysis adjustment module, an intelligent recommendation module and a video evaluation module, the video extraction module is configured to preprocess an input video and extract video features to establish a training data set, the interaction action module is configured to capture gesture actions, voice commands and touch actions for watching real-time video images, and the analysis and adjustment module is configured to determine emotion types according to fusion feature vectors. The intelligent recommendation module is configured to determine a first recommendation video according to an emotion type, determine a second recommendation video according to an interaction strategy, and determine a third recommendation video based on a machine learning algorithm according to historical watching data and historical interaction behaviors, and the video evaluation module is configured to determine historical matching similarity; and evaluating the recommended video quality according to the historical matching similarity. The method has the characteristic of intelligently recommending the video.

Description

Technical Field

[0001] The present invention relates to the field of video interaction technology, and in particular to an intelligent video content analysis and interaction system. Background Art

[0002] With the rapid development of artificial intelligence and machine learning technologies, video content analysis technology has been widely used in many fields. Users are faced with the problem of selecting appropriate videos from massive videos. Especially in the fields of education, entertainment, and advertising, how to recommend high-quality video content that meets their interests and needs has become a problem faced by intelligent interactive systems. However, existing video analysis and interactive systems have limitations, lack the ability to interact with users, rely only on static data analysis, lack real-time, dynamic user feedback, and are difficult to adjust the interactive mode according to user preferences or changes in viewing habits. In addition, it is impossible to intelligently recommend related videos based on user emotions or historical viewing data. Secondly, in the traditional recommendation model, the quality of the video itself cannot be effectively guaranteed. Low-quality videos will affect the user's interactive effect and even cause user loss.

[0003] Therefore, how to provide an intelligent video content analysis and interaction system is a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0004] In view of this, the present invention proposes an intelligent video content analysis and interaction system, which aims to solve the problems of difficulty in adjusting the interaction mode, inability to intelligently recommend related videos and inability to effectively guarantee the quality of the video itself.

[0005] The present invention proposes an intelligent video content analysis and interaction system, comprising: Video extraction module, interactive action module, analysis and adjustment module, intelligent recommendation module and video evaluation module; The video extraction module is configured to pre-process the input video and extract video features to establish a training data set, train the deep learning model according to the training data set to obtain a target model, and output a video result for the real-time video image based on the target model; The interactive action module is configured to capture gesture actions, voice commands and touch actions of watching the real-time video image, and calculate preference values ​​of each mode according to the gesture actions, the voice commands and the touch actions, adjust the interactive strategy according to the preference values, and determine whether to adjust the video playback speed according to the interactive strategy; The analysis and adjustment module is configured to collect facial images of a person watching the real-time video image and pre-process them to obtain facial data, collect physiological signals and pre-process them to obtain physiological signal data, establish a convolutional network model, input the facial data and the physiological signal data into the convolutional network model to obtain a feature vector, perform splicing or weighted processing on the feature vector to obtain a fused feature vector, and determine the emotion type according to the fused feature vector; The intelligent recommendation module is configured to determine a first recommended video according to the emotion type, determine a second recommended video according to the interaction strategy, and determine a third recommended video based on a machine learning algorithm according to historical viewing data and historical interaction behaviors; The video evaluation module is configured to collect a historical recommended video data set, compare the first recommended video, the second recommended video and the third recommended video with the historical recommended video data set respectively, determine the historical matching similarity, and evaluate the recommended video quality according to the historical matching similarity.

[0006] Furthermore, the video extraction module is configured to pre-process the input video and extract video features to establish a training data set, and when the deep learning model is trained according to the training data set to obtain a target model, it includes: The video extraction module converts the video frames of the input video into input format data of the deep learning model, and extracts the video features using the machine learning model according to the input format data. The video features are obtained by the following formula:

[0007]

[0008] in, represents the input format data, Represents the preprocessing operation process, represents the video feature, Represents the feature extraction process of the machine learning model; The video extraction module collects statistics of all video features to establish a video feature data set, extracts video features from the video feature data set, and trains the deep learning model according to the video features to obtain a target model.

[0009] Further, the interactive action module is configured to capture gesture actions, voice commands and touch actions of watching the real-time video image, and calculate the preference value of each mode according to the gesture actions, the voice commands and the touch actions, including: The preference value is derived from the following formula:

[0010]

[0011]

[0012] in, represents the gesture action weight, represents the voice command weight, Touch action weight, represents the weight of the i-th gesture action, represents the weight of the i-th voice command, represents the weight of the i-th touch action, Indicates the frequency of gesture actions, Indicates the frequency of the voice command, Indicates the frequency of touch action, is the gesture action preference value, is the voice command preference value, Preference value for touch action.

[0013] Furthermore, when the interaction strategy is adjusted according to the preference value, it includes: The interactive action module selects an interactive mode according to the preference value; when Respectively greater than and When , the interactive action module selects the gesture action interactive mode; when Respectively greater than and When the interactive action module selects the voice command interactive mode; when Respectively greater than and When , the interactive action module selects the touch action interactive mode; The gesture action interaction mode records the number of gesture action sliding times, and the touch action interaction mode records the number of touch action clicking times.

[0014] Further, when determining whether to adjust the video playback speed according to the interaction strategy, it includes: The interactive action module presets a time period, a first sliding threshold, and a first click threshold; When the number of sliding gestures is greater than the first sliding threshold in the preset time period, the interactive action module plays the real-time video image at 2 times speed or 0.5 times speed; When the number of sliding gestures is less than or equal to the first sliding threshold in the preset time period, the interactive action module plays the real-time video image at 1 times the speed; When the number of clicks of the touch action is greater than the first click threshold in the preset time period, the interactive action module plays the real-time video image at 2 times speed or 0.5 times speed; When the number of touch action clicks in the preset time period is less than or equal to the first click threshold, the interactive action module plays the real-time video image at 1 times the speed.

[0015] Further, the analysis and adjustment module is configured to collect facial images of the person watching the real-time video image and pre-process to obtain facial data, and collect physiological signals and pre-process to obtain physiological signal data, including: The analysis and adjustment module collects the first pixel value of the facial color image and converts it into the grayscale value of the grayscale image, and performs a normalization process on the second pixel value of the grayscale image to obtain a normalized value, wherein the normalized value is in the range of [0, 1]; The gray value is obtained by the following formula:

[0016] in, represents the pixel value of the red channel, represents the pixel value of the green channel, represents the pixel value of the blue channel, Represents grayscale value; The normalized value is obtained by the following formula: ; in, represents the normalized value, represents the minimum pixel value among the second pixel values, represents the maximum pixel value among the second pixel values, represents any second pixel value; The analysis and adjustment module obtains facial data according to the normalized value, filters and standardizes the physiological signal using a bandpass filter to obtain initial signal data, reduces noise on the initial signal data based on wavelet transform to obtain second signal data, and obtains physiological signal data by extracting the second signal data.

[0017] Furthermore, a convolutional network model is established, and when the facial data and the physiological signal data are input into the convolutional network model to obtain a feature vector, the method includes: The analysis and adjustment module substitutes the facial data into the convolutional network model, and the convolutional layer of the convolutional network model uses The convolution kernel extracts the local features of the facial data and uses The activation function performs nonlinear transformation on the local features, and the convolutional network model adopts The pooling layer downsamples the feature map after the nonlinear transformation; gradually adding the convolutional layer and the pooling layer, repeating the steps of extracting local features of the facial data, the nonlinear transformation and the downsampling to obtain a final feature map, flattening the final feature map into a one-dimensional vector and inputting it into a fully connected layer to obtain an initial facial image feature vector; Substitute the physiological signal data into the fully connected layer to obtain an initial physiological signal feature vector.

[0018] Further, the feature vectors are concatenated or weighted to obtain a fused feature vector, and the emotion type is determined according to the fused feature vector, including: The analysis and adjustment module performs standardization processing on the initial facial image feature vector and the initial physiological signal feature vector to obtain a final image feature vector and a final physiological signal feature vector respectively, and the final image feature vector and the final physiological signal feature vector are spliced ​​to obtain the fusion feature vector; The final image feature vector and the final physiological signal feature vector are weighted to obtain the fused feature vector, and the fused feature vector obtained by the weighted processing is obtained by the following formula: ; in, represents the fused feature vector, represents the final image feature vector, represents the final physiological signal feature vector, represents the image weight coefficient, represents the physiological signal weight coefficient, and + =1; The fused feature vector is input into a classifier to obtain a confidence vector, and the emotion type is determined according to the confidence vector.

[0019] Further, the intelligent recommendation module is configured to determine the first recommended video according to the emotion type, determine the second recommended video according to the interaction strategy, and determine the third recommended video based on the machine learning algorithm according to the historical viewing data and the historical interaction behavior, including: The intelligent recommendation module determines a first recommended video according to the reverse type of the emotion type, determines a second recommended video according to the touch action interaction mode, calculates a video preference value according to historical viewing data and historical interaction behaviors, and determines a third recommended video according to the video preference value; The video preference value is obtained by the following formula: ; in, Indicates the video preference value. represents the weight of the i-th historical viewing data, represents the weight of the i-th historical interaction behavior, Represents historical viewing data, Represents historical interactive behavior; when If the preference value is greater than or equal to the preset preference value, it is determined that the third recommended video is recommended; when If the preference value is less than the preset preference value, it is determined that the third recommended video is not recommended.

[0020] Further, comparing the first recommended video, the second recommended video, and the third recommended video with the historical recommended video dataset respectively to determine historical matching similarity, and evaluating the quality of the recommended video according to the historical matching similarity includes: The video evaluation module counts historical recommended videos of the same historical recommended video type in the historical recommended video data set, compares the first recommended video, the second recommended video and the third recommended video with the historical recommended video, respectively, obtains a first historical matching similarity, a second historical matching similarity and a third historical matching similarity, and sets a matching threshold; When the first historical matching similarity is greater than the matching threshold, the quality of the first recommended video is judged to be qualified; if it is less than or equal to the matching threshold, the quality of the first recommended video is judged to be unqualified; when is greater than or equal to the preset preference value, when judging to recommend the third recommended video, if the third historical matching similarity is greater than the matching threshold, the quality of the third recommended video is judged to be qualified; if it is less than or equal to the matching threshold, the quality of the third recommended video is judged to be unqualified; when If the preference value is less than a preset preference value, it is determined that the third recommended video is not recommended, a fourth recommended video is determined according to the emotion type, and the fourth recommended video is compared with the historical recommended video to obtain a fourth historical matching similarity.

[0021] Compared with the prior art, the beneficial effect of the present invention is that by combining the three interaction modes of gesture, voice and touch, the system can dynamically adjust the interaction strategy according to the user's operating habits, thereby improving the interactivity of the system. The personalization and flexibility of interactive video viewing are improved. By integrating the emotion type, interaction strategy, historical viewing data and historical interaction behavior, the accuracy of the system video recommendation is improved. By comparing the historical matching similarity between the recommended video and the historical recommended video, the quality of the recommended video can be evaluated in real time, which not only ensures that the recommended video content meets the user's needs, but also avoids recommending low-quality videos, ensuring that the system recommends videos of good quality every time. According to the user's operating habits, the system can adjust the video playback speed in real time to adapt to their different viewing rhythms. By collecting facial data and physiological signals, the system can accurately identify the emotional state, thereby taking into account emotional fluctuations when recommending videos and providing suitable videos, thereby enhancing the compatibility and flexibility of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings: Figure 1 A schematic diagram of the structure of an intelligent video content analysis and interaction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features described in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0024] See also Figure 1As shown, this embodiment provides an intelligent video content analysis and interaction system, including: a video extraction module, an interactive action module, an analysis and adjustment module, an intelligent recommendation module and a video evaluation module. The video extraction module is configured to pre-process the input video and extract video features to establish a training data set, train the deep learning model according to the training data set to obtain a target model, and output video results for real-time video images based on the target model. The interactive action module is configured to capture gestures, voice commands and touch actions for watching real-time video images, and calculate the preference value of each mode according to the gestures, voice commands and touch actions, adjust the interaction strategy according to the preference value, and determine whether to adjust the video playback speed according to the interaction strategy. The analysis and adjustment module is configured to collect facial expressions of people watching real-time video images. The image is collected and preprocessed to obtain facial data, physiological signals are collected and preprocessed to obtain physiological signal data, a convolutional network model is established, the facial data and the physiological signal data are input into the convolutional network model to obtain a feature vector, the feature vector is spliced ​​or weighted to obtain a fused feature vector, and the emotion type is determined according to the fused feature vector. The intelligent recommendation module is configured to determine a first recommended video according to the emotion type, determine a second recommended video according to the interaction strategy, determine a third recommended video based on the historical viewing data and the historical interactive behavior and based on the machine learning algorithm, and the video evaluation module is configured to collect a historical recommended video data set, compare the first recommended video, the second recommended video and the third recommended video with the historical recommended video data set respectively, determine the historical matching similarity, and evaluate the quality of the recommended video according to the historical matching similarity.

[0025] Specifically, the video extraction module preprocesses the input video, extracts video features from the preprocessed video content and establishes a training data set. The training data set trains the deep learning model to generate a target model. The target model is used to analyze the characters, objects, scenes and emotional states of the characters in the real-time video images to determine the video results. The video results are the video types of the real-time video, such as comedy, horror, adventure and love. The interactive action module uses a gesture recognition algorithm to capture and interpret gestures to respond to gesture instructions, such as: swipe up to view the previous video, swipe down to view the next video. The interactive action module uses natural language processing technology to adapt to different voice patterns and accents. The interactive action module uses perceptual touch technology to sense the strength of the touch screen. When a specific object in the real-time video image is selected by voice and touch, the interactive action module dynamically optimizes the interaction strategy according to the preference value, thereby adjusting the video playback speed. The analysis and adjustment module obtains facial data by collecting facial images in real time, cleaning the facial images, and removing redundant pixels. The analysis and adjustment module collects the user's physiological signals, including heart rate and breathing rate, through sensors, and preprocesses them to remove noise to obtain physiological signal data. The analysis and adjustment module constructs a convolutional network model, extracts features from facial data and physiological signal data to obtain feature vectors, and determines the emotion type, such as pleasure, tension, and concentration, by fusing the feature vectors. The intelligent recommendation module determines the first recommended video based on the emotion type. For example, when the current emotion type is judged to be tension, the first recommended video is a soothing video. When the current emotion type is judged to be depression, the first recommended video is a cheerful video to relieve depression. The intelligent recommendation module determines the second recommended video based on the interaction strategy. The second recommended video is an associated video of an object in the real-time video image in the interactive mode. The intelligent recommendation module intelligently recommends the third recommended video based on historical viewing data and historical interactive behaviors. The video evaluation module summarizes all historical recommended videos to establish a historical recommended video dataset, and evaluates whether the quality of the recommended video is qualified based on the historical matching similarity.

[0026] It is understandable that the various modules work together to improve the intelligence of the system and enhance the integration of sentiment analysis and intelligent recommendation, so that the system can not only accurately analyze the video type, but also sense the emotional state in real time to determine the first recommended video. Secondly, the second recommended video is determined based on the interaction strategy, which enhances the flexibility and compatibility of the system. The third recommended video is determined based on the machine learning algorithm, which improves the scientificity and rationality of the system's recommended videos.

[0027] In some embodiments of the present application, the video extraction module is configured to preprocess the input video and extract video features to establish a training data set. When the deep learning model is trained according to the training data set to obtain a target model, the video extraction module converts the video frames of the input video into input format data of the deep learning model, and extracts video features using a machine learning model according to the input format. The video features are obtained by the following formula:

[0028]

[0029] in, Indicates input format data, Represents the preprocessing operation process, Represents video features, It represents the feature extraction process of the machine learning model. The video extraction module counts all video features to establish a video feature data set. The video extraction module extracts video features from the video feature data set and trains the deep learning model according to the video features to obtain the target model.

[0030] It can be understood that the video frames of the input video are preprocessed, including denoising, scaling and cropping operations, to ensure the accuracy of the input format data. The machine learning model used is a time series model, and the input format data is input into the time series model. The temporal features are extracted, and the temporal features are video features. The deep learning model is trained by video features to ensure the target model's ability to process video features, so that the target model can recognize people, objects and scenes in real-time video images, ensuring the accuracy of the video results obtained by the system.

[0031] In some embodiments of the present application, the interactive action module is configured to capture gesture actions, voice commands, and touch actions for watching real-time video images, and calculate the preference values ​​of each mode according to the gesture actions, voice commands, and touch actions, including: the preference value is obtained by the following formula:

[0032]

[0033]

[0034] in, represents the gesture action weight, represents the voice command weight, Touch action weight, represents the weight of the i-th gesture action, represents the weight of the i-th voice command, represents the weight of the i-th touch action, Indicates the frequency of gesture actions, Indicates the frequency of the voice command, Indicates the frequency of touch action, is the gesture action preference value, is the voice command preference value, Preference value for touch action.

[0035] It can be understood that the gesture action preference value reflects the degree of dependence on gesture interaction, the voice command preference value reflects the degree of dependence on voice interaction, and the touch action preference value reflects the degree of dependence on touch action. By calculating the preference value of each mode, the system can quantify the preference degree of each mode interaction, thereby improving the flexibility of the system.

[0036] In some embodiments of the present application, when adjusting the interaction strategy according to the preference value, the interactive action module selects the interaction mode according to the preference value. Respectively greater than and When , the interactive action module selects the gesture action interaction mode. Respectively greater than and When , the interactive action module selects the voice command interactive mode. Respectively greater than and When the touch action interaction mode is selected, the interactive action module records the number of gesture action slides in the gesture action interaction mode, and the touch action interaction mode records the number of touch action clicks.

[0037] Specifically, by comparing , and , the system dynamically adjusts its own interaction mode. The interactive action module improves the gesture interaction sensitivity according to the gesture interaction mode, and establishes gesture interaction tutorials and prompts to improve the system's compatibility with gestures. The interactive action module adjusts the voice command options according to the voice command interaction mode, improves the system's voice recognition response speed and voice recognition sensitivity, so that the system can adapt to voice commands with different accents to fast forward, rewind, view the previous video, and view the next video on real-time video images. The interactive action module improves the system's touch response speed and touch sensitivity according to the touch action interaction mode, and establishes touch input shortcuts and touch customization options.

[0038] It can be understood that by selecting different interaction modes, the flexibility and compatibility of the system are improved, the sensitivity is improved accordingly for different interaction modes, the waste of resources caused by the system processing too many data instructions is avoided, and the response speed of the system is enhanced.

[0039] In some embodiments of the present application, when determining whether to adjust the video playback speed based on the interaction strategy, it includes: a preset time period for the interactive action module, a first sliding threshold, and a first click threshold. When the number of gesture action sliding times is greater than the first sliding threshold in the preset time period, the interactive action module plays the real-time video image at 2 times the speed or 0.5 times the speed. When the number of gesture action sliding times is less than or equal to the first sliding threshold in the preset time period, the interactive action module plays the real-time video image at 1 times the speed. When the number of touch action clicks is greater than the first click threshold in the preset time period, the interactive action module plays the real-time video image at 2 times the speed or 0.5 times the speed. When the number of touch action clicks is less than or equal to the first click threshold in the preset time period, the interactive action module plays the real-time video image at 1 times the speed.

[0040] Specifically, the preset time period is 60 seconds, the first sliding threshold is 5 times, and the first click threshold is 5 times. The preset time period, the first sliding threshold, and the first click threshold can be adaptively adjusted. Within 60 seconds, when the number of gesture sliding times exceeds 5 times, the real-time video image is quickly browsed or slowly viewed according to the sliding direction. For example, when the real-time video image slides forward more than 5 times, the real-time video image is adjusted to 2 times the speed, and when the real-time video image slides backward more than 5 times, the real-time video image is adjusted to 0.5 times the speed. If the number of gesture sliding times is less than or equal to 5 times, the real-time video image is played normally at 1 times the speed. The number of touch action clicks is implemented in the same way as the number of gesture sliding times, which will not be repeated here.

[0041] It is understandable that by dynamically adjusting the playback speed of real-time video images, the requirements for fast browsing of real-time video images or slow viewing of real-time video image details are met, thereby improving the flexibility and adaptability of the system.

[0042] In some embodiments of the present application, the analysis and adjustment module is configured to collect facial images of real-time video images and pre-process them to obtain facial data, and collect physiological signals and pre-process them to obtain physiological signal data, including: the analysis and adjustment module collects the first pixel value of the facial color image and converts it into the grayscale value of the grayscale image, and normalizes the second pixel value of the grayscale image to obtain a normalized value, the normalized value is [0,1], and the grayscale value is obtained by the following formula:

[0043] in, represents the pixel value of the red channel, represents the pixel value of the green channel, represents the pixel value of the blue channel, Represents the grayscale value, and the normalized value is obtained by the following formula: ; in, represents the normalized value, represents the minimum pixel value among the second pixel values, represents the maximum pixel value among the second pixel values, Represents any second pixel value, the analysis and adjustment module obtains facial data according to the normalized value, uses a bandpass filter to filter and standardize the physiological signal to obtain initial signal data, reduces noise on the initial signal data based on wavelet transform to obtain second signal data, and obtains physiological signal data by extracting the second signal data.

[0044] Specifically, the analysis and adjustment module converts the color image into a grayscale image, and the grayscale processing reduces the computational complexity of the system. By calculating the grayscale value, the accuracy of the system in reflecting the pixel brightness is improved, and a unified brightness feature is provided for the subsequent normalization processing. By mapping the second pixel value of the grayscale image to the range of [0,1], the planning value is made consistent, while retaining the characteristics of the facial color image, reducing the impact of illumination changes or contrast differences on feature extraction, and improving the accuracy of the system analysis. In addition, the planning value in the range of [0,1] helps the subsequent machine learning model to deeply learn its features, reducing the convergence difficulties caused by differences in the distribution of planning values, and providing reliable data for the subsequent establishment of a convolutional network model. The analysis and adjustment module generates multiple variant images for the pixel points corresponding to the normalized value through rotation, translation, scaling, and flipping operations to obtain facial data, which improves the diversity of the training data of the subsequent machine learning model, enables the machine learning model to adapt to different facial data, reduces the dependence on specific data distribution, improves the generalization ability of the machine learning model, and enhances the adaptability of the system to the machine learning model.

[0045] It can be understood that the analysis and adjustment module uses a bandpass filter to filter the physiological signal to remove noise and interference. The transfer function of the bandpass filter is expressed as: ; in, represents the transfer function of the bandpass filter, and are the low frequency cutoff frequency and high frequency cutoff frequency of the bandpass filter, respectively. Indicates the frequency of physiological signals and performs data standardization on the frequency of physiological signals. Data standardization adopts The standardization algorithm standardizes the frequency of physiological signals to obtain initial signal data to eliminate the amplitude differences between different signals. The analysis and adjustment module then performs signal noise reduction based on wavelet transform. Wavelet transform can effectively separate the noise component and useful signal component in the signal to obtain the second signal data. The analysis and adjustment module obtains the physiological signal data by extracting the time domain characteristics and frequency domain characteristics of the second signal data, which helps the system to accurately capture the emotion type.

[0046] In some embodiments of the present application, a convolutional network model is established, and when the facial data and the physiological signal data are input into the convolutional network model to obtain a feature vector, the analysis and adjustment module substitutes the facial data into the convolutional network model, and the convolutional layer of the convolutional network model uses The convolution kernel extracts the local features of the facial data and uses The activation function performs nonlinear transformation on local features. The convolutional network model uses The pooling layer downsamples the feature map after nonlinear transformation, gradually adds convolutional layers and pooling layers, repeats the steps of extracting local features of facial data, nonlinear transformation and downsampling to obtain the final feature map, flattens the final feature map into a one-dimensional vector and inputs it into the fully connected layer to obtain the initial facial image feature vector, and substitutes the physiological signal data into the fully connected layer to obtain the initial physiological signal feature vector.

[0047] It is understandable that the convolutional layer passes The convolution kernel extracts local features of facial data and retains facial correlation, such as facial texture and contour. The activation function performs nonlinear mapping on local features, which enhances the system's ability to process local features. The pooling layer downsamples the feature map after nonlinear transformation, reducing the dimension of the feature map. The stacking of multiple layers of convolution and pooling is gradually increased to ensure the accuracy of local feature expression. Finally, the feature map and physiological signal data are input to the fully connected layer, which enables the system to improve the complementarity of different modal information, thereby improving the accuracy of the system's recognition of emotion types. Establishing a convolutional network model can reduce errors caused by manual processing, effectively process facial data and physiological signal data, and improve the system's response speed in identifying emotion types.

[0048] In some embodiments of the present application, the feature vectors are spliced ​​or weighted to obtain a fused feature vector, and when the emotion type is determined according to the fused feature vector, the analysis and adjustment module standardizes the initial facial image feature vector and the initial physiological signal feature vector to obtain the final image feature vector and the final physiological signal feature vector, respectively, the final image feature vector and the final physiological signal feature vector are spliced ​​to obtain a fused feature vector, the final image feature vector and the final physiological signal feature vector are weighted to obtain a fused feature vector, and the fused feature vector obtained by the weighted processing is obtained by the following formula: ; in, represents the fused feature vector, represents the final image feature vector, represents the final physiological signal feature vector, represents the image weight coefficient, represents the physiological signal weight coefficient, and + =1, the fused feature vector is input into the classifier to obtain the confidence vector, and the emotion type is determined according to the confidence vector.

[0049] Understandably, standardization takes The standardization algorithm standardizes the initial facial image feature vector and the initial physiological signal feature vector to ensure that the final image feature vector and the final physiological signal feature vector have the same vector scale. The analysis and adjustment module uses a one-dimensional splicing method to splice the final image feature vector and the final physiological signal feature vector to obtain a fused feature vector. Another weighted method can also be used to obtain a fused feature vector. + =1, the two weight coefficients can be adaptively adjusted. The analysis and adjustment module inputs the fused feature vector into the classifier to obtain the confidence vector, and determines the emotion type according to the maximum value of the confidence vector. The cheerfulness of the emotion type decreases as the value decreases. For example, if the maximum value of the confidence vector is 0.8, the emotion type 0.8 is set as cheerfulness, 0.7 is set as hesitation, and 0.6 is set as sadness. By quantifying the maximum value of the confidence vector, the system can determine the corresponding emotion type according to the maximum value, which is conducive to the interactive system to distinguish the emotional state and improve the system's ability to process the emotional state.

[0050] In some embodiments of the present application, the intelligent recommendation module is configured to determine the first recommended video according to the emotion type, determine the second recommended video according to the interaction strategy, and determine the third recommended video based on the historical viewing data and the historical interaction behavior based on the machine learning algorithm, including: the intelligent recommendation module determines the first recommended video according to the reverse type of the emotion type, determines the second recommended video according to the touch action interaction mode, calculates the video preference value according to the historical viewing data and the historical interaction behavior, and determines the third recommended video according to the video preference value, and the video preference value is obtained by the following formula: ; in, Indicates the video preference value. represents the weight of the i-th historical viewing data, represents the weight of the i-th historical interaction behavior, Represents historical viewing data, Represents historical interaction behavior, when is greater than or equal to the preset preference value, then the third recommended video is recommended. If the preference value is less than the preset preference value, it is determined that the third recommended video is not recommended.

[0051] It can be understood that the first recommended video is a video of the opposite type of the emotion type. For example, when the emotion type is judged to be sad, the first recommended video is a cheerful video. When the emotion type is judged to be nervous, the first recommended video is a video that soothes the emotion. When the emotion type is joyful and serious, the intelligent recommendation module denies the first recommended video. In the touch action interaction mode, if an object in the real-time video image is touched, a second recommended video related to the object is provided to fully understand the information of the object. The preset preference value is 0.7. When the video preference value exceeds 0.7, the third recommended video is recommended, otherwise the third recommended video is not recommended. The third recommended video is an associated video of the same type of preference. The historical viewing data consists of historical video results, historical viewing time and historical video scores. The historical interactive behavior consists of pausing, replaying, skipping, liking and commenting on historical videos. Through the first recommended video, the second recommended video and the third recommended video, the system can intelligently recommend associated videos, reduce the blindness of the system, and enhance the scientificity and rationality of the system's recommendation of associated videos.

[0052] In some embodiments of the present application, the first recommended video, the second recommended video, and the third recommended video are respectively compared with the historical recommended video data set to determine the historical matching similarity. When evaluating the quality of the recommended video according to the historical matching similarity, it includes: the video evaluation module counts the historical recommended videos of the same historical recommended video type in the historical recommended video data set, and compares the first recommended video, the second recommended video, and the third recommended video with the historical recommended video to obtain the first historical matching similarity, the second historical matching similarity, and the third historical matching similarity, respectively, and sets a matching threshold. When the first historical matching similarity is greater than the matching threshold, the quality of the first recommended video is judged to be qualified; if it is less than or equal to the matching threshold, the quality of the first recommended video is judged to be unqualified; when If the third historical matching similarity is greater than or equal to the preset preference value, then the third recommended video is recommended. If the third historical matching similarity is greater than the matching threshold, then the quality of the third recommended video is judged to be qualified. If it is less than or equal to the matching threshold, then the quality of the third recommended video is judged to be unqualified. If the preference value is less than the preset preference value, it is determined that the third recommended video is not recommended, and a fourth recommended video is determined according to the emotion type. The fourth recommended video is compared with the historical recommended videos to obtain a fourth historical matching similarity.

[0053] Specifically, the video evaluation module counts the same video types in the historical recommended video data set, collects historical recommended videos of the same type as the first recommended video, collects historical recommended videos of the same type as the second recommended video, and collects historical recommended videos of the same type as the third recommended video. The first recommended video, the second recommended video, and the third recommended video are compared with the historical recommended videos respectively. The comparison includes: video subtitle content, emotional state of video characters, background music of video content, and color of video environment, and the first historical matching similarity, the second historical matching similarity, and the third historical matching similarity are obtained. The matching threshold is set to 0.6. When the first historical matching similarity is greater than 0.6, the first recommended video is judged to be of qualified quality and can be recommended to the user. Otherwise, it is unqualified and cannot be recommended. The second historical matching similarity is processed in the same way as the first historical matching similarity, which will not be repeated here. When If the third recommended video is greater than or equal to the preset preference value, the third recommended video is selected for recommendation and the third historical matching similarity is obtained. When the third historical matching similarity is greater than 0.6, the third recommended video is judged to be of qualified quality and can be recommended to the user. Otherwise, it is unqualified and cannot be recommended. If the preference value is less than the preset preference value, it is determined that the third recommended video is not recommended. At this time, the fourth recommended video is determined according to the emotion type. The fourth recommended video and the first recommended video are processed in the same way, both are reverse type videos of the emotion type. The fourth recommended video is compared with the historical recommended video, and the comparison is consistent with the first recommended video, the second recommended video and the third recommended video are compared with the historical recommended videos respectively. The explanation is not repeated here. The fourth historical matching similarity is compared with the matching threshold. When the fourth matching similarity is greater than 0.6, the fourth recommended video is judged to be of qualified quality and can be recommended to the user. Otherwise, it is unqualified and cannot be recommended.

[0054] It is understandable that the video evaluation module will also detect the learning progress of the teaching video in real time, including the time point of watching the teaching video and the number of times the teaching video is repeatedly watched, and interactive questions and tests are raised at different time points to evaluate the learning and comprehension ability. The video evaluation module adjusts the learning difficulty according to the evaluation of the learning and comprehension ability. For example, when a certain part of the teaching content is well mastered, the difficulty of the teaching video is increased. On the contrary, if there is difficulty in learning and understanding, basic teaching videos and additional explanation videos are provided. The video evaluation module can also detect the eye movement metric based on the sensor. The video evaluation module calculates the engagement index based on the viewing time of the advertisement and the eye movement metric. The engagement index is equal to the product of the eye movement metric and the viewing time of the advertisement. The engagement index indicates the degree of participation in the advertisement. When the engagement index is lower than the threshold value of 0.5 set by the system, the video evaluation module generates an advertisement feedback report to adjust the content of the advertisement. Through a series of evaluations and judgments, the system can screen out high-quality videos for intelligent recommendation, thereby improving the accuracy of the recommended videos and effectively ensuring the quality of the recommended videos.

[0055] In summary, the beneficial effects of the present invention are: by combining the three interaction modes of gesture, voice and touch, the system can dynamically adjust the interaction strategy according to the user's operating habits, thereby improving the interactivity of the system. The personalization and flexibility of interactive video viewing are improved, and the accuracy of the system video recommendation is improved by integrating the emotional type, interaction strategy, historical viewing data and historical interaction behavior. By comparing the historical matching similarity obtained by comparing the recommended video with the historical recommended video, the quality of the recommended video can be evaluated in real time, which not only ensures that the recommended video content meets the user's needs, but also avoids recommending low-quality videos, ensuring that the system recommends videos of good quality every time. According to the user's operating habits, the system can adjust the video playback speed in real time to adapt to their different viewing rhythms. By collecting facial data and physiological signals, the system can accurately identify the emotional state, thereby taking into account emotional fluctuations when recommending videos and providing suitable videos, thereby enhancing the compatibility and flexibility of the system.

[0056] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0057] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0058] These computer program instructions may also be stored in a computer-readable storage device that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable storage device produce a product including an instruction device that implements the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0059] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An intelligent video content analysis and interaction system, characterized in that: include: Video extraction module, interactive action module, analysis and adjustment module, intelligent recommendation module and video evaluation module; The video extraction module is configured to pre-process the input video and extract video features to establish a training data set, train the deep learning model according to the training data set to obtain a target model, and output a video result for the real-time video image based on the target model; The interactive action module is configured to capture gesture actions, voice commands and touch actions of watching the real-time video image, and calculate preference values ​​of each mode according to the gesture actions, the voice commands and the touch actions, adjust the interactive strategy according to the preference values, and determine whether to adjust the video playback speed according to the interactive strategy; The analysis and adjustment module is configured to collect facial images of a person watching the real-time video image and pre-process them to obtain facial data, collect physiological signals and pre-process them to obtain physiological signal data, establish a convolutional network model, input the facial data and the physiological signal data into the convolutional network model to obtain a feature vector, perform splicing or weighted processing on the feature vector to obtain a fused feature vector, and determine the emotion type according to the fused feature vector; The intelligent recommendation module is configured to determine a first recommended video according to the emotion type, determine a second recommended video according to the interaction strategy, and determine a third recommended video based on a machine learning algorithm according to historical viewing data and historical interaction behaviors; The video evaluation module is configured to collect a historical recommended video data set, compare the first recommended video, the second recommended video and the third recommended video with the historical recommended video data set respectively, determine the historical matching similarity, and evaluate the recommended video quality according to the historical matching similarity.

2. The intelligent video content analysis and interaction system according to claim 1, characterized in that: The video extraction module is configured to pre-process the input video and extract video features to establish a training data set, and when the deep learning model is trained according to the training data set to obtain a target model, it includes: The video extraction module converts the video frames of the input video into input format data of the deep learning model, and extracts the video features using the machine learning model according to the input format data. The video features are obtained by the following formula: in, represents the input format data, Represents the preprocessing operation process, represents the video feature, Represents the feature extraction process of the machine learning model; The video extraction module collects statistics of all video features to establish a video feature data set, extracts video features from the video feature data set, and trains the deep learning model according to the video features to obtain a target model.

3. The intelligent video content analysis and interaction system according to claim 1, characterized in that: When the interactive action module is configured to capture gesture actions, voice commands, and touch actions of viewing the real-time video image, and calculate the preference values ​​of each mode according to the gesture actions, the voice commands, and the touch actions, the interactive action module includes: The preference value is derived from the following formula: in, represents the gesture action weight, represents the voice command weight, Touch action weight, represents the weight of the i-th gesture action, represents the weight of the i-th voice command, represents the weight of the i-th touch action, Indicates the frequency of gesture actions, Indicates the frequency of the voice command, Indicates the frequency of touch action, is the gesture action preference value, is the voice command preference value, Preference value for touch action.

4. The intelligent video content analysis and interaction system according to claim 3, characterized in that: When adjusting the interaction strategy according to the preference value, it includes: The interactive action module selects an interactive mode according to the preference value; when Respectively greater than and When , the interactive action module selects the gesture action interactive mode; when Respectively greater than and When the interactive action module selects the voice command interactive mode; when Respectively greater than and When , the interactive action module selects the touch action interactive mode; The gesture action interaction mode records the number of gesture action sliding times, and the touch action interaction mode records the number of touch action clicking times.

5. The intelligent video content analysis and interaction system according to claim 4, characterized in that: When determining whether to adjust the video playback speed according to the interaction strategy, it includes: The interactive action module presets a time period, a first sliding threshold, and a first click threshold; When the number of sliding gestures is greater than the first sliding threshold in the preset time period, the interactive action module plays the real-time video image at 2 times speed or 0.5 times speed; When the number of sliding gestures is less than or equal to the first sliding threshold in the preset time period, the interactive action module plays the real-time video image at 1 times the speed; When the number of clicks of the touch action is greater than the first click threshold in the preset time period, the interactive action module plays the real-time video image at 2 times speed or 0.5 times speed; When the number of touch action clicks in the preset time period is less than or equal to the first click threshold, the interactive action module plays the real-time video image at 1 times the speed.

6. The intelligent video content analysis and interaction system according to claim 5, characterized in that: The analysis and adjustment module is configured to collect facial images of the user watching the real-time video image and pre-process the facial data, and collect physiological signals and pre-process the physiological signal data, including: The analysis and adjustment module collects the first pixel value of the facial color image and converts it into the grayscale value of the grayscale image, and performs a normalization process on the second pixel value of the grayscale image to obtain a normalized value, wherein the normalized value is in the range of [0, 1]; The gray value is obtained by the following formula: in, represents the pixel value of the red channel, represents the pixel value of the green channel, represents the pixel value of the blue channel, Represents grayscale value; The normalized value is obtained by the following formula: ； in, represents the normalized value, represents the minimum pixel value among the second pixel values, represents the maximum pixel value among the second pixel values, represents any second pixel value; The analysis and adjustment module obtains facial data according to the normalized value, filters and standardizes the physiological signal using a bandpass filter to obtain initial signal data, reduces noise on the initial signal data based on wavelet transform to obtain second signal data, and obtains physiological signal data by extracting the second signal data.

7. The intelligent video content analysis and interaction system according to claim 6, characterized in that: Establishing a convolutional network model, and inputting the facial data and the physiological signal data into the convolutional network model to obtain a feature vector, comprising: The analysis and adjustment module substitutes the facial data into the convolutional network model, and the convolutional layer of the convolutional network model uses The convolution kernel extracts the local features of the facial data and uses The activation function performs nonlinear transformation on the local features, and the convolutional network model adopts The pooling layer downsamples the feature map after the nonlinear transformation; gradually adding the convolutional layer and the pooling layer, repeating the steps of extracting local features of the facial data, the nonlinear transformation and the downsampling to obtain a final feature map, flattening the final feature map into a one-dimensional vector and inputting it into a fully connected layer to obtain an initial facial image feature vector; Substitute the physiological signal data into the fully connected layer to obtain an initial physiological signal feature vector.

8. The intelligent video content analysis and interaction system according to claim 7, characterized in that: The feature vectors are concatenated or weighted to obtain a fused feature vector, and the emotion type is determined according to the fused feature vector, including: The analysis and adjustment module performs standardization processing on the initial facial image feature vector and the initial physiological signal feature vector to obtain a final image feature vector and a final physiological signal feature vector respectively, and the final image feature vector and the final physiological signal feature vector are spliced ​​to obtain the fusion feature vector; The final image feature vector and the final physiological signal feature vector are weighted to obtain the fused feature vector, and the fused feature vector obtained by the weighted processing is obtained by the following formula: in, represents the fused feature vector, represents the final image feature vector, represents the final physiological signal feature vector, represents the image weight coefficient, represents the physiological signal weight coefficient, and + =1; The fused feature vector is input into a classifier to obtain a confidence vector, and the emotion type is determined according to the confidence vector.

9. The intelligent video content analysis and interaction system according to claim 8, characterized in that: The intelligent recommendation module is configured to determine a first recommended video according to the emotion type, determine a second recommended video according to the interaction strategy, and determine a third recommended video based on a machine learning algorithm according to historical viewing data and historical interaction behaviors, including: The intelligent recommendation module determines a first recommended video according to the reverse type of the emotion type, determines a second recommended video according to the touch action interaction mode, calculates a video preference value according to historical viewing data and historical interaction behaviors, and determines a third recommended video according to the video preference value; The video preference value is obtained by the following formula: ； in, Indicates the video preference value. represents the weight of the i-th historical viewing data, represents the weight of the i-th historical interaction behavior, Represents historical viewing data, Represents historical interactive behavior; when If the preference value is greater than or equal to the preset preference value, it is determined that the third recommended video is recommended; when If the preference value is less than the preset preference value, it is determined that the third recommended video is not recommended.

10. The intelligent video content analysis and interaction system according to claim 9, characterized in that: The first recommended video, the second recommended video, and the third recommended video are respectively compared with the historical recommended video dataset to determine historical matching similarities, and the quality of the recommended videos is evaluated according to the historical matching similarities, including: The video evaluation module counts historical recommended videos of the same historical recommended video type in the historical recommended video data set, compares the first recommended video, the second recommended video and the third recommended video with the historical recommended video, respectively, obtains a first historical matching similarity, a second historical matching similarity and a third historical matching similarity, and sets a matching threshold; When the first historical matching similarity is greater than the matching threshold, the quality of the first recommended video is judged to be qualified; if it is less than or equal to the matching threshold, the quality of the first recommended video is judged to be unqualified; when is greater than or equal to the preset preference value, when judging to recommend the third recommended video, if the third historical matching similarity is greater than the matching threshold, the quality of the third recommended video is judged to be qualified; if it is less than or equal to the matching threshold, the quality of the third recommended video is judged to be unqualified; when If the preference value is less than a preset preference value, it is determined that the third recommended video is not recommended, a fourth recommended video is determined according to the emotion type, and the fourth recommended video is compared with the historical recommended video to obtain a fourth historical matching similarity.

Images