Video data processing method, device, computer equipment and storage medium

Abstract

The present application relates to a video data processing method, device, computer equipment, storage medium and computer program product. The method includes: determining multiple time points of the video to be processed, determining the multimodal data corresponding to each time point, the multimodal data at least including video images, dialogue audio and dialogue text; for each time point, performing emotion recognition based on the video image and dialogue audio of the time point, and obtaining the emotion recognition result; determining the suspense score of the time point according to the video image, dialogue text and emotion recognition result; screening the candidate time points according to the suspense score of each time point; performing integrity analysis based on the multimodal data of each candidate time point, and determining the target time point from the candidate time points according to the integrity analysis result. The present method can improve the video editing and plot annotation effects by accurately determining the target time point used to determine the end point of the video.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video data processing method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the development of computer technology, watching videos has become a common form of entertainment in people's daily lives. For videos with long duration, users may not be interested in all the content in the video and may want to watch selectively. In order to meet the needs of different users, videos can be edited to obtain shorter and more exciting video content clips.

[0003] In the process of segmenting and editing a video, a large amount of manual work is usually used to mark the time position of the content in the entire video that is suitable as the end point of the material, so as to provide it to other landing applications for selection and use in video editing and segmentation.

[0004] However, manual labeling consumes a lot of manpower and has a very slow labeling speed, resulting in low labeling efficiency. Summary of the invention

[0005] Based on this, it is necessary to provide a video data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve the efficiency of determining the target time point in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for processing video data. The method comprises:

[0007] Determine multiple time points of the video to be processed, and obtain multimodal data corresponding to each time point, wherein the multimodal data includes at least video images, dialogue audio, and dialogue text;

[0008] For any time point, emotion recognition is performed based on the video image and the audio of the lines at the time point to obtain the emotion recognition result;

[0009] Determine the suspense score of the targeted time point according to the video images, dialogue texts and emotion recognition results of the targeted time point;

[0010] According to the suspense score of each time point, candidate time points are screened from multiple time points;

[0011] An integrity analysis is performed on the multimodal data of each candidate time point, and a target time point is determined from the candidate time points based on the integrity analysis results; the target time point is used to determine the end point of the video.

[0012] In a second aspect, the present application also provides a video data processing device. The device comprises:

[0013] A data determination module, used to determine multiple time points of the video to be processed, and obtain multimodal data corresponding to each time point, wherein the multimodal data at least includes video images, dialogue audio and dialogue text;

[0014] The recognition module is used to perform emotion recognition at any time point based on the video image and the line audio at the time point to obtain the emotion recognition result;

[0015] A calculation module, used to determine the suspense score of the targeted time point according to the video picture, the line text and the emotion recognition result of the targeted time point;

[0016] A screening module, used to screen candidate time points from multiple time points according to the suspense score of each time point;

[0017] The positioning module is used to perform integrity analysis based on the multimodal data of each candidate time point, and determine the target time point from the candidate time points based on the integrity analysis results; the target time point is used to determine the end point of the video.

[0018] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned video data processing method when executing the computer program.

[0019] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the above-mentioned video data processing method are implemented.

[0020] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned video data processing method are implemented.

[0021] The above-mentioned video data processing method, device, computer equipment, storage medium and computer program product determine multiple time points of the video to be processed, and obtain multimodal data corresponding to each time point, wherein the multimodal data at least includes video screen, line audio and line text; further, emotion recognition is performed according to the video screen and line audio of the time point, and the emotion recognition result is obtained, so as to determine the suspense score of the time point according to the video screen, line text and emotion recognition result of the time point, that is, when calculating the suspense score of the time point, the application integrates the data of multiple modes such as video screen, line text and emotion recognition result, which can not only ensure the accuracy of the suspense score, but also has a certain robustness. Further, according to the suspense score of each time point, the candidate time point with suspense is screened out. Finally, the integrity analysis is performed according to the multimodal data of each candidate time point, and the target time point is determined again according to the integrity analysis result, wherein the target time point is used to determine the end point of the video. The suspense score is calculated by using the multimodal data in this application, and the candidate time points are screened according to the suspense score. The candidate time points are further analyzed for integrity, and the results of the integrity analysis are combined for screening again, so that the content at the end of the video has a certain logical integrity and suspense, completely getting rid of human involvement, and greatly improving the editing efficiency. At the same time, because the positioning of the target time point in this application is a standardized operation, it will not be differentiated due to different personnel, which ensures the accuracy of the prediction of the video end point. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A diagram showing an application environment of a video data processing method in an embodiment;

[0023] Figure 2 is a schematic flow chart of a video data processing method in one embodiment;

[0024] Figure 3 A flowchart of facial emotion recognition in one embodiment;

[0025] Figure 4 A structural block diagram of a network structure in an embodiment;

[0026] Figure 5 A schematic diagram of the effect of face detection in one embodiment;

[0027] Figure 6 is a stage structure diagram of a network structure in one embodiment;

[0028] Figure 7 is a structural block diagram of a network structure in another embodiment;

[0029] Figure 8A structural block diagram of a network structure in yet another embodiment;

[0030] Fig. 9 A schematic diagram of the structure of a speech emotion recognition model in one embodiment;

[0031] Fig.10 This is a diagram showing the trend of bullet screen data in an embodiment;

[0032] Fig.11 is a structural block diagram of a feature weighting process in one embodiment;

[0033] Fig.12 is an overall system architecture diagram in one embodiment;

[0034] Fig.13 In one embodiment, a target time point is determined;

[0035] Fig.14 is an overall system architecture diagram in another embodiment;

[0036] Fig.15 is a structural block diagram of a video processing device in one embodiment;

[0037] Fig.16 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0039] The video data processing method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, can be integrated on the server 104, or can be placed on the cloud or other network servers. The terminal 102 and the server 104 can be used alone to execute the video data processing method in this application, and the terminal 102 and the server 104 can be used together to execute the video data processing method in this application.

[0040] Taking the terminal 102 and the server 104 collaboratively executing this application as an example, when processing video data, the user can send the video to be processed to the server 104 through the terminal 102 based on his or her own video processing needs, such as video editing needs, video plot annotation needs, etc. The server 104 can obtain the video to be processed sent by the terminal 102, determine multiple time points of the video to be processed, and obtain multimodal data corresponding to each time point, wherein the multimodal data at least includes video screen, line audio and line text; the server 104 performs emotion recognition according to the video screen and line audio of the time point at any time point, and obtains the emotion recognition result; the server 104 determines the suspense score of the time point at the time point according to the video screen, line text and emotion recognition result of the time point at the time point; the server 104 selects candidate time points from the multiple time points according to the suspense score of each time point; the server 104 performs integrity analysis according to the multimodal data of each candidate time point, and determines the target time point from the candidate time point according to the integrity analysis result; the target time point is used to determine the end point of the video. The server 104 can feed back the target time point to the terminal 102, so that the user can perform related operations on the video based on the terminal 102, such as video editing, video plot annotation, etc.

[0041] Among them, the terminal 102 runs applications, such as video applications, video editing applications, live broadcast applications, social applications, educational applications, shopping applications and other applications with video data processing requirements. The terminal 102 can be, but is not limited to, various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices, portable wearable devices, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 104 can be implemented with an independent server or a server cluster composed of multiple servers. The server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication.

[0042] It should be noted that the embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc.

[0043] Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0044] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0045] The following is a detailed introduction to the video data processing method of this application:

[0046] In one embodiment, Figure 2 As shown, a video data processing method is provided, and the method is applied to a computer device (the computer device can be specifically Figure 1 The terminal 102 or the server 104 in the example is used to illustrate that the video data processing method includes the following steps:

[0047] Step 202, determine multiple time points of the video to be processed, and obtain multimodal data corresponding to each time point, wherein the multimodal data at least includes video images, dialogue audio, and dialogue text.

[0048] The video to be processed is a video that needs to be processed. For example, the video to be processed may be a video of a film or TV series, a video of a variety show, or a video generated by a user.

[0049] The time point is the time point at which the video content appears in the video progress bar. For a video, when describing the playback position of the video through the time point, the measurement unit of the time point can be multiple, such as hours (H), seconds (S) or minutes (min), etc. The specific measurement unit can be set in combination with the length of the video and the content of the video.

[0050] The multimodal data of the video to be processed refers to data in different forms that can reflect the characteristics of the video to be processed, and can usually include video images, dialogue audio, and dialogue text.

[0051] Video images are data reflecting video features in the form of video frames.

[0052] The dialogue audio is data reflecting the video features in the form of audio. The dialogue audio in this application can specifically be sound data that matches the video screen in the video to be processed.

[0053] The dialogue text is data reflecting the video features in the form of text. The dialogue text in this application can specifically be a text paragraph that matches the video screen and dialogue audio in the video to be processed.

[0054] In some embodiments, for the acquired video to be processed, the measurement unit of the time point is seconds. The multiple time points of the video to be processed determined by the computer device may include 1S, 2S, 3S, etc. Among them, when the time point is 1S, there is a corresponding video screen, line audio and line text; when the time point is 2S, there is a corresponding video screen, line audio and line text... It can be understood that the measurement unit of the time point can also be hours, minutes, etc. Correspondingly, the multimodal data obtained by the computer device is the multimodal data corresponding to the time point of x hours and the multimodal data corresponding to x minutes.

[0055] In some embodiments, when determining the multimodal data corresponding to each time point, the computer device can take the time point as the central time point, take a certain time period forward to form a time interval, take a certain time period backward to form a time interval, or take a certain time period forward and backward to form a time interval, and then obtain the multimodal data within the time interval.

[0056] In some embodiments, when the computer device takes a certain time period forward or backward to form a time interval with any time point as the center time point, the time period can be taken in seconds or minutes. For example, when taking values ​​in seconds, the time period taken can be 2S, 3S, etc. When taking values ​​in minutes, it can also be 1min, 2min, etc. The specific time period taken forward or backward can be adaptively adjusted in combination with the actual video length, video processing requirements, etc.

[0057] In some embodiments, when obtaining multimodal data according to a time interval, the computer device can obtain the dialogue segment in this time interval as dialogue text, obtain the voice segment in this time interval as dialogue audio, and obtain multiple video frames in this time interval as video pictures.

[0058] In other embodiments, when the number of video frames at a time point is too large, such as exceeding a set number threshold, the video frames may be subjected to frame extraction processing to obtain a preset number of key frames, and the video picture may be determined based on the key frames. The number of video frames included in the video picture at each time point may be 2-5 frames.

[0059] In some embodiments, the video to be processed can be sent to the computer device through the terminal when the user has a video editing requirement or a plot annotation requirement. The computer device obtains the video to be processed, and then the computer device can process the video data according to the obtained video to be processed, and provide the user with a target time point for video editing and plot annotation. After the user receives the target time point provided by the computer device through the terminal, the user can perform short video editing, highlight clip trailer editing, plot boundary annotation, etc. based on the terminal.

[0060] Step 204 , for any time point, emotion recognition is performed based on the video image and the dialogue audio at the time point to obtain an emotion recognition result.

[0061] Among them, emotion recognition includes emotion recognition of video images and emotion recognition of speech audio. Emotion recognition of video images is the process of emotion recognition of video content in the video images, and emotion recognition of speech audio is the process of emotion recognition of speech in speech audio.

[0062] Specifically, at any point in time, the computer device can perform emotion recognition on the video screen to obtain the corresponding first emotion recognition result, and perform emotion recognition on the dialogue audio to obtain the corresponding second emotion recognition result, and combine the recognition results of the video screen and the dialogue audio to obtain the final emotion recognition result.

[0063] In some embodiments, the video content in the video to be processed is related to the type of the video to be processed, and the type of the video to be processed may include a person type video, an animal type video, a natural landscape type video, etc. When the video to be processed is a person type video, the video content in the video screen may include a face, and the emotion category for the video screen may specifically be emotion recognition for a human face; when the video to be processed is an animal type video, the video content in the video screen may also include a face, and the emotion category for the video screen may specifically be emotion recognition for an animal face.

[0064] In other embodiments, when the type of video to be processed is a person type, after the computer device performs emotion recognition on the video screen, the obtained result may be the person's expression; when the type of video to be processed is an animal type video, after the computer device performs emotion recognition on the video screen, the obtained result may be the animal's expression.

[0065] In some embodiments, the dialogue audio may be a human voice or an animal voice, etc., and the dialogue audio may also be different based on the type of the video to be processed. If the type of the video to be processed is a character type video, the audio may be a human voice or a mixed sound of a human voice and background music; if the type of the video to be processed is an animal video type, the dialogue audio may be an animal voice, etc.

[0066] In some embodiments, the first emotion recognition result is a facial expression category obtained after emotion recognition is performed on a face in a video screen, and the facial expression category may include multiple categories such as an excitement category and a neutral category. For the facial expression category, the excitement category may include surprise, fear, laughter, crying, grimace, and anger; the neutral category may include expressionless, smiling, and sad.

[0067] In some embodiments, the second emotion recognition result is a speech emotion category obtained after emotion recognition is performed on the speech in the line audio, and the speech emotion category may also include multiple categories such as excitement category and indifference category. For the speech emotion category, the excitement category may include anger, surprise, happiness, crying, shouting, sadness, fear; the indifference category may include indifference and sadness.

[0068] Step 206, determining the suspense score of the targeted time point according to the video image, dialogue text and emotion recognition result at the targeted time point.

[0069] The suspense score is the calculated suspense degree of the time point, and the suspense degree can be used to determine whether the time point is a suspense point of the video to be processed. Generally, the value of the suspense score can be used to determine whether the time point is a suspense point.

[0070] Suspense points are points in the video that can attract users' interest, or are foreshadowing points before the climax of the plot. In actual film and television drama videos, suspense points are usually points that can arouse users' interest in continuing to watch. By setting suspense points, they pave the way for the development of subsequent plots.

[0071] Specifically, for each time point, the computer device can combine the video image, dialogue text and emotion recognition results of the time point to determine the suspense score of the time point, and judge whether the time point is a suspense point based on the suspense score of the time point. If so, a subsequent screening process is carried out; if not, the time point is discarded, that is, the time point is no longer involved in the determination of the subsequent target time point.

[0072] In some embodiments, the suspense score can be any parameter that can be used to provide a screening basis, such as a calculated confidence level, a score value, etc. When the suspense score is a calculated confidence level, it can be understood that the higher the confidence level of any time point, the more likely the time point is a suspense point.

[0073] Step 208 , screening candidate time points from multiple time points according to the suspense score of each time point.

[0074] The candidate time points are points in the video to be processed that contain exciting content that can arouse the user's interest. The candidate time points may specifically be suspense points screened from multiple time points.

[0075] Specifically, the computer device obtains the suspense score of each time point and screens the time points according to the suspense score of each time point, so as to screen out candidate time points from the multiple time points.

[0076] In some embodiments, when the computer device is screening the candidate time points, a suspense score threshold may be set and the screening is performed according to the suspense score threshold. For example, the computer device may determine the time points whose suspense scores are greater than the suspense score threshold as candidate time points, and discard the time points whose suspense score threshold is less than or equal to the suspense score threshold.

[0077] In other embodiments, the computer device may also segment the video to be processed according to the plot to obtain multiple sub-videos, and select candidate time points based on the sub-videos. When selecting candidate time points based on the sub-videos, the number of candidate time points for each unit may be set, and then the selection may be made based on the suspense score of the time points.

[0078] In some embodiments, candidate time points are screened from multiple time points based on the suspense score of each time point, including: performing curve fitting based on the suspense score of each time point to obtain a target fitting curve; and determining the time point corresponding to the peak in the target fitting curve as the candidate time point.

[0079] The target fitting curve is a curve fitted by the time points and the suspense scores, and can be used to characterize the correlation between the time points and the suspense scores. By fitting multiple time points and their corresponding suspense scores, a peak-valley graph can be obtained.

[0080] The plot at the time of the peak is more likely to be suspenseful.

[0081] Specifically, the computer device performs fitting through multiple time points and the suspense scores corresponding to each time point to obtain a target fitting curve. Further, the computer device can determine the slope of each point on the target fitting curve, determine the peak from each point with a slope equal to 0, and determine the time point corresponding to the peak as the candidate time point.

[0082] In some embodiments, the computer device can determine multiple peaks through the target fitting curve. The computer device can determine the time points corresponding to all the peaks as candidate time points. It is understandable that the computer device can also filter the peaks again to obtain the filtered peaks, and determine the time points corresponding to the filtered peaks as candidate time points, so that the obtained candidate time points are more streamlined, and on the basis of improving the determination accuracy of the target time point, the amount of calculation is reduced, and the processing efficiency of the video data is improved.

[0083] In some embodiments, the computer device may set a peak number threshold, and when the number of peaks obtained is greater than the peak number threshold, further screening is performed on the peaks to obtain target peaks. The setting of the peak number threshold may be adaptively adjusted in combination with efficiency requirements and accuracy requirements of video data processing.

[0084] In the above embodiment, the computer device performs fitting through multiple time points and the suspense scores corresponding to each time point to obtain a target fitting curve. Further, the computer device determines the peak from the target fitting curve and determines the time point corresponding to the peak as the candidate time point. The peak indicates that the probability of the plot at this time point being suspenseful is high, so that the candidate time point can be accurately determined from the time points, thereby improving the accuracy of determining the target time point.

[0085] Step 210, perform integrity analysis on the multimodal data of each candidate time point, and determine a target time point from the candidate time points based on the integrity analysis result; the target time point is used to determine the end point of the video.

[0086] The integrity analysis is used to analyze whether the plot of the video clips obtained by segmentation based on the candidate time points is logically complete, that is, whether it has logical integrity.

[0087] Specifically, the computer device can perform integrity analysis on the candidate time points in combination with the video screen, the line audio and the line text. The computer device can determine whether the candidate time point is the plot end point of the entire video or the plot end point of each unit in the video based on the video screen, and the computer device can retain the candidate time point that belongs to the plot end point.

[0088] The computer device can also determine whether the candidate time point is in the time segment where the voice and / or the line text appears based on the line audio and the line text, and retain the candidate time points that are not in the time segment where the voice and / or the line text appears.

[0089] The computer equipment can combine the results of each integrity analysis to determine the final target time point. By combining the three types of information, namely video images, dialogue audio and dialogue text, for integrity analysis, the integrity of the video content at the target time point can be guaranteed, avoiding the abrupt interruption of the edited video at the target time point in the middle of a complete plot, which can improve the overall quality of subsequent video materials and the viewing interest of users.

[0090] In some embodiments, when the computer device performs integrity analysis based on the video screen, it can first determine the plot end point in the video, compare the candidate time point with the plot end point, and obtain the integrity analysis result. For example, the computer device can perform difference calculation between the time point of the candidate time point and the time point of the plot end point to determine the time difference between the two. When the time difference exceeds the set time difference threshold, it indicates that the candidate time point is far away from the plot end point, and it is determined that the plot of the candidate time point does not have integrity. Among them, the time difference threshold can be set in minutes, such as 1min, 2min, etc., or in seconds, such as 10s, 48s, etc. In some embodiments, due to the various types of videos to be processed, the plot end point determined by the computer device can be only one or more. In the case where there are multiple plot end points, the computer device can select a plot end point close to the candidate time point for comparison, for example, select a plot end point close in time for comparison, and obtain the integrity analysis result.

[0091] In some embodiments, when the computer device performs an integrity analysis based on the dialogue audio and dialogue text, it can determine the time segment of the dialogue audio and the time segment of the dialogue text corresponding to the candidate time point, and compare the candidate time point with the time segment of the dialogue audio and the time segment of the dialogue text to obtain the integrity analysis result. For example, the computer device can determine whether the candidate time point falls within the time segment of the dialogue audio. If it falls within the time segment of the dialogue audio, it is determined that the candidate time point does not have integrity. The computer device can also determine whether the candidate time point falls within the time segment of the dialogue text. If it falls within the time segment of the dialogue text, it is determined that the candidate time point does not have integrity.

[0092] In the above-mentioned video data processing method, multiple time points of the video to be processed are determined, and multimodal data corresponding to each time point is obtained, wherein the multimodal data at least includes video screen, line audio and line text; further, for any time point, emotion recognition is performed according to the video screen and line audio of the time point, and the emotion recognition result is obtained, thereby determining the suspense score of the time point according to the video screen, line text and emotion recognition result of the time point, that is, when calculating the suspense score of the time point, the application integrates the data of multiple modes such as video screen, line text and emotion recognition result, which can not only ensure the accuracy of the suspense score, but also has a certain robustness. Further, according to the suspense score of each time point, the candidate time point with suspense is screened out. Finally, the integrity analysis is performed according to the multimodal data of each candidate time point, and the target time point is determined again according to the integrity analysis result, wherein the target time point is used to determine the end point of the video. The suspense score is calculated by using the multimodal data in this application, and the candidate time points are screened according to the suspense score. The candidate time points are further analyzed for integrity, and the results of the integrity analysis are combined for screening again, so that the content at the end of the video has a certain logical integrity and suspense, completely getting rid of human involvement, and greatly improving the editing efficiency. At the same time, because the positioning of the target time point in this application is a standardized operation, it will not be differentiated due to different personnel, which ensures the accuracy of the prediction of the video end point.

[0093] In some embodiments, the emotion recognition result includes a first emotion recognition result and a second emotion recognition result. For any point in time, expression recognition is performed based on the video image and dialogue audio at the targeted time point to obtain the emotion recognition result, including: for any point in time, facial emotion recognition is performed based on the video image at the targeted time point to obtain the first emotion recognition result; for any point in time, voice emotion recognition is performed based on the dialogue audio at the targeted time point to obtain the second emotion recognition result.

[0094] Among them, facial emotion recognition is the process of identifying faces appearing in the video screen, such as the faces of people and animals, to determine the facial emotions. Facial emotion recognition can specifically include detecting faces in the video screen and determining the facial emotion category of the detected faces.

[0095] Speech emotion recognition is the process of identifying the speech in the dialogue audio, such as the speech of a character or an animal, to determine the emotion of the speech. Facial emotion recognition can specifically include identifying whether there is speech in the dialogue audio, and if there is speech, determining the speech emotion category of the recognized speech.

[0096] Specifically, at any point in time, when the computer device performs facial emotion recognition on the video screen, it can be based on a pre-trained facial emotion recognition model; when the computer device performs speech emotion recognition on the dialogue audio, it can be based on a pre-trained speech emotion recognition model. Of course, the computer device can also use other algorithms that can be used for expression recognition to carry out recognition, and there is no limitation here.

[0097] In some embodiments, the facial emotion recognition model and the speech emotion recognition model can be a network model constructed based on an artificial intelligence algorithm. When constructing the facial emotion recognition model and the speech emotion recognition model, they can be constructed based on multiple types of algorithms such as supervised learning algorithms and unsupervised learning algorithms. There is no limitation here. For example, when constructing a facial emotion recognition model, MTCNN (Multi-task Cascaded Convolutional Networks), PCA (principal component analysis), and LFA (Local Face Analysis) can be used; when constructing a speech emotion recognition model, a variety of algorithms such as Resnet (Residual Network) and FPN (feature pyramid networks) can be used to construct it.

[0098] In some embodiments, the facial emotion recognition model may be composed of multiple models, such as a neural network model capable of realizing facial positioning and a neural network model capable of realizing facial emotion recognition. By using different neural network models to perform facial positioning and facial emotion recognition in a targeted manner, the accuracy of facial emotion recognition can be improved. Of course, the facial emotion recognition model may also be a neural network model that includes both facial detection and expression recognition functions. By using only one neural network model to complete facial emotion recognition, the efficiency of facial emotion recognition can be improved.

[0099] In some embodiments, the speech emotion recognition model may also be composed of multiple neural network models, such as a neural network model that can realize speech detection and a neural network model that can realize speech emotion recognition. By using different neural network models to perform targeted speech detection and speech emotion recognition, the accuracy of speech emotion recognition can be improved. Of course, the speech emotion recognition model may also be a neural network model that includes both speech detection function and emotion recognition function. Only one neural model is used to complete speech emotion recognition, which can improve the efficiency of speech emotion recognition.

[0100] In the above embodiment, the computer device performs emotion recognition on the video screen and on the audio of the dialogue to obtain corresponding emotion recognition results from the facial emotions and the voice emotions respectively. Since the facial emotions and the voice emotions are closely related to the plot, the target time point finally located can have a certain robustness and make the located target time point more accurate.

[0101] In some embodiments, for any point in time, facial emotion recognition is performed based on the video picture at the targeted time point to obtain a first emotion recognition result, including: for any point in time, facial detection is performed based on the video picture at the targeted time point, and a facial image is captured from the video picture based on the detection result; feature extraction is performed based on the facial image to obtain a vector representation of the facial image; based on the vector representation of the facial image, a facial expression category to which the facial image belongs is predicted, and a first emotion recognition result is determined based on the facial expression category.

[0102] Among them, face detection is the process of locating the face in the video screen. The vector representation can be used to reflect the characteristics of the facial image, that is, the semantic feature information of the facial image. The facial expression category is the type of facial emotion of the facial image predicted based on the semantic feature information of the facial image.

[0103] Specifically, for any time point, the computer device can use a pre-trained neural network model for facial detection to perform facial detection based on the video image at the time point, locate the position of the face, and based on the located position, extract the face in the video image, such as a human face or an animal face, to obtain a facial image. Further, the computer device can use a pre-trained network model for emotion recognition to perform feature encoding on the facial image to obtain a vector representation of the facial image, and finally perform facial emotion recognition on the facial image based on the vector representation to obtain a first emotion recognition result.

[0104] In some embodiments, the position of the face located by the computer device is the coordinates of a box that can locate the face, specifically four coordinate points of the box. The computer device can capture the facial image based on the four coordinate points of the box.

[0105] In some embodiments, reference Figure 3 As shown, this is a flowchart for facial emotion recognition:

[0106] in, Figure 3The flowchart includes a video screen, a facial positioning module, a facial encoding module, and an expression recognition module. The video screen includes multiple video frames, and the computer device can input the multiple video frames into the facial positioning module, and the facial positioning module locates the facial image from each video frame respectively, and cuts out the located facial image from the video frame. Furthermore, in the facial encoding module, the cut-out facial image can be encoded to obtain a vector representation of the facial image, and finally, the facial emotion recognition is performed in the expression recognition module based on the vector representation of the facial image to obtain a first emotion recognition result.

[0107] When the video to be processed is a character type video, the face positioning module may specifically refer to a human face positioning module, and the face encoding module may specifically refer to a human face encoding module.

[0108] In some embodiments, when performing face detection, the neural network model is developed based on the MTCNN network training. Figure 4 As shown, this is an example diagram of the entire MTCNN network structure:

[0109] The network structure of MTCNN can be divided into three modules: Proposal Network (P-Net), Refine Network (R-Net) and Output Network (O-Net). The following steps are the face detection process based on MTCNN:

[0110] (1) For the video screen at the time point, face detection can be performed on each video frame. The following process is described by taking face detection on one of the video frames as an example. First, the video frame can be transformed (resized) to different scales to construct an image pyramid to adapt to the detection of faces of different sizes. The image pyramid can be constructed by resizing the video frame according to the set scaling factor (resize_factor) until it is equal to the size required by P-Net, for example, it can be 12*12. Thus, an image pyramid can be obtained, that is, the original video frame and the scaled video frame. Among them, the value of resize_factor is determined according to the actual face size distribution in the video frame, for example, it can be set between 0.70-0.80. If the value of resize_factor is set to be relatively large, it is easy to prolong the reasoning time, and if it is small, it is easy to miss some small and medium-sized faces. In the embodiment of the present application, resize_factor can be set to 0.70, for example.

[0111] (2) Input the obtained image pyramid into P-Net to obtain a large number of candidates. P-Net is a candidate network for face regions, including three convolutional layers, the sizes of which are 5*5*10, 3*3*16 and 1*1*12 respectively. The output includes three parts: classification result (binary classification, representing whether it is a face), the position of the detected face frame (bbox) and the position of the located key points (landmarks). The embodiment of the present application does not limit the number of key points. Here, the number of key points is 5 as an example. The 5 key points can be the left eye, right eye, nose, left corner of the mouth and right corner of the mouth respectively.

[0112] The first part of the output is used to determine whether there is a face in the image. The output vector size is 1×1×2, that is, two values. The second part outputs the position of the bbox, which is generally called box regression. The image input to P-Net may not be the perfect position of the face frame. For example, sometimes the face is not exactly square, and the 12×12 video frame may be biased to the left or right. Therefore, it is necessary to output the offset of the relatively perfect face frame position. The offset size is 1×1×4, which means the relative offset of the horizontal coordinate of the upper left corner of the frame, the relative offset of the vertical coordinate of the upper left corner of the frame, the error of the width of the frame, and the error of the height of the frame. The third part outputs the position of the key points. Taking 5 key points as an example, the 5 key points correspond to the position of the left eye, the position of the right eye, the position of the nose, the position of the left mouth, and the position of the right mouth. Each key point needs to be represented in two dimensions, so the output vector size is 1×1×10.

[0113] All images are input to P-Net, and the map, offset and classification score are output. The shape of the map is (m, n, 16), where m and n are the length and width of the map. According to the classification score, a large part of the candidates can be screened out, and then the bbox is calibrated according to the 4 offsets to obtain the upper left and lower right coordinates of the bbox. According to the intersection over union (IOU) value, non-maximum suppression (NMS) is performed to screen out a large part of the candidates. The specific process is: according to the classification score, the tensor (num_left, 4) is obtained from large to small, that is, the upper left and lower right absolute coordinates of num_left bboxes. Each time, the IOU is calculated with the bbox coordinates of the maximum score value and the remaining coordinates, and the bbox with an IOU greater than 0.6 (the threshold is set in advance) is removed, and the maximum score value is moved to the final result. Repeating this operation will remove many bboxes with a lot of overlap, and finally get (num_left_after_nms, 16) candidates. These candidates need to be cut out from the original video frame according to the bbox coordinates, resized to 24*24 and input into R-Net.

[0114] (3) The candidate images selected by P-Net are fine-tuned by R-Net. The network structure of R-Net is different from that of P-Net. It has an additional fully connected layer, so it will achieve better results. Before entering R-Net, the images need to be scaled to 24×24×3. The output of R-Net is the same as that of P-Net. The purpose of R-Net is to remove a large number of non-face frames. According to the position of P-Net output, part of the image is cut out from the original video frame. When cutting out part of the image, it is necessary to cut out the square with the maximum side length of the bbox. This is to ensure that there is no deformation during resizing and to retain more details around the face frame. R-Net will still output two outputs of binary classification one-hot code (i.e. classification results), four outputs of bbox coordinate offset, i.e. the position of the face frame and ten outputs of the landmark position, i.e. the position of the key point. According to the binary classification score, most of the candidates that are not faces are removed, and the offset of the bbox of the screenshot is adjusted, that is, the x and y coordinates of the upper left, lower right and lower are adjusted up and down, left and right, and the NMS based on the IOU value described in P-Net is repeated again to filter out a large number of candidates. The final output of R-Net is also (num_left_after_R-net, 16). According to the position of the bbox, part of the image is cut out from the original video frame and input into O-Net. The square with the maximum side length is also used to avoid deformation and retain more details.

[0115] (4) After R-Net removes many candidates, the image is input into O-Net, which outputs the accurate location of the bbox and the landmark. From the network structure, we can see that O-Net has one more convolutional layer than R-Net, so the processing result will be more refined. The input image size is 48×48×3, and the output includes the classification result, the location of the detected face frame, and the location of the key points. The process of P-Net can be roughly repeated, but the difference is that this time, in addition to paying attention to the location of the bbox (the location can be expressed in coordinates), the coordinates of the landmark are also output. After classification screening and adjusted NMS screening, the accurate location of the face frame and the landmark are obtained.

[0116] In some embodiments, reference Figure 5 As shown in the figure, the effect of facial detection based on MTCNN is obtained:

[0117] Among them, from Figure 5 As can be seen in the figure, by inputting the video frame including the blank position into the multi-task convolutional neural network, namely MCTNN, the MCTNN performs face detection, and the MCTNN can obtain the bbox coordinates of all faces in the video frame. The bbox coordinates are the four points located in the face box. The computer device can cut the video frame by locating the calculated bbox coordinates, so as to obtain the face image in the video frame.

[0118] In some embodiments, the computer device may encode the facial image based on an encoding network, an encoding algorithm, etc. to obtain a vector representation of the facial image. The encoding network may be Resnet, which may include Resnet50, Resnet101, etc. Resnet50 may be divided into 5 stages, see Figure 6 As shown in FIG. 1 , it is a schematic diagram of the structure of the residual network. The residual network can specifically be Resnet50. Resnet50 can be divided into five stages, namely stage 0, stage 1, stage 2, stage 3 and stage 4.

[0119] In some embodiments, the structure of each stage of Resnet50 can be seen in Figure 7As shown, the structure of Stage 0 is relatively simple and can be regarded as a preprocessing of the input. The last four stages are composed of bottleneck convolution layers (Bottleneck, BTNK) and have similar structures. The shape of Stage 0 input is (3,224,224), 3 is the number of channels, and the two 224s are the height and width respectively. The structure of Stage 0 includes the first and second layers. The first layer includes three operations in sequence, namely Conv (Convolutional Neural Network, Convolutional Neural Network), Batch Normalization (Batch Normalization, BN), and Rectified Linear Unit (RELU). The convolution kernel size of Conv is 7×7, the number of convolution kernels is 64, and the stride of the convolution kernel is 2, which can be represented by / 2.

[0120] The second layer is the maximum pooling layer (MAXPOOL), whose kernel size is 3×3 and stride is 2. The shape of the Stage0 output is (64,56,56), where 64 is equal to the number of convolution kernels in the first convolution layer of stage0, and 56 is equal to 224 / 2 / 2 (a stride of 2 will halve the input size).

[0121] Stage 1 contains 3 Bottlenecks (including 1 Bottleneck Convolution 1 (BTNK1) and 2 Bottleneck Convolution 2 (BTNK2)), and the remaining 3 stages include 4 Bottlenecks (including 1 BTNK1 and 2 BTNK2), 6 Bottlenecks (including 1 BTNK1 and 5 BTNK2), and 3 Bottlenecks (including 1 BTNK1 and 2 BTNK2). The shape of Stage 1 output is (256, 56, 56), the shape of Stage 2 output is (512, 28, 28), the shape of Stage 3 output is (1024, 14, 14), and the shape of Stage 4 output is (2048, 7, 7). Therefore, the facial image is converted into a 2048 vector representation through Resnet50, which can represent the semantic feature information of this facial image.

[0122] It should be noted that the structures of BTNK1 and BTNK2 can be found in Figure 8 As shown in the figure, BTNK2 has two variable parameters C and W, namely C and W in the input shape (C, W, W), 3 convolutional layers on the left side of BTNK2 and related BN and RELU. Compared with BTNK2, BTNK1 has one more convolutional layer on the right side, which plays a role in matching the difference in input and output dimensions.

[0123] In the above embodiment, for any time point, the computer device first locates the position of the face, and cuts the video screen based on the located facial position to obtain a facial image. Further, the computer device performs feature encoding on the facial image to obtain a vector representation of the facial image, and finally recognizes the facial expression based on the vector representation, so that the first emotion recognition result can be accurately obtained.

[0124] In some embodiments, for any point in time, speech emotion recognition is performed based on the dialogue audio at the targeted time point to obtain a second emotion recognition result, including: for any point in time, extracting the acoustic feature representation of the dialogue audio at the targeted time point; encoding based on the acoustic feature representation to obtain a coded feature output; performing speech recognition based on the coded feature output to obtain a speech recognition result for the targeted time point; when the speech recognition result indicates the presence of speech, performing speech emotion recognition based on the coded feature output to obtain a second emotion recognition result.

[0125] The acoustic feature represents the speech feature information used to characterize the speech audio. Speech recognition is a recognition process used to identify whether there is speech in the speech audio, such as whether there is a human voice or other speech that can be used for speech emotion recognition in the speech audio.

[0126] Specifically, for any point in time, the computer device can extract the acoustic feature representation of the point in time, and use a pre-trained speech emotion recognition model or encoding algorithm to encode the acoustic feature representation. Among them, the speech emotion recognition model can use convolution kernels of different sizes or convolution kernels of different dimensions to encode the acoustic feature representation, so that more detailed speech feature information can be obtained when encoding the acoustic feature representation. Furthermore, the computer device performs speech recognition based on the obtained encoding feature output to determine whether there is speech in the dialogue audio at the time point. If there is speech, the speech emotion is recognized based on the encoding feature output to obtain a second emotion recognition result.

[0127] In some embodiments, the acoustic feature representation is speech feature information including the time domain features and frequency domain features of the line audio, wherein the time domain features include a large amount of time domain information, such as audio loudness, sampling point amplitude, etc. The frequency domain features include information about the sampling point frequency, so that by extracting the acoustic feature representation, the characteristics of the line audio can be reflected, which is helpful for speech recognition. Of course, the acoustic feature representation can also only include the time domain features of the line audio in the time dimension, or only include the frequency domain features of the line audio in the frequency dimension. Specifically, the speech feature information included in the acoustic feature representation can be adaptively adjusted in combination with the accuracy requirements of speech recognition.

[0128] In the above embodiment, for any time point, the computer device can extract the acoustic feature representation of the time point and encode the acoustic feature representation. Further, the computer device performs speech recognition based on the obtained encoded feature output to determine whether there is speech in the recognized audio at the time point. If there is speech, speech emotion recognition is finally performed based on the encoded feature output to accurately obtain the second emotion recognition result.

[0129] In some embodiments, encoding is performed based on acoustic feature representation to obtain a coding feature output, including: performing a first encoding based on the acoustic feature representation to obtain a first coding output; the first encoding is a coding focusing on time domain perception; performing a second encoding based on the acoustic feature representation to obtain a second coding output; the second encoding is a multidimensional encoding; performing a third encoding based on the acoustic feature representation to obtain a third coding output; the third encoding is a coding focusing on frequency domain perception; and determining the coding feature output based on the first coding output, the second coding output, and the third coding output.

[0130] Among them, the first encoding focuses on encoding the acoustic feature representation from the time dimension, the multi-dimensional encoding encodes the acoustic feature representation from both the time dimension and the frequency dimension, and the third encoding focuses on encoding the acoustic feature representation from the frequency dimension.

[0131] Specifically, when the computer device encodes the acoustic feature representation, different processing branches can be used to perform targeted encoding on the acoustic feature representation. When performing the first encoding, the computer device can use the first one-dimensional encoding processing branch to expand, the first one-dimensional encoding processing branch is based on a one-dimensional dimension to encode the acoustic feature representation, and the first one-dimensional encoding branch focuses on information extraction in the time dimension; when performing the second encoding, the computer device can use the two-dimensional encoding processing branch to expand, the two-dimensional encoding processing branch is based on two dimensions at the same time to encode the acoustic feature representation, that is, to extract information in the time dimension and the frequency dimension at the same time. When performing the third encoding, the computer device can use the second one-dimensional encoding processing branch to expand, and the second one-dimensional encoding branch focuses on information extraction in the frequency dimension.

[0132] In some embodiments, the encoding focusing on time domain perception can be to extract information in the time dimension only for the acoustic feature representation. Of course, it can also be to extract information in the time dimension first, and then extract information in the frequency dimension from the frequency dimension. This allows the frequency domain information to be acquired while focusing on the acquisition of time domain information, thereby maintaining the complementarity of time domain information and frequency domain information, and improving the accuracy of subsequent speech emotion recognition.

[0133] In some embodiments, the encoding focusing on frequency domain perception can be to extract information in the frequency dimension only for the acoustic feature representation. Of course, it can also be to extract information in the frequency dimension first, and then extract information in the time dimension from the time dimension. This allows the acquisition of time domain information while focusing on frequency domain information acquisition, thereby maintaining the complementarity of time domain and frequency domain information and improving the accuracy of subsequent speech emotion recognition.

[0134] In some embodiments, when the computer device encodes the acoustic feature representation in different dimensions, the first one-dimensional encoding processing branch, the second one-dimensional encoding processing branch, and the two-dimensional encoding processing branch can be implemented by convolution. If convolution layers of different dimensions are used, the one-dimensional convolution layer encodes from one dimension, and the first one-dimensional encoding processing branch and the second one-dimensional encoding processing branch can be encoded by the one-dimensional convolution layer. As for whether the one-dimensional dimension is the time dimension or the frequency dimension, it can be set according to actual needs; the two-dimensional convolution layer encodes from two dimensions at the same time, and the two-dimensional encoding processing branch can be encoded by the two-dimensional convolution layer.

[0135] In some embodiments, after obtaining the first coding output, the second coding output, and the third coding output, the computer device can process the three coding outputs, such as fusion processing, averaging processing, etc., so that the time domain and frequency domain can maintain information complementarity while obtaining more detailed feature information.

[0136] In the above embodiment, when the computer device encodes the acoustic feature representation, it performs the first encoding, the second encoding and the third encoding on the acoustic feature representation respectively, so as to obtain the encoding output with different emphasis dimensions, and finally determines the encoding feature output according to the encoding output with different emphasis dimensions, which pays attention to the perceptual learning in the time dimension as well as the perceptual learning in the frequency dimension, so that more detailed information can be obtained.

[0137] In some embodiments, a first encoding is performed based on the acoustic feature representation to obtain a first encoding output, including: first performing feature extraction in the time dimension and then performing feature extraction in the frequency dimension on the acoustic feature representation through a convolution kernel of a first size to obtain a first intermediate feature; performing feature extraction on the acoustic feature representation through a convolution kernel of a second size to obtain a second intermediate feature; the second size is larger than the first size; and fusing the first intermediate feature and the second intermediate feature to obtain the first encoding output.

[0138] Among them, the convolution kernel of the first size is a small-sized convolution kernel, and the convolution kernel of the second size is a convolution kernel that is larger than the convolution kernel of the first size.

[0139] In some embodiments, the small size may be smaller than the first preset size, the large size may be larger than the second preset size, and the second preset size is greater than or equal to the first preset size. The actual values ​​of the first preset size and the second preset size may be adaptively set in combination with specific video data processing scenarios, video data processing accuracy, etc.

[0140] In some embodiments, when a convolution kernel of a first size is used to extract features from acoustic features, the first one-dimensional processing branch may be composed of several convolution kernels. For coding that focuses on time domain perception, the size of the front convolution kernel in the first one-dimensional processing branch may be set to be more biased towards the time domain, that is, the convolution kernel is larger in size in the time domain, such as setting the size of the convolution kernel in the first half to be more biased towards the time domain, and the size of the convolution kernel in the second half to be more biased towards the frequency domain. Of course, it is also possible to set the size of two-thirds of the convolution kernels in the first one-dimensional processing branch to be more biased towards the time domain. The specific number of convolution kernels that are more biased towards the time domain can be set based on actual needs, as long as it is determined that the first one-dimensional processing branch is set to be more biased towards the extraction of time domain information.

[0141] In some embodiments, the input of acoustic features can be represented by t*n, where t represents time domain information and n represents frequency domain information. In the first one-dimensional processing branch, the size of the first half of the convolution kernel can be 3*1, and the size of the second half of the convolution kernel can be 1*3. Among them, 3*1 can represent that the size of the convolution kernel is 3 in the time domain and 1 in the frequency domain, so it has a larger size in the time domain and a greater sensitivity, so more information in the time domain is extracted during calculation.

[0142] In some embodiments, when a computer device extracts features of an acoustic feature representation through a convolution kernel of a first size, the convolution kernel of the first size is also connected to a pooling layer adapted thereto, thereby obtaining a first intermediate feature through the convolution kernel of the first size and the pooling layer.

[0143] In other embodiments, the computer device extracts features of the acoustic feature representation through a convolution kernel of a second size, and the convolution kernel of the second size is also connected to a pooling layer adapted thereto, thereby obtaining a second intermediate feature through the convolution kernel of the second size and the pooling layer.

[0144] In the above embodiment, the computer device uses a convolution kernel of a first size to extract features focusing on the time dimension to obtain a first intermediate feature, and then uses a convolution kernel of a second size larger than the first size to extract acoustic representation features to obtain a second intermediate feature. By fusing the first intermediate feature and the second intermediate feature, a third encoded output is obtained, which can not only obtain features in the time domain and frequency domain, but also obtain more complete information due to the use of convolution kernels of different sizes, thereby effectively improving the accuracy of video data processing.

[0145] In some embodiments, a third encoding is performed based on the acoustic feature representation to obtain a third encoding output, including: first performing feature extraction in the frequency dimension and then performing feature extraction in the time dimension on the acoustic feature representation through a convolution kernel of a first size to obtain a third intermediate feature; performing feature extraction on the acoustic feature representation through a convolution kernel of a second size to obtain a second intermediate feature; the second size is larger than the first size; and fusing the third intermediate feature and the second intermediate feature to obtain a third encoding output.

[0146] Among them, this embodiment focuses on feature extraction of the frequency dimension. Similar to the aforementioned feature extraction focusing on the time dimension, the number of convolution kernels on the second one-dimensional processing branch can also be set to obtain a third intermediate feature focusing on the frequency dimension.

[0147] In some embodiments, when a convolution kernel of a first size is used to extract features of acoustic features, the second one-dimensional processing branch may be composed of several convolution kernels. For coding that focuses on frequency domain perception, the size of the front convolution kernel in the second one-dimensional processing branch may be set to be more biased towards the frequency domain, that is, the convolution kernel is larger in size in the frequency domain, such as setting the size of the convolution kernel in the first half to be more biased towards the frequency domain, and the size of the convolution kernel in the second half to be more biased towards the time domain. Of course, it is also possible to set the size of two-thirds of the convolution kernels in the second one-dimensional processing branch to be more biased towards the frequency domain. The specific number of convolution kernels whose sizes are more biased towards the frequency domain can be set based on actual needs, as long as it is determined that the second one-dimensional processing branch is set to be more biased towards the extraction of frequency domain information.

[0148] In some embodiments, the input of acoustic features can be represented by t*n, where t represents time domain information and n represents frequency domain information. In the one-dimensional processing branch, the size of the first half of the convolution kernel is 1*2, and the size of the second half of the convolution kernel is 2*1. Among them, 1*2 means that the size of the convolution kernel is 2 in the frequency domain and 1 in the time domain, so it has a larger size in the frequency domain and a greater sensitivity, so it extracts more information in the frequency domain during calculation.

[0149] In some embodiments, when the computer device extracts features of the acoustic feature representation through a convolution kernel of a first size, the convolution kernel of the first size is also connected to a pooling layer adapted thereto, thereby obtaining a third intermediate feature through the convolution kernel of the first size and the pooling layer.

[0150] In the above embodiment, the computer device uses a convolution kernel of a first size to perform feature extraction focusing on the frequency dimension to obtain a third intermediate feature, and then uses a convolution kernel of a second size that is larger than the first size to extract the acoustic representation feature to obtain a second intermediate feature. By fusing the third intermediate feature and the second intermediate feature, a third encoded output is obtained, which not only obtains features in the time domain and frequency domain, but also makes information acquisition more complete due to the use of convolution kernels of different sizes, thereby effectively improving the accuracy of video data processing.

[0151] In some embodiments, reference Fig. 9 As shown in FIG. 1 , it is a flow chart of performing speech emotion recognition on the lines audio to obtain the second emotion recognition result:

[0152] Among them, reference Fig. 9 As shown, it is a structural diagram of the speech emotion recognition model in this embodiment. The speech emotion recognition model can first process the line audio to obtain an acoustic feature representation. Among them, the speech emotion recognition model can calculate the log-mel spectrum feature of the line audio to obtain the mel frequency. Mel frequency is a nonlinear frequency scale. It can be understood that the line audio itself is a time domain signal, and the time domain signal also contains a large amount of time domain information, such as: audio loudness and sampling point amplitude information, which can reflect the characteristics of the line audio and help voice recognition. Therefore, the acoustic feature representation is a feature representation that includes both time domain information and frequency domain information.

[0153] After obtaining the acoustic feature representation, the speech emotion recognition model can perform speech recognition on the acoustic feature representation of the currently input line audio to obtain a speech recognition result. When the speech recognition result indicates the presence of speech, speech emotion recognition is performed, wherein the result of the speech emotion recognition can be used for weighting the subsequent second encoded data.

[0154] The speech emotion recognition model is divided into three branches, namely the left branch, the middle branch and the right branch. Among the three branches, the left and right branches are composed of a one-dimensional small convolution kernel, i.e., a convolution kernel of the first size, and a residual connection, which is a convolution kernel of the second size. The middle branch is composed of a two-dimensional small convolution kernel. The three branches are merged to obtain the mean, and finally passed through the pooling layer and shape reconstruction to obtain the encoded feature output.

[0155] Furthermore, when it is determined that there is speech in the dialogue audio based on the encoded feature output, it is decided whether to start the branch of speech emotion recognition. After starting the branch of speech emotion recognition, the second emotion recognition result can be determined based on the pooling layer, the convolution layer, the shape reconstruction, the multi-layer perceptron and the classification layer.

[0156] in, Fig. 9 In the figure, the upper part of the left branch, namely the first, second and third layers, is a one-dimensional convolution and one-dimensional pooling in the time dimension, and the lower part, namely the fourth, fifth and sixth layers, is a one-dimensional convolution and one-dimensional pooling in the frequency dimension. The architecture of the right branch is opposite to that of the left branch. The purpose of this architecture is to allow half of the network layers to focus on perceptual learning in the time domain, and half of the network layers to focus on perceptual learning in the frequency domain, so that the network can obtain detailed information in the time domain and frequency domain respectively. Then a residual connection is added to the left and right branches respectively, which is to use a large convolution kernel and a large pooling layer to splice with the results of the last branch, so that semantic information combined by the bottom and high layers can be obtained.

[0157] The middle branch uses a two-dimensional small convolution kernel and a corresponding pooling layer. The features of the three branches are calculated and averaged, and then the final encoding feature output is obtained through a pooling layer on a channel. A classification layer (softmax) for binary classification is set in the middle branch to determine whether it is a human voice.

[0158] After judgment, if the current dialogue audio is an audio segment containing voice, the branch of life emotion recognition and classification will be opened, which is the lower right corner of the figure. The lower right corner is composed of pooling layer, convolution layer, shape reconstruction, multi-layer perceptron, and classification layer. After passing the classification layer in the lower right corner, the second emotion recognition result can be calculated.

[0159] A large number of small convolution kernels are used as the components of the convolution layer in the speech emotion recognition model, thereby replacing large convolution kernels. If small convolution kernels are used, in order to achieve the same convolution scale calculation, it is necessary to expand deeper network layers, so when using small convolution kernels, the network depth of the entire system is higher. The use of small convolution kernels can pay attention to more details when performing feature calculations, because under the calculation of the entire convolution kernel, the smaller the size, the more changes in details can be noticed. In conjunction with the recognition scenarios of human voice recognition and human voice emotion classification, the differences between different people’s voice characteristics are very small, so it is easier to capture details.

[0160] In some embodiments, the emotion recognition result includes a first emotion recognition result and a second emotion recognition result; determining the suspense score for the targeted time point based on the video picture, dialogue text and emotion recognition result at the targeted time point, including: obtaining first encoding data of the video picture at the targeted time point, and obtaining video picture encoding features based on the first encoding data and the first emotion recognition result; obtaining second encoding data of the dialogue text at the targeted time point, and obtaining text encoding features based on the second encoding data and the second emotion recognition result; fusing the video picture encoding features and text encoding features at the targeted time point to obtain multimodal fusion features at the targeted time point; determining the suspense score for the targeted time point based on the multimodal fusion features.

[0161] The first coded data is obtained by encoding the video frames included in the video picture; the first coded data is obtained by encoding the dialogue text.

[0162] Specifically, for each time point, the computer device can determine the first emotion recognition result and the first coding data at the time point, determine the target confidence corresponding to the facial emotion through the first emotion recognition result, and weight or downgrade the first coding data based on the target confidence to obtain the video picture coding feature. The computer device can also determine the second emotion recognition result and the second coding data at the time point, determine the target confidence corresponding to the voice emotion through the second emotion recognition result, and weight or downgrade the second coding data based on the target confidence to obtain the text coding feature.

[0163] In some embodiments, the computer device may use various image coding networks to encode the entire video frame included in the video screen, such as the Resnet50 network mentioned above, or other coding networks other than the Resnet50 network, such as VGG (Visual Geometry Group, visual geometry network), without limitation.

[0164] In some embodiments, the computer device can input the entire video frame sequence for the targeted time point into the Resnet50 network, thereby encoding the entire video frame sequence into an embedding (vector representation) sequence based on the Resnet50 network, and using the embedding sequence of the video screen as the first encoded data.

[0165] In some embodiments, the computer device may use various text encoding networks to encode the dialogue text to obtain second encoded data. For example, the text encoding network may be BERT (Bidirectional Encoder Representations from Transformers), GPT3 (Generative Pre-trained Transformer 3), GPT4 (Generative Pre-trained Transformer 4), Clip (Contrastive Language-Image Pre-Training), etc., which is not limited in the embodiments of the present application.

[0166] In some embodiments, the computer device uses a BERT model to encode the dialogue text. The input of the BERT model is the dialogue text corresponding to the targeted time point, so that the embedding output can be performed for each dialogue at the targeted time point, and the embedding sequence of the text is used as the second encoded data.

[0167] In some embodiments, when a computer device fuses video picture coding features and text coding features, it may use various fusion methods such as Early Fusion and Late Fusion to perform multi-modal fusion.

[0168] In some embodiments, for the targeted time point, when determining the target confidence corresponding to the facial emotion, the computer device can determine the maximum confidence among the confidences of the first target category corresponding to each frame of the video at that time point, and determine the first emotion weighted value based on the maximum confidence and the first target category corresponding to the maximum confidence, and use the first emotion weighted value as the target confidence of the facial expression.

[0169] The first target category of each video frame is the category corresponding to the calculated maximum confidence of the video frame. For example, for one of the video frames, the confidence of the facial expression category of the video frame belonging to the surprise category is 0.3, the confidence of belonging to the fear category is 0.3, the confidence of belonging to the laughter category is 0.6, the confidence of belonging to the crying category is 0.2, the confidence of belonging to the grimace category is 0.5, the confidence of belonging to the anger category is 0.1; the confidence of belonging to the expressionless category is 0.4, the confidence of belonging to the smile category is 0.7, and the confidence of belonging to the sadness category is 0.45. It can be seen that the first target category of the video frame is the smile category.

[0170] In some embodiments, when the computer device determines the first emotion weighted value based on the maximum confidence and the first target category corresponding to the maximum confidence, if the first target category corresponding to the maximum confidence belongs to the excitement category, the maximum confidence can be directly used as the first emotion weighted value, thereby weighting the first coded data through the first emotion weighted value. If the first target category corresponding to the maximum confidence belongs to the bland category, the computer device can set the first emotion weighted value to 0, or assign a smaller value to the first emotion weighted value, such as less than a set threshold value, which can be 0.2, 0.3, etc., thereby down-weighting the first coded data through the first emotion weighted value.

[0171] In some embodiments, there is a time point A, including three video frames: a first video frame, a second video frame, and a third video frame. The first target category of the first video frame is fear, with a confidence of 0.9; the first target category of the second video frame is laughter, with a confidence of 0.7; the first target category of the third video frame is crying, with a confidence of 0.6. The maximum confidence is 0.9, and its corresponding first target category is fear, which belongs to the category of excitement. It can be determined that the first emotion weighted value is 0.9.

[0172] In some embodiments, there is a time point B, including three video frames: a first video frame, a second video frame, and a third video frame. The first target category of the first video frame is sadness, with a confidence of 0.8; the first target category of the second video frame is surprise, with a confidence of 0.4; the first target category of the third video frame is anger, with a confidence of 0.7. The maximum confidence is 0.8, and its corresponding first target category is sadness, which belongs to the ordinary category, and the first emotion weighted value can be determined to be 0.

[0173] In some embodiments, for the targeted time point, when determining the target confidence corresponding to the speech emotion, the computer device can determine the confidence of the second target category corresponding to the dialogue audio at that time point, determine the second emotion weighted value based on the maximum confidence and the second target category corresponding to the maximum confidence, and use the second emotion weighted value as the target confidence of the speech emotion.

[0174] Among them, the second target category of the speech audio is the category corresponding to the calculated maximum confidence of the speech audio. For example, for the speech audio, the confidence of the speech emotion category of the speech audio belonging to the surprise category is 0.7, the confidence of belonging to the fear category is 0.2, the confidence of belonging to the happiness category is 0.9, the confidence of belonging to the crying category is 0.1, the confidence of belonging to the shouting category is 0.6, the confidence of belonging to the calm category is 0.7; the confidence of belonging to the sadness category is 0.4, and the confidence of belonging to the fear category is 0.5. It can be seen that the second target category of the speech audio is the happiness category.

[0175] In some embodiments, when the computer device determines the second emotion weighted value based on the maximum confidence and the second target category corresponding to the maximum confidence, if the second target category corresponding to the maximum confidence belongs to the excitement category, the maximum confidence can be directly used as the second emotion weighted value, thereby weighting the second coded data through the second emotion weighted value. If the second target category corresponding to the maximum confidence belongs to the bland category, the computer device can set the second emotion weighted degree to 0, or assign a smaller value to the second emotion weighted value, such as less than a set threshold value, which can be 0.1, 0.25, etc., thereby down-weighting the second coded data through the second emotion weighted value.

[0176] In some embodiments, there is a time point C, and the second target category of the speech emotion category of the corresponding dialogue audio is anger, and the confidence is 0.8. Anger belongs to the excitement category, and the second emotion weighted value can be determined to be 0.8.

[0177] In some embodiments, there is a time point D, and the second target category of the speech emotion category of the corresponding dialogue audio is sadness, with a confidence level of 0.9. Sadness belongs to the bland category, and the second emotion weighted value can be determined to be 0.

[0178] In the above embodiment, for each time point, the computer device can determine the first emotion recognition result and the first coding data of the time point to obtain the video picture coding features. The computer device can also determine the second emotion recognition result and the second coding data of the time point to obtain the text coding features, so that a certain weight can be given to the corresponding coding data based on the emotion recognition results, and the representation of the modality of each time point can be improved, so as to adjust the feature proportion when calculating the suspense score, so as to improve the accuracy of the final suspense score calculation. Finally, by fusing the video picture coding features and the text coding features, a multimodal fusion feature is obtained. Based on the multimodal fusion feature, the accuracy of the suspense score calculation can be further improved.

[0179] In some embodiments, the multimodal data also includes interactive data, and video screen coding features are obtained based on the first coding data and the first emotion recognition result, including: determining the video screen coding features based on the first coding data, the first emotion recognition result, and the interactive data; and text coding features are obtained based on the second coding data and the second emotion recognition result, including: determining the text coding features based on the second coding data, the second emotion recognition result, and the interactive data.

[0180] Among them, interactive data is the data generated by the object (such as a user) interacting on the video playback platform based on the video to be processed, for example, it may be barrage data, comment data, etc.

[0181] Barrage data is the commentary subtitles that pop up in the video to be processed. Barrage data can give users a sense of "real-time interaction". Although the sending time of different barrage data is different, the barrage data sent at the same time basically have the same theme. At the same time, because it is the user's autonomous behavior, barrage data can represent the popularity of the video to be processed to a certain extent.

[0182] Specifically, for each time point, the computer device can determine the first emotion recognition result, the first coding data and the interaction data at the time point, determine the target confidence corresponding to the facial emotion through the first emotion recognition result, and weight or downgrade the first coding data based on the target confidence and the interaction data to obtain the video picture coding feature. The computer device can also determine the second emotion recognition result and the second coding data at the time point, determine the target confidence corresponding to the voice emotion through the second emotion recognition result, and weight or downgrade the second coding data based on the target confidence and the interaction data to obtain the text coding feature.

[0183] In some embodiments, the computer device may obtain interactive data generated during the historical playback of the video to be processed. If the video to be processed is a newly released video, there may be no interactive data when it is first released. In this case, a model for predicting the trend of barrage data can be established to simulate the trend of barrage data.

[0184] In some embodiments, Fig.10 As shown, when the interactive data is barrage data, the barrage data trend diagram obtained by the computer device. At each time point, there is a corresponding amount of barrage data. The computer device can normalize the amount of barrage data at each time point, normalize the amount of barrage data to a value between 0 and 1, and then use this value to weight the first coded data in combination with the first emotion recognition result, and weight the second coded data in combination with the second emotion recognition result.

[0185] In the above embodiment, the computer device introduces interactive data as a basis for judging the user's actual viewing reaction, thereby allowing the suspense calculation to introduce actual interactive data. To a certain extent, the barrage data can be used to correct the final located suspense point, so that the content at the final selected target time point is more in line with the actual user's suspense perception, thereby improving the suspense accuracy of the target time point, allowing users to have more interest in watching the feature film.

[0186] In some embodiments, the first emotion recognition result includes the first target category corresponding to each of the N video frames at the targeted time point, and the confidence of each of the N first target categories; N is a natural number greater than 1; according to the first coding data, the first emotion recognition result, and the interaction data, the video picture coding features are determined, including: determining the maximum confidence among the confidences of the N first target categories; determining the first emotion weighted value based on the maximum confidence and the first target category corresponding to the maximum confidence; determining the first interaction weighted value according to the interaction data; determining the video picture coding features according to the first coding data, the first emotion weighted value and the first interaction weighted value at the targeted time point.

[0187] Specifically, for any time point, the way in which the computer device determines the first emotion weighted value at the time point can refer to the way in which the first emotion weighted value is determined in the above embodiment when the interaction data is not introduced, and will not be repeated here. The computer device can perform normalization processing based on the determined interaction data, use the result of the normalization processing as the first interaction weighted value, and multiply the first coded data at the time point, the first emotion weighted value, and the first interaction weighted value to determine the video picture coding features.

[0188] In the above embodiment, for any time point, the computer device determines the video picture coding features by multiplying the first coding data, the first emotion weighted value and the first interaction weighted value based on the time point, fully considering the correlation between different modalities and improving the accuracy of suspense score calculation.

[0189] In some embodiments, a first emotion weighted value is determined based on the maximum confidence and the first target category corresponding to the maximum confidence, including: when the first target category corresponding to the maximum confidence belongs to the excitement category, the first emotion weighted value is determined based on the maximum confidence; when the first target category corresponding to the maximum confidence belongs to the neutral category, a preset value is used as the first emotion weighted value.

[0190] Specifically, when the computer device determines the first emotion weighted value based on the maximum confidence and the first target category corresponding to the maximum confidence, if the first target category corresponding to the maximum confidence belongs to the excitement category, the maximum confidence can be directly used as the first emotion weighted value, thereby weighting the first coded data through the first emotion weighted value. If the first target category corresponding to the maximum confidence belongs to the bland category, the computer device can set the first emotion weighted value to 0, or assign a smaller value to the first emotion weighted value, such as less than a set threshold value, which can be 0.2, 0.3, etc., thereby down-weighting the first coded data through the first emotion weighted value.

[0191] In the above embodiment, the computer device sets different first emotion weighting values ​​based on the first target category corresponding to the maximum confidence level, and can adjust the feature proportion during calculation. Since suspense is often generated in excited expressions, when the first target category belongs to the excited category, the first coded data is weighted by the first emotion weighting value, and when the first target category belongs to the plain category, the first coded data is down-weighted by the first emotion weighting value, which can improve the accuracy of the final suspense calculation.

[0192] In some embodiments, the second emotion recognition result includes a second target category corresponding to the dialogue audio at the targeted time point, and the confidence of the second target category; text encoding features are determined based on the second coding data, the second emotion recognition result, and the interaction data, including: determining a second emotion weighted value based on the confidence of the second target category; determining a second interaction weighted value based on the interaction data; determining text encoding features based on the second coding data, the second emotion weighted value, and the second interaction weighted value at the targeted time point.

[0193] Specifically, for any time point, the way in which the computer device determines the second emotion weighted value at the time point can refer to the way in which the second emotion weighted value is determined in the above embodiment when the interaction data is not introduced, and will not be repeated here. The computer device can perform normalization processing based on the determined interaction data, use the result of the normalization processing as the second interaction weighted value, and multiply the second coded data at the time point, the second emotion weighted value, and the second interaction weighted value to determine the text coding feature.

[0194] In some embodiments, when the computer device determines the second emotion weighted value based on the maximum confidence and the second target category corresponding to the maximum confidence, if the second target category corresponding to the maximum confidence belongs to the excitement category, the maximum confidence can be directly used as the second emotion weighted value, thereby weighting the second coded data through the second emotion weighted value. If the second target category corresponding to the maximum confidence belongs to the bland category, the computer device can set the second emotion weighted value to 0, or assign a smaller value to the second emotion weighted value, such as less than a set threshold value, which can be 0.1, 0.25, etc., thereby down-weighting the second coded data through the second emotion weighted value.

[0195] In some embodiments, Fig.11 As shown, this is a structural diagram for feature weighting:

[0196] in, Fig.11The first emotion recognition results shown in the figure include: excited expressions and neutral expressions. The excited expressions include: surprise, fear, laughter, crying, grimace, and anger; the neutral expressions include: no expression, smile, and sadness. The second emotion recognition results also include: excited expressions and neutral expressions. The excited expressions include: anger, surprise, happiness, crying, shouting, sadness, and fear; the neutral expressions include: neutral and sadness.

[0197] Take the determination of the weighted value of the first emotion at any time point as an example for explanation: when determining the weighted value of the first emotion based on the first emotion recognition result, multiple first target categories and the confidence of each first target category can be determined for the time point. The computer device can maximize the confidence of the category of excited expression in each first target category, and also maximize the confidence of the category of neutral expression in each first target category, and compare the two obtained confidences. If the confidence of the excited expression is larger, the final confidence is equal to the confidence of the excited expression. If the confidence of the neutral expression is larger, the final confidence is equal to zero.

[0198] Furthermore, after calculating the first emotion weighted value at the time point, the computer device can multiply the first emotion weighted value at the time point with the first coding data, i.e., the vector representation (embedding), and then multiply it with the normalized barrage data to obtain the weighted video picture coding features. The text coding features are also weighted in the same way, and finally the two weighted feature coding sequences are concatted together, and the suspense score is calculated based on the concatenation result.

[0199] In the above embodiment, for any time point, the computer device determines the text coding features by multiplying the second coding data, the second emotion weighted value and the second interaction weighted value based on the time point, fully considering the correlation between different modalities and improving the accuracy of the suspense score calculation.

[0200] In some embodiments, reference Fig.12 As shown, it is a system architecture diagram of suspense judgment in the present application in one embodiment. The video to be processed in the system architecture diagram is a video of a TV series, wherein: Fig.12 A total of four modal data are involved, namely video images, dialogue text, dialogue audio, and the barrage trend curve provided by the platform. The barrage trend curve can be used to determine the amount of barrage data at each time point.

[0201] The video screen can be input into a video coding network for feature coding to obtain first coded data. The video screen can correspond to facial emotion recognition, and after facial emotion recognition, a first emotion recognition result corresponding to the video screen is identified, so that the first coded data can be weighted based on the first emotion recognition result. When the first emotion recognition results obtained in facial emotion recognition are all flat expressions, the computer device can remove the time point by filtering the time period, that is, the time point does not participate in the weighting.

[0202] The dialogue text can be input into the text encoding network for feature encoding to obtain the second encoded data. The time audio can correspond to the tone and emotion recognition. After the voice emotion recognition, the second emotion recognition result corresponding to the dialogue audio is identified, so that the second encoded data can be weighted based on the second emotion recognition result. When the second emotion recognition results obtained in the tone and emotion recognition are all flat expressions, the computer device can remove the time point by filtering the time period, that is, the time point does not participate in the weighting.

[0203] Among them, the first emotion recognition result obtained by performing facial emotion recognition based on the video screen can be combined with the amount of barrage data determined by the trend of the barrage curve to weight the first coded data. The second emotion recognition result based on the voice emotion recognition of the lines audio can be combined with the amount of barrage data determined by the trend of the barrage curve to weight the second coded data. Finally, the two weighted results are combined with the barrage trend curve to calculate the result of suspense judgment, so that the suspense score can be calculated for each time point in the video.

[0204] In some embodiments, an integrity analysis is performed based on the multimodal data of each candidate time point, and a target time point is determined from the candidate time points based on the integrity analysis results, including: determining multiple storyboards in the video to be processed based on the video screen; determining candidate time points that do not meet the integrity conditions among the candidate time points based on the multiple storyboards, dialogue audio, and dialogue text; and eliminating the candidate time points that do not meet the integrity conditions to obtain the target time point.

[0205] The storyboard is a micro-unit of the video. For a video to be processed, it can include only one storyboard or multiple storyboards, all of which can together form the entire video. The specific number of storyboards is related to the actual scene and length of the video.

[0206] For the same storyboard, the video frame contents included therein are strongly correlated, that is, the video frame contents will have certain similarities; for different storyboards, the video frame contents therein are relatively different.

[0207] The integrity condition is a condition set to determine whether the content of the candidate time point satisfies the logical integrity. When setting the integrity condition, it can be set in combination with the storyboard information of the storyboard of the video to be processed, the time when the line audio is located in the video to be processed, and the time when the line text is located in the video to be processed.

[0208] Specifically, the computer device can determine the video frame sequence of the video to be processed based on the video screen, calculate the similarity between the video frames included in the video screen based on the video frame sequence, and determine multiple storyboards in the video to be processed according to the similarity. Further, the computer device can determine whether the candidate time point position meets the integrity condition based on the determined multiple storyboards, line audio and line text, retain the candidate time point position that meets the integrity condition; eliminate the candidate time point position that does not meet the integrity condition, and obtain the target time point position.

[0209] In some embodiments, when calculating the storyboard based on the video frame sequence, the video frame sequence includes a plurality of video frames arranged in time order. For any two adjacent video frames in the video frame sequence, the correlation between the two adjacent video frames is calculated, thereby obtaining the correlation between any two adjacent video frames. Then, it is determined whether the two adjacent video frames belong to the same storyboard based on the correlation, and if they do not belong to the same storyboard, a storyboard dividing line between the two storyboards is obtained.

[0210] In some embodiments, if the correlation between two adjacent video frames is lower than a correlation threshold, it means that the correlation between the two adjacent video frames is particularly low, and the two video frames are more likely to belong to different storyboards. Therefore, the time point corresponding to the two adjacent video frames is determined as the storyboard dividing line between the two adjacent storyboards, thereby dividing the video to be processed into multiple storyboards based on the storyboard dividing line.

[0211] In some embodiments, when determining the split shot, the computer device can calculate the split shot only based on the video frame of the candidate point, so that the determined split shot is only related to the candidate time point, effectively reducing the amount of calculation. It is understandable that the computer device can also calculate the split shot based on the video frame of the entire video to be processed, so that the determined split shot is more complete and the accuracy of subsequent judgment is improved.

[0212] In the above embodiment, the computer device performs integrity analysis by combining multiple storyboards, dialogue audio and dialogue text, eliminates candidate time points that do not meet the integrity conditions, and obtains the target time point. This can ensure that the video content at the target time point is complete, avoid abrupt situations at the target time point, and improve the overall quality of subsequent video materials and user viewing interest.

[0213] In some embodiments, based on multiple storyboards, dialogue audio and dialogue text, candidate time points in the candidate time points that do not meet the integrity condition are determined, including: for any storyboard, the candidate time points in a preset part of the targeted storyboard are used as candidate time points that do not meet the integrity condition; based on the dialogue audio, a first time period in which the dialogue appears in the video to be processed is determined, and the candidate time points in the first time period are used as candidate time points that do not meet the integrity condition; based on the dialogue text, a second time period in which the dialogue appears in the video to be processed is determined, and the candidate time points in the second time period are used as candidate time points that do not meet the integrity condition.

[0214] Among them, the preset part of the storyboard is determined according to the plot of the storyboard. Generally speaking, the first half of the storyboard represents that the plot has just begun, and the end often represents that the plot in this storyboard has basically ended. If the final target time point is at the beginning or interruption of the storyboard, it will often cause a truncation of the plot, resulting in a more abrupt ending of the final short video editing. Therefore, the preset part of the storyboard is the beginning or interruption of the storyboard.

[0215] The first time period is the time interval formed by the paragraphs in which the speech audio appears. For each candidate time point, the speech audio corresponds to the first time period.

[0216] The second time period is the time interval formed by the paragraphs in which the dialogue text appears. For each candidate time point, the dialogue text corresponds to a second time period.

[0217] Specifically, for any storyboard, the computer device can determine the candidate time point in the preset part of the storyboard as a time point that does not meet the integrity condition, the candidate time point in the first time period as a candidate time point that does not meet the integrity condition, and the candidate time point in the second time period as a candidate time point that does not meet the integrity condition, so as to ensure that the target time point cannot appear in the time segment where voice and text appear.

[0218] In some embodiments, when setting the preset part of the storyboard, the storyboard line of the storyboard can be used as a reference to determine each storyboard unit, and then the middle position of the storyboard is determined in combination with the time information of each storyboard, that is, the start point of the storyboard and the end point of the storyboard, and the part before the middle position is set as the preset part of the storyboard. It is understandable that the one-third or one-quarter of the storyboard can also be determined in combination with the time information of each storyboard, and the part before the one-third or one-quarter is set as the preset part of the storyboard. The setting of the specific preset part can be adaptively adjusted in combination with the actual storyboard content, application scenario, etc.

[0219] In some embodiments, the computer device performs a completeness analysis based on multiple storyboards, dialogue audio, and dialogue text, and eliminates candidate time points that do not meet the completeness condition. Fig.13 In the figure, all arrows represent candidate time points determined by suspense scores. Among them, the dotted arrows represent the candidate time points eliminated during the screening process, and the solid arrows represent the candidate time points that are finally selected with completeness and suspense.

[0220] In the above embodiment, the computer device compares the candidate time point with the preset part of the storyboard, compares the candidate time point with the first time period and the second time period respectively, eliminates the candidate time point that does not meet the integrity condition, and obtains the target time point, so as to accurately determine the target time point with integrity and suspense.

[0221] In some embodiments, the video data processing method further includes: determining the video end point as a candidate editing point, and performing video editing based on the candidate editing point; or, marking the plot according to the video end point; the plot marking includes marking the video end point in the video playback progress bar.

[0222] Among them, the obtained video ending point can be used as a candidate point for video editing, and can also be used for plot annotation. For example, in many short video creation platforms, many users manually select clips when editing short videos, that is, they manually watch the material collection and select the appropriate material entry point and ending point. In this application, the video ending point suitable for short video editing in the video is quickly and accurately located automatically, and can be directly given to the user during creation. Then the user does not need to make a selection after watching, and can directly select from the video ending points recommended by this application, which greatly improves the efficiency of creation and can improve the efficiency of the short video creation platform.

[0223] For example, in a long video online platform, in this application, the video ending point suitable for short video editing is located quickly and accurately through automation, and the video ending point is displayed in the progress bar of the platform's film and television drama playback. This video ending point is used as the dividing line between different theme plots in the series, so that the audience can choose to skip certain plots and watch them directly, which can greatly enhance the audience's viewing experience.

[0224] For example, in the later stage of film and television drama production, relevant highlight trailers and other clips will be produced according to the distribution of episodes of the main film and television drama. In this application, the video ending point suitable for short video editing is located quickly and accurately through automation, which can be used as a candidate for the short video ending point of the highlight trailer, which can ensure the logical completeness of the trailer plot while retaining suspense at the end, greatly attracting users' viewing interest, and attracting audiences to watch the main film on the platform, so as to promote and divert traffic for the long video platform.

[0225] The present application also provides an application scenario, which applies the above-mentioned video data processing method.

[0226] Specifically, the application of the video data processing in this application scenario is as follows:

[0227] refer to Fig.14 As shown, it is an overall system architecture diagram of the present application in an embodiment. The whole system is divided into two parts, namely, suspense judgment and completeness judgment. The left part is the video of the TV series, that is, the location of the suspenseful content points in the video to be processed, which involves four dimensional modal data, namely, video picture, line text, time audio and bullet screen trend curve provided by the platform side.

[0228] Among them, the video screen can be mapped to facial emotion recognition. After identifying the strong expression, the features encoded in the video frame are weighted.

[0229] Among them, the first emotion recognition result obtained by performing facial emotion recognition based on the video screen can be combined with the amount of barrage data determined by the trend of the barrage curve to weight the first coded data. The second emotion recognition result based on the voice emotion recognition of the lines audio can be combined with the amount of barrage data determined by the trend of the barrage curve to weight the second coded data. Finally, the two weighted results are combined with the barrage trend curve to calculate the result of suspense judgment, so that the suspense score can be calculated for each time point in the video.

[0230] The completeness judgment on the right uses information from three modalities, namely, the completeness judgment from the three complementary modalities of dialogue audio, video screen and dialogue text.

[0231] By judging the completeness of the ending point located in the left module through the right side, it is possible to screen out the target time point that has both completeness and suspense. The target time point can be used to determine the ending point of the short video material.

[0232] The embodiment of the present application constructs a fully automatic method for locating the ending point of short video materials for film and television dramas. In the entire process, the appropriate ending point is completely automatically located, and the content at the ending point can have a certain logical integrity and suspense, completely getting rid of manual participation, greatly improving the efficiency of editing, and enabling the entire system to be industrialized. At the same time, because this solution uses the established neural network model to predict the ending point, the positioning calculation is completely standardized and will not be differentiated due to different personnel.

[0233] The embodiment of the present application innovatively uses multimodal data, including facial expressions, dialogue text, dialogue audio, interactive data, etc. By combining multiple modal data together, it can respond to different types of film and television drama themes, so that the located ending point has a certain robustness, and by using multiple modal data together to locate the final suspense point, the located suspense point can be made more exciting and accurate.

[0234] The embodiment of the present application innovatively applies the correlation between different modes to the model, and adjusts the feature proportion in the final calculation by weighting different modes, thereby improving the accuracy of the final suspense calculation judgment.

[0235] The embodiment of the present application innovatively introduces barrage data as a basis for judging the user's actual viewing reaction, thereby allowing suspense positioning to introduce actual behavioral performance data, which can be more in line with the actual audience's suspense perception, thereby improving the suspense accuracy at the target time point, and allowing users to have more interest in watching the feature film.

[0236] In addition to locating suspense points in film and television dramas, the embodiments of the present application also combine data from multiple modes to determine whether the content at the candidate time point has a logically complete performance. By determining the logical integrity of the candidate time point, the integrity of the video content at the end can be ensured, avoiding abrupt endings at the video, thereby improving the overall quality of the short video material and increasing users' viewing interest.

[0237] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0238] Based on the same inventive concept, the embodiment of the present application also provides a video data processing device for implementing the video data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more video data processing device embodiments provided below can refer to the limitations of the video data processing method above, and will not be repeated here.

[0239] In one embodiment, Fig.15 As shown, a video data processing device 1500 is provided, comprising: a data determination module 1502, an identification module 1504, a calculation module 1506, a screening module 1508 and a positioning module 1510, wherein:

[0240] The data determination module 1502 is used to determine multiple time points of the video to be processed and obtain multimodal data corresponding to each time point, wherein the multimodal data at least includes video images, dialogue audio and dialogue text.

[0241] The recognition module 1504 is used to perform emotion recognition at any time point based on the video image and the line audio at the time point to obtain an emotion recognition result.

[0242] The calculation module 1506 is used to determine the suspense score of the time point according to the video picture, the dialogue text and the emotion recognition result at the time point.

[0243] The screening module 1508 is used to screen candidate time points from multiple time points according to the suspense score of each time point.

[0244] The positioning module 1510 is used to perform integrity analysis based on the multimodal data of each candidate time point, and determine the target time point from the candidate time point based on the integrity analysis result; the target time point is used to determine the end point of the video. In some embodiments, the recognition module 1504 is also used to perform facial emotion recognition based on the video screen at any time point to obtain a first emotion recognition result; for any time point, perform voice emotion recognition based on the line audio at the time point to obtain a second emotion recognition result.

[0245] In some embodiments, the identification module 1504 includes a first identification module;

[0246] The first recognition module is used to perform facial detection at any time point based on the video picture at the time point, and capture a facial image from the video picture based on the detection result; perform feature extraction based on the facial image to obtain a vector representation of the facial image; predict the facial expression category to which the facial image belongs based on the vector representation of the facial image, and determine a first emotion recognition result based on the facial expression category.

[0247] In some embodiments, the identification module 1504 further includes a second identification module;

[0248] The second recognition module is used to extract the acoustic feature representation of the dialogue audio at any time point; encode based on the acoustic feature representation to obtain the encoded feature output; perform speech recognition based on the encoded feature output to obtain the speech recognition result at the time point; when the speech recognition result indicates the presence of speech, perform speech emotion recognition based on the encoded feature output to obtain a second emotion recognition result.

[0249] In some embodiments, the second recognition module is also used to perform a first encoding based on the acoustic feature representation to obtain a first encoding output; the first encoding is an encoding that focuses on time domain perception; the second encoding is performed based on the acoustic feature representation to obtain a second encoding output; the second encoding is a multidimensional encoding; the third encoding is performed based on the acoustic feature representation to obtain a third encoding output; the third encoding is an encoding that focuses on frequency domain perception; and the encoding feature output is determined based on the first encoding output, the second encoding output, and the third encoding output.

[0250] In some embodiments, the second recognition module is also used to first extract features in the time dimension and then extract features in the frequency dimension on the acoustic feature representation through a convolution kernel of a first size to obtain a first intermediate feature; extract features on the acoustic feature representation through a convolution kernel of a second size to obtain a second intermediate feature; the second size is larger than the first size; and the first intermediate feature and the second intermediate feature are fused to obtain a first encoded output.

[0251] In some embodiments, the second recognition module is also used to first extract features in the frequency dimension and then extract features in the time dimension on the acoustic feature representation through a convolution kernel of a first size to obtain a third intermediate feature; extract features on the acoustic feature representation through a convolution kernel of a second size to obtain a second intermediate feature; the second size is larger than the first size; and the third intermediate feature and the second intermediate feature are fused to obtain a third encoded output.

[0252] In some embodiments, the emotion recognition result includes a first emotion recognition result and a second emotion recognition result; the calculation module 1506 is also used to obtain the first coding data of the video screen at the targeted time point, and obtain the video screen coding feature according to the first coding data and the first emotion recognition result; obtain the second coding data of the dialogue text at the targeted time point, and obtain the text coding feature according to the second coding data and the second emotion recognition result; fuse the video screen coding feature and the text coding feature at the targeted time point to obtain the multimodal fusion feature at the targeted time point; determine the suspense score at the targeted time point according to the multimodal fusion feature.

[0253] In some embodiments, the calculation module 1506 is further used to determine the video picture coding features according to the first coding data, the first emotion recognition result, and the interaction data; and is also used to determine the text coding features according to the second coding data, the second emotion recognition result, and the interaction data. In some embodiments, the first emotion recognition result includes the first target category corresponding to each of the N video frames at the time point, and the confidence of each of the N first target categories; N is a natural number greater than 1; the calculation module 1506 is also used to determine the maximum confidence among the confidences of the N first target categories; determine the first emotion weighted value based on the maximum confidence and the first target category corresponding to the maximum confidence; determine the first interaction weighted value according to the interaction data; determine the video picture coding features according to the first coding data, the first emotion weighted value, and the first interaction weighted value at the time point.

[0254] In some embodiments, the calculation module 1506 is also used to determine the first emotion weighted value based on the maximum confidence when the first target category corresponding to the maximum confidence belongs to the excitement category; and to use a preset value as the first emotion weighted value when the first target category corresponding to the maximum confidence belongs to the neutral category.

[0255] In some embodiments, the second emotion recognition result includes the second target category corresponding to the dialogue audio at the targeted time point, and the confidence of the second target category; the calculation module 1506 is also used to determine the second emotion weighted value based on the confidence of the second target category; determine the second interaction weighted value according to the interaction data; determine the text coding features according to the second coding data, the second emotion weighted value and the second interaction weighted value at the targeted time point.

[0256] In some embodiments, the screening module 1508 is further used to perform curve fitting according to the suspense score of each time point to obtain a target fitting curve; and determine the time point corresponding to the peak in the target fitting curve as the candidate time point.

[0257] In some embodiments, the positioning module 1510 is also used to determine multiple shots in the video to be processed based on the video screen; determine the candidate time points that do not meet the integrity condition among the candidate time points based on the multiple shots, dialogue audio and dialogue text; eliminate the candidate time points that do not meet the integrity condition to obtain the target time point.

[0258] In some embodiments, the positioning module 1510 is also used to, for any storyboard, use the candidate time points in the preset part of the storyboard as candidate time points that do not meet the completeness condition; determine the first time period in which the lines appear in the video to be processed based on the line audio, and use the candidate time points in the first time period as candidate time points that do not meet the completeness condition; determine the second time period in which the lines appear in the video to be processed based on the line text, and use the candidate time points in the second time period as candidate time points that do not meet the completeness condition.

[0259] In some embodiments, the video data processing apparatus further includes an application module;

[0260] The application module is used to determine the video end point as a candidate editing point and perform video editing based on the candidate editing point; and is also used to mark the plot according to the video end point; the plot marking includes marking the video end point in the video playback progress bar.

[0261] Each module in the above video data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0262] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.16As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store video data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video data processing method is implemented.

[0263] Those skilled in the art will understand that Fig.16 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0264] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0265] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0266] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0267] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0268] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0269] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0270] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video data processing method, characterized in that: The method comprises: Determine multiple time points of the video to be processed, and obtain multimodal data corresponding to each time point, wherein the multimodal data includes at least video images, dialogue audio, and dialogue text; For any time point, emotion recognition is performed based on the video image and the audio of the lines at the time point to obtain the emotion recognition result; Determine the suspense score of the targeted time point according to the video images, dialogue texts and emotion recognition results of the targeted time point; Screening candidate time points from the multiple time points according to the suspense score of each time point; An integrity analysis is performed based on the multimodal data of each candidate time point, and a target time point is determined from the candidate time points based on the integrity analysis results; the target time point is used to determine the end point of the video; the integrity analysis is used to analyze whether the plot of the video clip obtained by segmentation based on the candidate time points is logically complete.

2. The method according to claim 1, characterized in that The emotion recognition result includes a first emotion recognition result and a second emotion recognition result. For any time point, expression recognition is performed according to the video image and the line audio at the time point to obtain the emotion recognition result, including: For any time point, facial emotion recognition is performed based on the video image at the time point to obtain a first emotion recognition result; At any time point, speech emotion recognition is performed based on the dialogue audio at the time point to obtain a second emotion recognition result.

3. The method according to claim 2, characterized in that The step of performing facial emotion recognition at any time point according to the video picture at the time point to obtain a first emotion recognition result includes: For any time point, perform face detection based on the video picture at the time point, and capture a face image from the video picture based on the detection result; Performing feature extraction based on the facial image to obtain a vector representation of the facial image; The facial expression category to which the facial image belongs is predicted according to the vector representation of the facial image, and a first emotion recognition result is determined according to the facial expression category.

4. The method according to claim 2, characterized in that: The step of performing speech emotion recognition at any time point according to the speech audio at the time point to obtain a second emotion recognition result includes: For any time point, extract the acoustic feature representation of the dialogue audio at the time point; Encoding based on the acoustic feature representation to obtain a coded feature output; Perform speech recognition according to the coded feature output to obtain a speech recognition result at the time point; In a case where the speech recognition result indicates the presence of speech, speech emotion recognition is performed based on the coded feature output to obtain a second emotion recognition result.

5. The method according to claim 4, characterized in that The encoding based on the acoustic feature representation to obtain the encoded feature output includes: Performing a first encoding based on the acoustic feature representation to obtain a first encoding output; the first encoding is an encoding focusing on time domain perception; Performing a second encoding based on the acoustic feature representation to obtain a second encoding output; the second encoding is a multi-dimensional encoding; Performing a third encoding based on the acoustic feature representation to obtain a third encoding output; the third encoding is an encoding focusing on frequency domain perception; A coding feature output is determined based on the first coding output, the second coding output, and the third coding output.

6. The method according to claim 5, characterized in that The performing first encoding based on the acoustic feature representation to obtain a first encoding output includes: Using a convolution kernel of a first size, the acoustic feature representation is first subjected to feature extraction in a time dimension, and then subjected to feature extraction in a frequency dimension, to obtain a first intermediate feature; Extracting features from the acoustic feature representation using a convolution kernel of a second size to obtain a second intermediate feature; the second size is larger than the first size; The first intermediate feature and the second intermediate feature are fused to obtain a first encoding output.

7. The method according to claim 5, characterized in that The performing a third encoding based on the acoustic feature representation to obtain a third encoding output includes: Using a convolution kernel of a first size, the acoustic feature representation is first subjected to feature extraction in the frequency dimension, and then subjected to feature extraction in the time dimension, to obtain a third intermediate feature; Extracting features from the acoustic feature representation using a convolution kernel of a second size to obtain a second intermediate feature; the second size is larger than the first size; The third intermediate feature and the second intermediate feature are fused to obtain a third encoding output.

8. The method according to claim 1, characterized in that: The emotion recognition result includes a first emotion recognition result and a second emotion recognition result; and determining the suspense score at the time point according to the video picture, the dialogue text and the emotion recognition result at the time point includes: Acquire first coded data of the video picture at the time point, and obtain a coded feature of the video picture according to the first coded data and the first emotion recognition result; Acquire second encoding data of the dialogue text at the time point, and obtain text encoding features according to the second encoding data and the second emotion recognition result; Fusion of video picture coding features and text coding features at the targeted time point to obtain multimodal fusion features at the targeted time point; The suspense score for the time point is determined according to the multimodal fusion features.

9. The method according to claim 8, characterized in that The multimodal data also includes interactive data, and the video picture coding features are obtained according to the first coding data and the first emotion recognition result, including: Determine video picture coding features according to the first coding data, the first emotion recognition result, and the interaction data; The obtaining of text encoding features according to the second encoding data and the second emotion recognition result includes: Determine text encoding features based on the second encoding data, the second emotion recognition result, and the interaction data.

10. The method according to claim 9, characterized in that The first emotion recognition result includes the first target category corresponding to each of the N video frames at the time point, and the confidence of each of the N first target categories; N is a natural number greater than 1; The determining of video picture coding features according to the first coding data, the first emotion recognition result, and the interaction data includes: Determine the maximum confidence among the confidences of the N first target categories; Determining a first emotion weighted value based on the maximum confidence and a first target category corresponding to the maximum confidence; Determining a first interaction weighted value according to the interaction data; The video picture coding features are determined according to the first coding data at the time point, the first emotion weighted value and the first interaction weighted value.

11. The method according to claim 10, characterized in that Determining a first emotion weighted value based on the maximum confidence and a first target category corresponding to the maximum confidence includes: In a case where the first target category corresponding to the maximum confidence level belongs to the excitement category, determining a first emotion weighted value based on the maximum confidence level; In the case that the first target category corresponding to the maximum confidence level belongs to the bland category, a preset value is used as the first emotion weighted value.

12. The method according to claim 9, wherein the second emotion recognition result comprises a second target category corresponding to the speech audio at the time point, and a confidence level of the second target category; and determining the text encoding feature according to the second encoding data, the second emotion recognition result, and the interaction data comprises: Determining a second emotion weighted value based on the confidence of the second target category; determining a second interaction weighted value according to the interaction data; The text coding feature is determined based on the second coding data for the targeted time point, the second emotion weighted value and the second interaction weighted value.

13. The method according to claim 1, characterized in that The step of selecting candidate time points from the multiple time points according to the suspense score of each time point includes: Performing curve fitting according to the suspense scores at each time point to obtain a target fitting curve; The time point corresponding to the peak in the target fitting curve is determined as the candidate time point.

14. The method according to claim 1, characterized in that The step of performing integrity analysis on the multimodal data of each candidate time point and determining the target time point from the candidate time points according to the integrity analysis result includes: Determine, according to the video screen, a plurality of storyboards in the video to be processed; Based on the multiple storyboards, the dialogue audio, and the dialogue text, determining the candidate time points that do not meet the completeness condition among the candidate time points; Eliminate the candidate time points that do not meet the integrity condition and obtain the target time point.

15. The method according to claim 14, characterized in that The determining, based on the plurality of storyboards, the dialogue audio, and the dialogue text, the candidate time points that do not satisfy the completeness condition among the candidate time points includes: For any storyboard, the candidate time points in the preset part of the storyboard are regarded as candidate time points that do not meet the integrity condition; Determine a first time period in which the speech appears in the video to be processed based on the speech audio, and use the candidate time points in the first time period as candidate time points that do not meet the integrity condition; A second time period in which the lines appear in the video to be processed is determined based on the lines text, and candidate time points in the second time period are used as candidate time points that do not meet the integrity condition.

16. The method according to any one of claims 1 to 15, characterized in that: The method further comprises: Determine the video ending point as a candidate editing point, and perform video editing based on the candidate editing point; or, Plot marking is performed according to the end point of the video; the plot marking includes marking the end point of the video in the playback progress bar of the video.

17. A video data processing device, characterized in that: The device comprises: A data determination module, used to determine multiple time points of the video to be processed, and obtain multimodal data corresponding to each time point, wherein the multimodal data at least includes video images, dialogue audio and dialogue text; The recognition module is used to perform emotion recognition at any time point based on the video image and the line audio at the time point to obtain the emotion recognition result; A calculation module, used to determine the suspense score of the targeted time point according to the video picture, the line text and the emotion recognition result of the targeted time point; A screening module, configured to screen candidate time points from the plurality of time points according to the suspense score of each time point; A positioning module is used to perform integrity analysis based on the multimodal data of each candidate time point, and determine the target time point from the candidate time point based on the integrity analysis result; the target time point is used to determine the end point of the video; the integrity analysis is used to analyze whether the plot of the video clip obtained by segmentation based on the candidate time point is logically complete.

18. The device according to claim 17, characterized in that The recognition module is also used to perform facial emotion recognition at any time point based on the video image at the time point to obtain a first emotion recognition result; and to perform voice emotion recognition at any time point based on the dialogue audio at the time point to obtain a second emotion recognition result.

19. The device according to claim 18, characterized in that The identification module comprises: The first recognition module is used to perform face detection at any time point based on the video picture at the time point, and capture a facial image from the video picture based on the detection result; perform feature extraction based on the facial image to obtain a vector representation of the facial image; predict the facial expression category to which the facial image belongs based on the vector representation of the facial image, and determine a first emotion recognition result based on the facial expression category.

20. The device according to claim 18, characterized in that The identification module also includes: The second recognition module is used to extract the acoustic feature representation of the dialogue audio at any time point; encode based on the acoustic feature representation to obtain a coded feature output; perform speech recognition based on the coded feature output to obtain a speech recognition result for the time point; and when the speech recognition result indicates the presence of speech, perform speech emotion recognition based on the coded feature output to obtain a second emotion recognition result.

21. The device according to claim 20, characterized in that The second recognition module is also used to perform a first encoding based on the acoustic feature representation to obtain a first encoding output; the first encoding is an encoding that focuses on time domain perception; a second encoding is performed based on the acoustic feature representation to obtain a second encoding output; the second encoding is a multidimensional encoding; a third encoding is performed based on the acoustic feature representation to obtain a third encoding output; the third encoding is an encoding that focuses on frequency domain perception; and a coding feature output is determined based on the first coding output, the second coding output, and the third coding output.

22. The device according to claim 21, characterized in that The second recognition module is further used to first extract features in the time dimension and then extract features in the frequency dimension of the acoustic feature representation using a convolution kernel of a first size to obtain a first intermediate feature; and extract features in the acoustic feature representation using a convolution kernel of a second size to obtain a second intermediate feature; The second size is larger than the first size; and the first intermediate feature and the second intermediate feature are fused to obtain a first encoding output.

23. The device according to claim 21, characterized in that The second recognition module is further used to first extract features in the frequency dimension and then extract features in the time dimension of the acoustic feature representation using a convolution kernel of a first size to obtain a third intermediate feature; and to extract features in the acoustic feature representation using a convolution kernel of a second size to obtain a second intermediate feature; The second size is larger than the first size; and the third intermediate feature and the second intermediate feature are fused to obtain a third encoding output.

24. The device according to claim 17, characterized in that The emotion recognition result includes a first emotion recognition result and a second emotion recognition result; the calculation module is also used to obtain first coding data of the video screen at the targeted time point, and obtain video screen coding features according to the first coding data and the first emotion recognition result; obtain second coding data of the dialogue text at the targeted time point, and obtain text coding features according to the second coding data and the second emotion recognition result; fuse the video screen coding features and text coding features at the targeted time point to obtain multimodal fusion features at the targeted time point; determine the suspense score at the targeted time point according to the multimodal fusion features.

25. The device according to claim 24, characterized in that The multimodal data also includes interactive data, and the calculation module is also used to determine the video picture encoding features based on the first encoding data, the first emotion recognition result, and the interactive data; and is also used to determine the text encoding features based on the second encoding data, the second emotion recognition result, and the interactive data.

26. The device according to claim 25, characterized in that The first emotion recognition result includes the first target category corresponding to each of the N video frames at the time point, and the confidence of each of the N first target categories; N is a natural number greater than 1; The calculation module is also used to determine the maximum confidence among the confidences of the N first target categories; Determining a first emotion weighted value based on the maximum confidence and a first target category corresponding to the maximum confidence; A first interaction weighting value is determined according to the interaction data; and a video picture coding feature is determined according to the first coding data at the targeted time point, the first emotion weighting value and the first interaction weighting value.

27. The device according to claim 26, characterized in that The calculation module is also used to determine the first emotion weighted value based on the maximum confidence when the first target category corresponding to the maximum confidence belongs to the excitement category; and to use a preset value as the first emotion weighted value when the first target category corresponding to the maximum confidence belongs to the calm category.

28. According to the device of claim 25, the second emotion recognition result includes a second target category corresponding to the dialogue audio at the targeted time point, and the confidence of the second target category; the calculation module is also used to determine a second emotion weighted value based on the confidence of the second target category; determine a second interaction weighted value according to the interaction data; determine text coding features according to the second coding data at the targeted time point, the second emotion weighted value and the second interaction weighted value.

29. The device according to claim 17, characterized in that The screening module is also used to perform curve fitting according to the suspense scores of each of the time points to obtain a target fitting curve; and determine the time point corresponding to the peak in the target fitting curve as a candidate time point.

30. The device according to claim 17, characterized in that The positioning module is also used to determine multiple shots in the video to be processed based on the video screen; determine the candidate time points that do not meet the integrity condition among the candidate time points based on the multiple shots, the dialogue audio and the dialogue text; and eliminate the candidate time points that do not meet the integrity condition to obtain the target time point.

31. The device according to claim 30, characterized in that The positioning module is further used to, for any storyboard, use the candidate time points in the preset part of the storyboard as candidate time points that do not meet the integrity condition; determine the first time period in which the lines appear in the video to be processed based on the line audio, and use the candidate time points in the first time period as candidate time points that do not meet the integrity condition; A second time period in which the lines appear in the video to be processed is determined based on the lines text, and candidate time points in the second time period are used as candidate time points that do not meet the integrity condition.

32. The device according to any one of claims 17 to 31, characterized in that The device also includes: an application module, which is used to determine the video ending point as a candidate editing point and perform video editing based on the candidate editing point; or, it is used to perform plot annotation according to the video ending point; the plot annotation includes marking the video ending point in the video playback progress bar.

33. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 16 are implemented.

34. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

35. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 16 are implemented.

Images