Video summary generation method and device

Abstract

The present application belongs to the field of artificial intelligence technology, and specifically relates to a method and device for generating a video summary. The embodiment of the present invention can be applied to vehicle-mounted scenes. The video summary generation method includes: obtaining multiple frames to be processed from the video to be processed; merging each frame to be processed according to the text content of the frame to be processed to obtain multiple segment intervals; determining the first emotional intensity of the segment interval according to the text content of the frame to be processed in each segment interval; determining the second emotional intensity of the segment interval according to the expression in the frame to be processed in each segment interval; performing a weighted summation operation on the first emotional intensity and the second emotional intensity to obtain an emotional score for the segment interval; according to the video time period corresponding to the segment interval whose emotional score is greater than or equal to a preset value, intercepting the video to be processed to obtain video segments, and splicing the video segments into a video summary. Based on this method, video summaries can be automatically generated efficiently and accurately.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and specifically relates to a method and device for generating a video summary. Background Art

[0002] Online video sites have experienced tremendous growth in recent years, providing not only vast amounts of video data, but also a rapidly growing amount. Therefore, for users, directly selecting video summaries (such as climaxes and highlights) from this vast library of videos to watch has become a time-saving option. Furthermore, websites can improve the user experience by recommending video summaries to users and encourage them to watch the full video.

[0003] Unlike traditional video summaries, which strive to fully describe the video content, extracting video summaries aims to identify the most popular clips. Generally, generating such summaries requires time-consuming and labor-intensive manual editing, which is labor-intensive and inefficient. Furthermore, the resulting video summaries are significantly influenced by the operator's subjective aesthetic judgment and may not accurately reflect the highlights of the entire video.

[0004] Therefore, how to generate video summaries efficiently and accurately is a technical problem that needs to be solved urgently.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0006] The purpose of this application is to provide a method and apparatus for generating a video summary, which at least to some extent solves the technical problem of how to efficiently and accurately generate a video summary in the related art.

[0007] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0008] According to one aspect of an embodiment of the present application, a method for generating a video summary is provided. The method for generating a video summary includes:

[0009] Acquire multiple frames to be processed from the video to be processed, and perform text recognition on each of the frames to be processed to obtain text content corresponding to the frames to be processed;

[0010] Merging the frames to be processed according to the text contents corresponding to the frames to be processed to obtain a plurality of segment intervals, wherein the text contents of the segment intervals are determined by the text contents of the frames to be processed that are merged to form the segment intervals;

[0011] Extracting features from the text content of the frames to be processed in each segment interval to obtain text features, and determining a first emotion intensity of the segment interval based on the text features, wherein the first emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval in terms of the text;

[0012] Extracting features of expressions in frames to be processed in each of the segment intervals to obtain expression features, and determining a second emotion intensity of the segment interval based on the expression features, wherein the second emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval in terms of expression;

[0013] Performing a weighted sum operation on the first emotion intensity and the second emotion intensity to obtain an emotion score for the segment interval;

[0014] According to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value, the video to be processed is intercepted to obtain video segments, and the video segments are spliced ​​into a video summary.

[0015] According to one aspect of an embodiment of the present application, a video summary generation device is provided. The video summary generation device includes:

[0016] A text recognition module is configured to obtain a plurality of frames to be processed from the video to be processed, and perform text recognition on each of the frames to be processed to obtain text content corresponding to the frames to be processed;

[0017] a segment interval acquisition module configured to merge the frames to be processed according to the text contents corresponding to the frames to be processed to obtain a plurality of segment intervals, wherein the text contents of the segment intervals are determined by the text contents of the frames to be processed that are merged into the segment intervals;

[0018] a first emotion intensity acquisition module configured to extract features from the text content of the frames to be processed in each segment interval to obtain text features, and determine a first emotion intensity of the segment interval based on the text features, wherein the first emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval in terms of text;

[0019] a second emotion intensity acquisition module configured to extract features of expressions in frames to be processed in each of the segment intervals to obtain expression features, and determine a second emotion intensity of the segment interval based on the expression features, wherein the second emotion intensity is used to represent the intensity of the emotion in the expression of each frame to be processed in the segment interval;

[0020] an emotion score calculation module, configured to perform a weighted sum operation on the first emotion intensity and the second emotion intensity to obtain an emotion score for the segment interval;

[0021] The video segment splicing module is configured to intercept the video to be processed to obtain video segments according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value, and splice the video segments into a video summary.

[0022] In some embodiments of the present application, based on the above technical solution, the segment interval acquisition module includes:

[0023] a single sentence interval acquisition unit configured to calculate text similarity between the text contents of two adjacent frames to be processed when both frames have text content, and when the text similarity is greater than a preset threshold, merge the two adjacent frames to be processed into a single sentence interval, wherein the text content of the single sentence interval is determined by the text content of the two adjacent frames to be processed;

[0024] a first merging unit configured to merge two adjacent single sentence intervals into one text cluster interval when the time interval between the two adjacent single sentence intervals is less than a first preset interval;

[0025] The second merging unit is configured to merge two adjacent text cluster intervals into one segment interval when the time interval between the two adjacent text cluster intervals is less than a second preset interval.

[0026] In some embodiments of the present application, based on the above technical solution, the first emotion intensity acquisition module includes:

[0027] a text feature extraction unit configured to extract features from the text content of each text cluster interval in each of the segment intervals to obtain text features;

[0028] An emotion confidence determination unit is configured to determine the emotion type and emotion confidence corresponding to each of the text cluster intervals according to the text features, wherein the emotion confidence is the confidence that the text cluster interval is of the emotion type;

[0029] The first emotion intensity determining unit is configured to determine the first emotion intensity of the segment interval according to the emotion confidence of the text cluster interval whose emotion type belongs to the preset emotion type in the segment interval.

[0030] In some embodiments of the present application, based on the above technical solution, the first emotion intensity determination unit includes:

[0031] a target text cluster interval determination subunit configured to determine a text cluster interval in the segment interval whose emotion type belongs to a preset emotion type as a target text cluster interval;

[0032] The first emotion intensity determination subunit is configured to perform an average operation on the emotion confidence of each target text cluster interval in the segment interval to obtain a first emotion intensity of the segment interval.

[0033] In some embodiments of the present application, based on the above technical solution, the second emotion intensity acquisition module includes:

[0034] A face detection unit is configured to perform face detection on each of the frames to be processed to obtain a face image corresponding to the frame to be processed;

[0035] an expression feature extraction unit configured to extract features from the face image to obtain expression features corresponding to the frame to be processed;

[0036] an expression confidence determination unit configured to determine an expression type and an expression confidence corresponding to the frame to be processed according to the expression feature, wherein the expression confidence is a confidence that the text cluster interval is the expression type;

[0037] The second emotion intensity determining unit is configured to determine the second emotion intensity of the segment interval according to the expression confidence of the to-be-processed frame whose expression type in the segment interval belongs to the preset expression type.

[0038] In some embodiments of the present application, based on the above technical solution, the second emotion intensity determination unit includes:

[0039] a target frame determining unit configured to determine a to-be-processed frame in the segment interval whose expression type belongs to a preset expression type as a target frame;

[0040] The second emotion intensity determination subunit is configured to perform an averaging operation on the expression confidences of the target frames in the segment interval to obtain a second emotion intensity of the segment interval.

[0041] In some embodiments of the present application, based on the above technical solution, the video summary generation device further includes:

[0042] a reference interval determining unit configured to determine a first preset number of frames to be processed adjacent to a target frame to be processed as a reference interval of the target frame to be processed;

[0043] a frame number counting unit configured to count the number of frames to be processed corresponding to each expression type in a reference interval of the target frame to be processed;

[0044] The expression type correction unit is configured to correct the expression type of the target frame to be processed to the same expression type when the number of frames to be processed corresponding to the same expression type in the reference interval is greater than or equal to a second preset number, and the second preset number is less than the first preset number.

[0045] In some embodiments of the present application, based on the above technical solution, the video summary generation device further includes:

[0046] a video segment position obtaining unit, configured to obtain a start position and an end position of each of the video segments in the video to be processed;

[0047] a first change value detection unit configured to intercept a first detection interval of the video to be processed by shifting the start position forward by a first preset time period, and calculate a change value of a color histogram between all adjacent video frames within the first detection interval;

[0048] a first video segment updating unit configured to, when a change value of a color histogram between adjacent video frames within the first detection interval exceeds a first preset value, use a later video frame among the adjacent video frames as a new starting position of the video segment to update the video segment;

[0049] a second change value detection unit configured to intercept a second detection interval of the video to be processed by shifting the end position back by a second preset time period, and calculate a change value of a color histogram between all adjacent video frames within the second detection interval;

[0050] The second video segment updating unit is configured to update the video segment by using the preceding video frame among the adjacent video frames as the re-determined end position of the video segment when the change value of the color histogram between adjacent video frames in the second detection interval exceeds a second preset value.

[0051] In some embodiments of the present application, based on the above technical solution, the video summary generation device further includes:

[0052] a segment interval sorting unit, configured to sort the segment intervals in descending order of sentiment scores;

[0053] a segment interval acquisition unit configured to acquire a preset target duration, and acquire one or more segment intervals from each of the segment intervals in descending order of sentiment scores according to the target duration, so that a total duration of video segments obtained by intercepting the video to be processed according to the video time periods corresponding to the acquired segment intervals is less than or equal to the target duration;

[0054] The preset value determining unit is configured to use the lowest emotion score among the acquired emotion scores of the segment intervals as the preset value.

[0055] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the video summary generation method in the above technical solution is implemented.

[0056] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the video summary generation method in the above technical solution by executing the executable instructions.

[0057] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video summary generation method described in the above technical solution.

[0058] In the technical solution provided in the embodiment of the present application, multiple frames to be processed are obtained from the video to be processed, and text recognition is performed on each frame to be processed to obtain the text content corresponding to the frame to be processed; each frame to be processed is merged according to the text content corresponding to the frame to be processed to obtain multiple segment intervals; feature extraction is performed on the text content of the frame to be processed in each segment interval to obtain text features, and the first emotion intensity of the segment interval is determined based on the text features; feature extraction is performed on the expression in the frame to be processed in each segment interval to obtain expression features, and the second emotion intensity of the segment interval is determined based on the expression features; a weighted summation operation is performed on the first emotion intensity and the second emotion intensity to obtain an emotion score of the segment interval; according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to a preset value, the video to be processed is intercepted to obtain video segments, and the video segments are spliced ​​into a video summary, so that a video summary can be automatically generated according to the emotion intensity of each segment in the video, and the emotion score is jointly generated by the first emotion intensity and the second emotion intensity, so that the calculated emotion score is more accurate and comprehensive, and the video summary can be generated efficiently and accurately.

[0059] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0061] Figure 1 The following schematically shows an exemplary system architecture block diagram applying the technical solution of the present application.

[0062] Figure 2 The following schematically illustrates the steps of the video summary generation method provided in the embodiment of the present application.

[0063] Figure 3 The embodiment of the present application schematically illustrates the steps of merging the frames to be processed according to the text contents corresponding to the frames to be processed to obtain multiple segment intervals.

[0064] Figure 4 The embodiment of the present application schematically illustrates the steps of extracting features from the text content of the frames to be processed in each segment interval to obtain text features, and determining the first emotion intensity of the segment interval based on the text features.

[0065] Figure 5 The embodiment of the present application schematically illustrates the steps of determining the first emotion intensity of a segment interval according to the emotion confidence of a text cluster interval whose emotion type in the segment interval belongs to a preset emotion type.

[0066] Figure 6 The embodiment of the present application schematically illustrates the steps of extracting features of expressions in frames to be processed in each segment interval to obtain expression features, and determining the second emotion intensity of the segment interval based on the expression features.

[0067] Figure 7 The embodiment of the present application schematically illustrates the steps of determining the second emotion intensity of a segment interval according to the expression confidence of a frame to be processed whose expression type in the segment interval belongs to a preset expression type.

[0068] Figure 8 The following schematically illustrates the steps after determining the expression type and expression confidence corresponding to the frame to be processed based on the expression features in an embodiment of the present application.

[0069] Figure 9 The embodiment of the present application schematically illustrates the steps after intercepting the video to be processed to obtain video segments according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value.

[0070] Figure 10 The embodiment of the present application schematically illustrates the steps of intercepting the video to be processed to obtain video segments according to the video time period corresponding to the segment interval with an emotion score greater than or equal to a preset value, and before splicing the video segments into a video summary.

[0071] Figure 11 The structural block diagram of the video summary generation device provided in an embodiment of the present application is schematically shown.

[0072] Figure 12 The structural block diagram of an electronic device suitable for implementing the embodiments of the present application is schematically shown. DETAILED DESCRIPTION

[0073] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0074] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0075] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0076] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0077] Before providing a detailed description of the information processing methods, information processing devices and other technical solutions provided in the embodiments of this application, a brief introduction to the artificial intelligence technology and cloud technology involved in some embodiments of this application is first given.

[0078] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0079] Natural language processing (NLP) is a key area of ​​research in computer science and artificial intelligence. It studies the theories and methods that enable effective communication between humans and computers using natural language. Natural language processing (NLP) integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language we use in everyday life—and is closely linked to the study of linguistics. Natural language processing technologies typically include text processing, semantic understanding, machine translation, robotic question answering, and knowledge graphs.

[0080] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0081] Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a key support. The backend services of technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark, which will need to be transmitted to the backend system for logical processing. Different levels of data will be processed separately. All kinds of industry data require strong system support, which can only be achieved through cloud computing.

[0082] Cloud computing refers to the delivery and usage model of IT infrastructure, enabling on-demand, scalable access to required resources over the internet. Broadly speaking, cloud computing refers to the delivery and usage model of services, enabling on-demand, scalable access to required services over the internet. These services can be IT-related, software-related, internet-related, or other services. Cloud computing is the product of the convergence of traditional computer and network technologies, including grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0083] Cloud computing has rapidly grown, driven by the internet, real-time data streams, the diversification of connected devices, and the growing demand for search services, social networks, mobile commerce, and open collaboration. Unlike previous parallel and distributed computing approaches, the emergence of cloud computing will fundamentally revolutionize the entire internet and enterprise management model.

[0084] Artificial intelligence cloud services, also known as AIaaS (AI as a Service), are a mainstream AI platform offering. Specifically, AIaaS platforms break down several common AI services and provide them as standalone or packaged services in the cloud. This service model is similar to an AI-themed marketplace: all developers can access one or more of the platform's AI services through APIs. Experienced developers can also use the platform's AI framework and infrastructure to deploy and maintain their own cloud AI services.

[0085] The video summary generation method and device provided by this application are described in detail below in conjunction with specific implementation methods.

[0086] Figure 1 The following schematically shows an exemplary system architecture block diagram applying the technical solution of the present application.

[0087] like Figure 1As shown, system architecture 100 may include terminal device 110, network 120, and server 130. Terminal device 110 may include various electronic devices such as smartphones, tablet computers, laptop computers, and desktop computers. Server 130 may be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Network 120 may be a communication medium of various connection types capable of providing a communication link between terminal device 110 and server 130, such as a wired communication link or a wireless communication link.

[0088] Depending on the implementation requirements, the system architecture in the embodiments of the present application can have any number of terminal devices, networks, and servers. For example, the server 130 can be a server group consisting of multiple server devices. In addition, the technical solutions provided in the embodiments of the present application can be applied to the terminal device 110, can also be applied to the server 130, or can be implemented by the terminal device 110 and the server 130 together, and this application does not make any special restrictions on this.

[0089] The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in this application.

[0090] For example, after the server 130 obtains the video to be processed from the terminal device 110 or other servers, the server 130 can execute the video summary generation method provided by the present application, by obtaining multiple frames to be processed from the video to be processed, and performing text recognition on each frame to be processed to obtain the text content corresponding to the frame to be processed; merging each frame to be processed according to the text content corresponding to the frame to be processed to obtain multiple segment intervals; performing feature extraction on the text content of the frame to be processed in each segment interval to obtain text features, and determining the first emotion intensity of the segment interval based on the text features; and performing text recognition on the expression content of the frame to be processed in each segment interval. The method extracts the expression features from the emotion and determines the second emotion intensity of the segment interval according to the expression features; performs a weighted summation operation on the first emotion intensity and the second emotion intensity to obtain the emotion score of the segment interval; according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value, intercepts the video to be processed to obtain video segments, and splices the video segments into a video summary. Thus, the video summary can be automatically generated according to the emotion intensity of each segment in the video, and the emotion score is jointly generated by the first emotion intensity and the second emotion intensity, so that the calculated emotion score is more accurate and comprehensive, and the video summary can be generated efficiently and accurately.

[0091] Figure 2 The schematic diagram shows the steps of the video summary generation method provided by the embodiment of the present application. The execution subject of the video summary generation method can be a terminal device or a server. The terminal device includes but is not limited to a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc. Figure 2 As shown, the information processing method may mainly include the following steps S210 to S260:

[0092] S210. Acquire multiple frames to be processed from the video to be processed, and perform text recognition on each frame to be processed to obtain text content corresponding to the frame to be processed.

[0093] The video to be processed may be a video uploaded by a user for generating a video summary, or may be a video to be recommended to a user on a video portal website through the generated video summary.

[0094] Specifically, text recognition may be performed on the frame to be processed through OCR (Optical Character Recognition).

[0095] Acquiring multiple frames to be processed from the video to be processed may include extracting multiple frames to be processed from the video to be processed. Specifically, the multiple frames to be processed may be extracted from the video to be processed at a preset frequency. For example, the multiple frames to be processed may be extracted from the video to be processed at a preset frequency of 10 FPS, that is, the multiple frames to be processed may be extracted from the video to be processed at a fixed time interval of 0.1 seconds.

[0096] In some implementations, performing text recognition on each frame to be processed to obtain text content corresponding to the frame to be processed may be performing text recognition on a preset area in each frame to be processed to obtain text content corresponding to the processing frame.

[0097] Specifically, the preset area may be the area corresponding to the subtitles in the frame to be processed. Alternatively, the preset area may be the lower middle portion of the frame to be processed. In this case, the text content obtained by performing text recognition on each frame to be processed may be subtitle content. In other embodiments, the text content obtained by performing text recognition on each frame to be processed may also be news headlines, course notes, etc., and this application does not impose any particular limitation on this.

[0098] Text recognition is performed on each frame to be processed to obtain the text content corresponding to the frame to be processed, thereby obtaining the information {frame time, text content} corresponding to the frame to be processed.

[0099] S220. Merge the frames to be processed according to the text contents corresponding to the frames to be processed to obtain a plurality of segment intervals, wherein the text contents of the segment intervals are determined by the text contents of the frames to be processed that are merged into the segment intervals.

[0100] Specifically, the text content of a segment interval can be obtained by deduplicating and fusing the text content of the frames to be processed. In other words, the information corresponding to the segment interval can be obtained: {interval time period, text content}. The interval time period is determined by the frame times of the first and last frames to be processed within the segment interval.

[0101] Figure 3 The embodiment of the present application schematically illustrates the steps of merging the frames to be processed according to the text content corresponding to the frames to be processed to obtain multiple segment intervals. Figure 3 As shown, based on the above embodiments, in some embodiments, the steps of merging the frames to be processed according to the text contents corresponding to the frames to be processed to obtain multiple segment intervals in step S220 may further include the following steps S310 to S330:

[0102] S310. When two adjacent frames to be processed have text content, the text similarity of the text content of the two adjacent frames to be processed is calculated. When the text similarity is greater than a preset threshold, the two adjacent frames to be processed are merged into a single sentence interval, and the text content of the single sentence interval is determined by the text content of the two adjacent frames to be processed;

[0103] S320. When the time interval between two adjacent single sentence intervals is less than a first preset interval, merging the two adjacent single sentence intervals into one text cluster interval;

[0104] S330 . When the time interval between two adjacent text cluster intervals is less than a second preset interval, merge the two adjacent text cluster intervals into one segment interval.

[0105] Specifically, the text similarity of the text contents of two adjacent frames to be processed is calculated by obtaining the first text content of the first frame to be processed and the second text content of the second frame to be processed adjacent to the first frame to be processed, detecting the number of repeated words in the first text content and the second text content, and counting the average number of text words in the first text content and the second text content, and then taking the ratio of the number of repeated words to the average number of text words as the text similarity between the first frame to be processed and the second frame to be processed.

[0106] Merging two adjacent frames to be processed into a single sentence interval may be done by taking the frame moments of the two adjacent frames to be processed as the endpoints of the interval time period of the segment interval, and then determining the text content of the single sentence interval based on the text content of the two adjacent frames to be processed. Specifically, determining the text content of the single sentence interval based on the text content of the two adjacent frames to be processed may be done by selecting one of the text contents of the two adjacent frames to be processed as the text content of the single sentence interval. For example, the recognition confidence obtained by performing OCR text recognition on the frame to be processed may be obtained, and the text content of the frame to be processed with higher recognition confidence may be selected from the text contents of the two adjacent frames to be processed as the text content of the single sentence interval. The recognition confidence is used to indicate the accuracy or reliability of the OCR text recognition of the frame to be processed.

[0107] Merge two adjacent single-sentence intervals into a text cluster interval. Specifically, the start position of the earlier single-sentence interval of the two adjacent single-sentence intervals can be used as the start position of the text cluster interval, and the end position of the later single-sentence interval can be used as the end position of the text cluster interval. This results in the interval time period of the text cluster interval. Then, the text content of the two adjacent single-sentence intervals is spliced ​​in chronological order to form the text content of the text cluster interval.

[0108] When the time interval between two adjacent single-sentence intervals is less than a first preset interval, the two adjacent single-sentence intervals are merged into a text cluster interval. In this way, the texts of the same cluster, that is, the texts of the same paragraph, can be divided into the same text cluster interval. The text cluster intervals can be accurately divided, and the subtitle plots of the same paragraph can be avoided from being divided into different text cluster intervals, thereby improving the viewing experience of the generated video summary.

[0109] Merge two adjacent text cluster intervals into a single segment interval. Specifically, the start position of the earlier text cluster interval is used as the start position of the merged segment interval, and the end position of the later text cluster interval is used as the end position of the merged segment interval. This results in the time period of the merged segment interval. Then, the text content of the two adjacent text cluster intervals is spliced ​​in chronological order to form the text content of the segment interval.

[0110] Specifically, the second preset interval may be greater than or equal to the first preset interval. The first time interval may be 2s, 3s, 4s, etc. The second time interval may be 4s, 4.4s, 5s, etc.

[0111] When the time interval between two adjacent text cluster intervals is less than the second preset interval, the two adjacent text cluster intervals are merged into a single segment interval, which can classify two closely spaced text clusters into the same segment interval n. It can be understood that text cluster intervals that are closely spaced in video time are more likely to have plot and emotion correlations. Therefore, when the time interval between two adjacent text cluster intervals is less than the second preset interval, the two adjacent text cluster intervals are merged into a single segment interval. This can merge text cluster intervals with a high probability of plot correlations into the same segment interval, thereby improving the accuracy of the resulting segment interval division, avoiding the segmentation of highly correlated plots, and thus improving the viewing experience of the generated video summary.

[0112] In some implementations, when the time interval between two adjacent single sentence intervals is less than a first preset interval, merging the two adjacent single sentence intervals into one text cluster interval in S320 may be:

[0113] When the time interval between two adjacent single sentence intervals is less than a first preset interval, and the total number of text words in the text contents of the two adjacent single sentence intervals is less than a preset number of words, the two adjacent single sentence intervals are merged into one text cluster interval.

[0114] This avoids the generation of text clusters with excessively large characters, which would otherwise result in excessive processing granularity and reduce the accuracy of feature extraction for the text clusters. It also facilitates the merging of multiple single-sentence intervals with fewer characters into a single text cluster, thereby improving the completeness of the text meaning and the accuracy of feature extraction for the text clusters. This, in turn, helps improve the accuracy of the first emotion intensity of the segment interval derived based on the emotion type and emotion confidence of the text cluster. For example, the preset number of characters can be 20, 25, 30, or 35, depending on the specific embodiment.

[0115] S230. Extract features of the text content of the frames to be processed in each segment interval to obtain text features, and determine the first emotion intensity of the segment interval based on the text features. The first emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval in terms of text.

[0116] Specifically, feature extraction is performed on the text content of the frames to be processed in each segment interval to obtain text features, and the first emotion intensity of the segment interval is determined based on the text features. This can be achieved by using a feature extractor in a pre-trained machine learning model to extract features from the text content of the frames to be processed in each segment interval to obtain text features, and then inputting the text features into a classifier in the pre-trained machine learning model to determine the first emotion intensity of the segment interval. In some embodiments, feature extraction can also be performed on the text content of the frames to be processed in each segment interval to obtain text features through other methods such as a word vector feature extraction network, and the first emotion intensity of the segment interval can be determined through text features through other methods such as a voter.

[0117] Feature extractors and classifiers can be integrated into a machine learning model and trained together. Specifically, a text recognition dataset, such as the NLPCC2014 sentiment multi-classification public dataset, can be used to train the machine learning model. In a specific example, the machine learning model can be built based on a neural network model, such as the BERT model, though this application does not impose any specific limitations on this.

[0118] Figure 4 The embodiment of the present application schematically illustrates the steps of extracting features from the text content of the frames to be processed in each segment interval to obtain text features, and determining the first emotion intensity of the segment interval based on the text features. Figure 4 As shown, based on the above embodiments, in some embodiments, step S230 of extracting features from the text content of the frames to be processed in each segment interval to obtain text features, and determining the first emotion intensity of the segment interval according to the text features, may further include the following steps S410 to S430:

[0119] S410. Extracting features from the text content of each text cluster interval in each segment interval to obtain text features;

[0120] S420. Determine the emotion type and emotion confidence corresponding to each text cluster interval according to the text features, where the emotion confidence is the confidence that the text cluster interval is the emotion type;

[0121] S430. Determine a first emotion intensity of the segment interval according to the emotion confidence of the text cluster interval whose emotion type in the segment interval belongs to a preset emotion type.

[0122] The text features of each text cluster interval in each segment interval are extracted by extracting features from the text content of the to-be-processed frames in each segment interval using a feature extractor in a pre-trained machine learning model. The sentiment type and sentiment confidence corresponding to each text cluster interval are determined based on the text features by inputting the text features into a classifier in a pre-trained machine learning model to determine the sentiment type and sentiment confidence corresponding to each text cluster interval.

[0123] Thus, the first emotion intensity of a segment interval can be calculated by calculating the emotion type and emotion confidence of each text cluster interval within the segment interval, thereby quantitatively reflecting the emotion intensity of the segment interval, facilitating the subsequent selection of video segments required for generating a video summary. Furthermore, it can be understood that a text cluster interval is composed of one or more single-sentence intervals, forming a relatively complete paragraph and thus possessing relatively complete emotional meaning. Therefore, by determining the first emotion intensity of a segment interval based on the emotion confidence of text cluster intervals within the segment interval whose emotion type belongs to a preset emotion type, the emotion intensity of the segment interval can be more accurately estimated.

[0124] In other embodiments, step S230 of extracting features from the text content of the frames to be processed in each segment interval to obtain text features, and determining the first emotion intensity of the segment interval based on the text features, may further include the following steps:

[0125] Extracting features from the text content of each single sentence interval in each segment interval to obtain text features;

[0126] Determine the sentiment type and sentiment confidence corresponding to each single sentence interval based on text features;

[0127] The first emotion intensity of the segment interval is determined according to the emotion confidence of the single sentence interval whose emotion type in the segment interval belongs to the preset emotion type.

[0128] Specifically, the first emotion intensity of a segment interval is determined based on the emotion confidence of a single sentence interval within the segment interval whose emotion type belongs to a preset emotion type. This can be done by identifying a single sentence interval within the segment interval whose emotion type belongs to the preset emotion type as a target single sentence interval, and averaging the emotion confidence of each target single sentence interval within the segment interval to obtain the first emotion intensity of the segment interval. This allows for a more granular calculation of the first emotion intensity of a segment interval.

[0129] Specifically, emotion types may include anger, disgust, fear, happiness, sadness, surprise, and a natural lack of emotion.

[0130] Figure 5 The embodiment of the present application schematically shows the steps of determining the first emotion intensity of a segment interval according to the emotion confidence of the text cluster interval in which the emotion type in the segment interval belongs to the preset emotion type. Figure 5 As shown, based on the above embodiments, in some embodiments, determining the first emotion intensity of the segment interval according to the emotion confidence of the text cluster interval whose emotion type belongs to the preset emotion type in the segment interval in step S430 may further include the following steps S510 to S520:

[0131] S510. Determine the text cluster interval whose emotion type in the segment interval belongs to the preset emotion type as the target text cluster interval;

[0132] S520. Perform an average operation on the sentiment confidence of each target text cluster interval in the segment interval to obtain a first sentiment intensity of the segment interval.

[0133] Therefore, a text cluster interval is composed of one or more single-sentence intervals and has a relatively complete paragraph, so it can have a relatively complete emotional meaning. Therefore, the emotional confidence of each target text cluster interval in the segment interval is averaged to obtain the first emotional intensity of the segment interval, which can estimate the emotional intensity of the segment interval as a whole, which is convenient for the subsequent selection of video segments required to generate video summaries.

[0134] Furthermore, the emotional intensity of aspects of the text that belong to a preset emotional type can be calculated, thereby facilitating the generation of a video summary with user-defined emotional tendencies, thereby improving user experience.

[0135] S240. Extract features of expressions in the frames to be processed in each segment interval to obtain expression features, and determine the second emotion intensity of the segment interval based on the expression features. The second emotion intensity is used to represent the intensity of the emotion in the expression of each frame to be processed in the segment interval.

[0136] Specifically, after detecting the facial expressions in the frames to be processed in each segment interval, the feature extractor of the pre-trained machine learning model is used to extract features from the detected facial expression information to obtain expression features, and the classifier of the pre-trained machine learning model is used to determine the second emotion intensity of the segment interval based on the expression features. In some embodiments, other methods can also be used to extract features from the detected expressions to obtain expression features, and other methods such as a voting machine can be used to determine the second emotion intensity of the segment interval based on the expression features. Specifically, expressions can include human facial expressions, animal facial expressions, animated facial expressions, etc.

[0137] Figure 6 The embodiment of the present application schematically illustrates the steps of extracting features of expressions in frames to be processed in each segment interval to obtain expression features, and determining the second emotion intensity of the segment interval based on the expression features. Figure 6 As shown, based on the above embodiments, in some embodiments, the step S240 of extracting features of expressions in the frames to be processed in each segment interval to obtain expression features, and determining the second emotion intensity of the segment interval according to the expression features, may further include the following steps S610 to S640:

[0138] S610 performs face detection on each frame to be processed to obtain a face image corresponding to the frame to be processed;

[0139] S620. Extract features from the facial image to obtain expression features corresponding to the frame to be processed;

[0140] S630. Determine the expression type and expression confidence corresponding to the frame to be processed according to the expression feature, where the expression confidence is the confidence that the text cluster interval is the expression type;

[0141] S640. Determine the second emotion intensity of the segment interval according to the expression confidence of the to-be-processed frame whose expression type in the segment interval belongs to the preset expression type.

[0142] Face detection is performed on each frame to be processed. For each video frame to be processed, key point detection is performed through three cascaded convolutional neural networks (MTCNN) to obtain the coordinate positions of five facial key points (left eye, right eye, nose, left corner of the mouth, and right corner of the mouth), thereby determining the position of the face image corresponding to the frame to be processed. In a specific embodiment, since the faces in the video frames often have side faces or tilted heads, an affine transformation algorithm is also used to adjust the coordinates of the five key points obtained by face detection so that all faces are transformed to the position of the front face. In some embodiments, the obtained face images can also be uniformly resized to 112*112 pixels for cropping, which can facilitate the standardization of subsequent processing and improve the accuracy of subsequent facial expression recognition.

[0143] Feature extraction is performed on the face image to obtain the expression features corresponding to the frame to be processed. The face image can be input into a pre-trained feature extractor to obtain the expression features corresponding to the frame to be processed.

[0144] The expression type and expression confidence corresponding to the frame to be processed are determined based on the expression features. In a specific example, the expression features of the frame to be processed can be output to a pre-trained expression classifier to obtain the expression type and expression confidence corresponding to the frame to be processed.

[0145] The feature extractor and expression classifier can be integrated into a machine learning model and trained together. Specifically, the machine learning model can be trained using an expression recognition dataset such as the FER2013 public expression recognition dataset. In a specific example, the machine learning model can be built using a neural network model such as ResNet50, which is not specifically limited in this application. Specifically, during the training process of the machine learning model, an angular margin loss function can be used to train the network to obtain more accurate expression classification results.

[0146] Specifically, expression types may include anger, disgust, fear, happiness, sadness, surprise, and neutral emotion. Expression types may have a one-to-one correspondence with emotion types, so that after a user selects a certain type of video summary editing direction, the application determines the corresponding expression type and emotion type according to the type selected by the user.

[0147] Figure 7 The embodiment of the present application schematically shows the steps of determining the second emotion intensity of the segment interval according to the expression confidence of the frame to be processed whose expression type in the segment interval belongs to the preset expression type. Figure 7 As shown, based on the above embodiments, in some embodiments, determining the second emotion intensity of the segment interval according to the expression confidence of the to-be-processed frame whose expression type in the segment interval belongs to the preset expression type in step S640 may further include the following steps S710 and S720:

[0148] S710. Determine the frame to be processed whose expression type in the segment interval belongs to the preset expression type as the target frame;

[0149] S720. Perform an average operation on the expression confidence of each target frame in the segment interval to obtain the second emotion intensity of the segment interval.

[0150] Thus, the frames in the segment interval whose expression types belong to the preset expression types are identified as target frames. The expression confidence levels of the target frames in the segment interval are then averaged to obtain the second emotion intensity of the segment interval. This allows for quantified calculation of the emotion intensity of the segment interval at the frame level. Furthermore, the emotion intensity of aspects belonging to the preset expression types can be calculated, facilitating the generation of video summaries with user-defined emotion tendencies, thereby improving the user experience.

[0151] Figure 8 The following schematically illustrates the steps after determining the expression type and expression confidence corresponding to the frame to be processed according to the expression features in the embodiment of the present application. Figure 8 As shown, based on the above embodiments, in some embodiments, after determining the expression type and expression confidence corresponding to the frame to be processed according to the expression features in step S630, the following steps S810 to S830 may be further included:

[0152] S810. Determine a first preset number of frames to be processed adjacent to the target frame to be processed as a reference interval of the target frame to be processed;

[0153] S820. Counting the number of frames to be processed corresponding to each expression type in the reference interval of the target frame to be processed;

[0154] S830. When the number of frames to be processed corresponding to the same expression type in the reference interval is greater than or equal to a second preset number, the expression type of the target frame to be processed is corrected to the same expression type, and the second preset number is less than the first preset number.

[0155] In some embodiments, S810 may be to determine as the reference interval of the target frame to be processed a third preset number of frames adjacent to the target frame to be processed that precede the target frame to be processed in time, and a fourth preset number of frames adjacent to the target frame to be processed that precede the target frame to be processed in time, wherein the first preset number is the sum of the third preset number and the fourth preset number.

[0156] In a specific embodiment, the first preset number can be 10, and the first preset number of frames to be processed adjacent to the target frame to be processed are determined as the reference interval of the target frame to be processed. Then, the 10 frames to be processed adjacent to the target frame to be processed can be determined as the reference interval of the target frame to be processed. In a specific embodiment, 5 frames to be processed that are adjacent to the target frame to be processed before the target frame to be processed, and 5 frames to be processed that are adjacent to the target frame to be processed after the target frame to be processed can be determined as the reference interval of the target frame to be processed. At this time, the second preset number can be 7, and the number of frames to be processed corresponding to each expression type in the reference interval of the target frame to be processed is counted. When the number of frames to be processed corresponding to the happy type in the reference interval is greater than or equal to 7, the expression type of the target frame to be processed is corrected to the happy type.

[0157] It is understood that the expression changes between adjacent frames are generally small. When the number of frames to be processed corresponding to the same expression type in the reference interval is greater than or equal to the second preset number, and the expression type of the target frame to be processed is different from the same expression type, the recognition of the expression type of the target frame to be processed is likely to be incorrect. Therefore, based on the above embodiment, it is possible to correct the recognition results of the expression type, reduce recognition errors, and thus improve the recognition accuracy of the expression type.

[0158] S250. Perform a weighted sum operation on the first emotion intensity and the second emotion intensity to obtain an emotion score for the segment interval.

[0159] Therefore, by performing comprehensive calculations on the first emotion intensity of the segment interval obtained based on text features and the second emotion intensity of the segment interval obtained based on expression features, the emotion score of the segment interval can be obtained, thereby realizing multimodal emotion recognition of text and vision in the segment interval of the video, which is beneficial to improving the accuracy of the obtained emotion score of the segment interval, and further beneficial to improving the accuracy and viewing experience of the video summary subsequently generated based on the emotion score of the segment interval.

[0160] Specifically, the first emotion intensity and the second emotion intensity can be weighted and summed using the following formula to obtain the emotion score of the segment interval:

[0161]

[0162] Among them, α is the weight of the second emotion intensity, and β is the weight of the first emotion intensity. The second emotional intensity. is the first emotion intensity. n is the number of frames to be processed that belong to the preset expression type in the segment interval, and m is the number of text cluster intervals that belong to the preset emotion type in the segment interval. faceIndicates the confidence level of the expression corresponding to the expression type of the frame to be processed. text Indicates the sentiment confidence corresponding to the sentiment type of the text cluster interval.

[0163] S260. According to the video time period corresponding to the segment interval with the emotion score greater than or equal to the preset value, intercept the video to be processed to obtain video segments, and splice the video segments into a video summary.

[0164] Therefore, according to the video time period corresponding to the segment interval with a sentiment score greater than or equal to a preset value, the video to be processed is intercepted to obtain video segments, and the video segments are spliced ​​into a video summary, which can achieve efficient and accurate generation of video summaries.

[0165] It can be understood that the segment intervals with higher emotional scores correspond to video segments with stronger emotions, stronger emotional colors, or more climaxes and exciting parts of the plot. According to the video time period corresponding to the segment intervals with emotional scores greater than or equal to the preset value, the video to be processed is intercepted to obtain video segments, and the video segments are spliced ​​into a video summary. This can enable the video summary to accurately present the exciting parts of the entire video, which can help to splice the most popular exciting segments with the audience into the video summary.

[0166] In certain embodiments of the present application, a plurality of frames to be processed are first extracted from the video to be processed, and a segment interval is obtained based on the text content of the frames to be processed. Then, a subsequent series of processing is performed on the segment interval to obtain a segment area with a sentiment score greater than or equal to a preset value, and finally, a video segment corresponding to the segment interval is intercepted. It can be understood that, compared with the solution of directly processing the video segment, extracting frames and processing the data obtained based on the extracted frames before intercepting the video segment can reduce the amount of data processing in the process of generating the video summary, improve the generation efficiency of the video summary, and achieve the effect of efficiently generating the video summary.

[0167] In some embodiments, the video summary generation method of the present application can be applied to the automatic video editing scenario on the user side, or the generated video summary can be applied to the presentation stream of recommending videos to users, etc., so as to efficiently and accurately generate a video summary consisting of emotionally strong and wonderful clips, save the labor cost of editing, reduce the errors of manual editing, and improve the user experience.

[0168] Figure 9 The following schematically illustrates the steps after intercepting the video to be processed to obtain the video segments according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value in the embodiment of the present application. Figure 9As shown, based on the above embodiments, in some embodiments, after intercepting the video to be processed to obtain video segments according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value in step S260, the following steps S910 to S950 may be further included:

[0169] S910. Get the starting position and ending position of each video clip in the video to be processed;

[0170] S920. According to the starting position, a first preset time period is shifted forward to intercept a first detection interval of the video to be processed, and a change value of the color histogram between all adjacent video frames in the first detection interval is calculated;

[0171] S930. When the change value of the color histogram between adjacent video frames within the first detection interval exceeds a first preset value, the later video frame in the adjacent video frames is used as the starting position of the video segment to re-determine the video segment;

[0172] S940. According to the end position, a second preset time period is shifted back to intercept a second detection interval of the video to be processed, and the change value of the color histogram between all adjacent video frames in the second detection interval is calculated;

[0173] S950. When the change value of the color histogram between adjacent video frames in the second detection interval exceeds a second preset value, the preceding video frame among the adjacent video frames is used as the re-determined end position of the video segment, and the video segment is updated.

[0174] In this way, the start and end points of the video segments corresponding to the segment intervals can be adjusted, making the segments more complete.

[0175] It is understandable that since the starting and ending positions of the segment interval are determined based on the time position of the video frame extraction, when the video segment is intercepted according to the corresponding time point, the video start and end may appear to be "swallowed words" or the shot is incomplete, and a video segment of a complete shot cannot be presented. However, through the above implementation, when the change value of the color histogram between adjacent video frames in the first detection interval exceeds the first preset value, the video frame at the back of the adjacent video frames is used as the re-determined starting position of the video segment, and the video segment is updated. When the change value of the color histogram between adjacent video frames in the second detection interval exceeds the second preset value, the video frame at the front of the adjacent video frames is used as the re-determined ending position of the video segment, and the video segment is updated, thereby achieving fine-tuning of the start and end time of the video segment, so that the video segment can present a complete shot, avoid the phenomenon of "swallowed words" or incomplete shot presentation, and improve the user's viewing experience of the generated video summary.

[0176] Specifically, the change value of the color histogram between adjacent video frames may be the average of the change values ​​of each color value in the color histogram between adjacent video frames. Alternatively, the change value of the color histogram between adjacent video frames may be the difference between the average values ​​of each color value in the color histogram between adjacent video frames.

[0177] It can be understood that when the change value of the color histogram between adjacent video frames exceeds the first preset value, it can be considered that the lens has been switched. Using the lens switching point as the re-determined starting position or ending position of the video segment can better ensure the lens integrity of the video segment, thereby improving the viewing experience of the produced video summary and enhancing the user viewing experience.

[0178] Figure 10 The embodiment of the present application schematically illustrates the steps of intercepting the video to be processed to obtain video segments according to the video time period corresponding to the segment interval with an emotion score greater than or equal to a preset value, and then splicing the video segments into a video summary. Figure 10 As shown, based on the above embodiments, in some embodiments, before intercepting the video to be processed to obtain video segments according to the video time periods corresponding to the segment intervals whose emotion scores are greater than or equal to the preset value in step S260 and splicing the video segments into a video summary, the following steps S1010 to S1030 may be further included:

[0179] S1010. Sort each segment interval in descending order according to the sentiment score;

[0180] S1020. Obtain a preset target duration, and according to the target duration, obtain one or more segment intervals from each segment interval in descending order of the emotion scores, so that the total duration of the video segments obtained by intercepting the video to be processed according to the video time period corresponding to the obtained segment intervals is less than or equal to the target duration;

[0181] S1030. The lowest emotion score among the acquired emotion scores of the segment intervals is used as a preset value.

[0182] In this way, a video summary with a duration not exceeding the preset target duration can be generated, which is conducive to using the user-defined target duration as the preset target duration and generating a video summary with a duration not exceeding the user-defined target duration, which is conducive to improving user experience.

[0183] It should be noted that although the steps of the method of the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0184] The following describes an apparatus embodiment of the present application, which can be used to execute the video summary generation method in the above-mentioned embodiment of the present application. Figure 11 The structure block diagram of the video summary generation device provided by the embodiment of the present application is schematically shown. Figure 11 As shown, the video summary generating apparatus 1100 may include:

[0185] The text recognition module 1110 is configured to obtain multiple frames to be processed from the video to be processed, and perform text recognition on each frame to be processed to obtain text content corresponding to the frame to be processed;

[0186] The segment interval acquisition module 1120 is configured to merge the frames to be processed according to the text content corresponding to the frames to be processed to obtain multiple segment intervals, wherein the text content of the segment interval is determined by the text content of the frames to be processed that are merged into the segment interval;

[0187] A first emotion intensity acquisition module 1130 is configured to extract features from the text content of the frames to be processed in each segment interval to obtain text features, and determine a first emotion intensity of the segment interval based on the text features. The first emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval.

[0188] The second emotion intensity acquisition module 1140 is configured to extract features of the expressions in the frames to be processed in each segment interval to obtain expression features, and determine the second emotion intensity of the segment interval based on the expression features, where the second emotion intensity is used to represent the intensity of the emotion in the expressions of each frame to be processed in the segment interval;

[0189] The emotion score calculation module 1150 is configured to perform a weighted sum operation on the first emotion intensity and the second emotion intensity to obtain an emotion score for the segment interval;

[0190] The video segment splicing module 1160 is configured to intercept the video to be processed to obtain video segments according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to a preset value, and splice the video segments into a video summary.

[0191] In some embodiments of the present application, based on the above embodiments, the segment interval acquisition module includes:

[0192] a single sentence interval acquisition unit configured to calculate the text similarity of the text contents of two adjacent frames to be processed when both frames have text contents, and when the text similarity is greater than a preset threshold, merge the two adjacent frames to be processed into a single sentence interval, where the text content of the single sentence interval is determined by the text contents of the two adjacent frames to be processed;

[0193] A first merging unit is configured to merge two adjacent single sentence intervals into one text cluster interval when the time interval between the two adjacent single sentence intervals is less than a first preset interval;

[0194] The second merging unit is configured to merge two adjacent text cluster intervals into one segment interval when the time interval between the two adjacent text cluster intervals is less than a second preset interval.

[0195] In some embodiments of the present application, based on the above embodiments, the first emotion intensity acquisition module includes:

[0196] A text feature extraction unit is configured to extract features from the text content of each text cluster interval in each segment interval to obtain text features;

[0197] An emotion confidence determination unit is configured to determine the emotion type and emotion confidence corresponding to each text cluster interval based on text features, where the emotion confidence is the confidence that the text cluster interval is of the emotion type;

[0198] The first emotion intensity determining unit is configured to determine a first emotion intensity of the segment interval according to the emotion confidence of the text cluster interval whose emotion type belongs to the preset emotion type in the segment interval.

[0199] In some embodiments of the present application, based on the above embodiments, the first emotion intensity determination unit includes:

[0200] a target text cluster interval determination subunit configured to determine a text cluster interval in the segment interval whose emotion type belongs to a preset emotion type as a target text cluster interval;

[0201] The first emotion intensity determination subunit is configured to perform an average operation on the emotion confidence of each target text cluster interval in the segment interval to obtain a first emotion intensity of the segment interval.

[0202] In some embodiments of the present application, based on the above embodiments, the second emotion intensity acquisition module includes:

[0203] A face detection unit is configured to perform face detection on each frame to be processed to obtain a face image corresponding to the frame to be processed;

[0204] The expression feature extraction unit is configured to extract features from the face image to obtain expression features corresponding to the frame to be processed;

[0205] An expression confidence determination unit is configured to determine the expression type and expression confidence corresponding to the frame to be processed based on the expression feature, where the expression confidence is the confidence that the text cluster interval is the expression type;

[0206] The second emotion intensity determining unit is configured to determine the second emotion intensity of the segment interval according to the expression confidence of the to-be-processed frame whose expression type belongs to the preset expression type in the segment interval.

[0207] In some embodiments of the present application, based on the above embodiments, the second emotion intensity determination unit includes:

[0208] a target frame determining unit configured to determine a to-be-processed frame in the segment interval whose expression type belongs to a preset expression type as a target frame;

[0209] The second emotion intensity determination subunit is configured to perform an average operation on the expression confidences of the target frames in the segment interval to obtain the second emotion intensity of the segment interval.

[0210] In some embodiments of the present application, based on the above embodiments, the video summary generation device further includes:

[0211] a reference interval determining unit configured to determine a first preset number of frames to be processed adjacent to a target frame to be processed as a reference interval of the target frame to be processed;

[0212] a frame number counting unit configured to count the number of frames to be processed corresponding to each expression type in a reference interval of the target frame to be processed;

[0213] The expression type correction unit is configured to correct the expression type of the target frame to be processed to the same expression type when the number of frames to be processed corresponding to the same expression type in the reference interval is greater than or equal to a second preset number, and the second preset number is less than the first preset number.

[0214] In some embodiments of the present application, based on the above embodiments, the video summary generation device further includes:

[0215] A video segment position acquisition unit is configured to acquire the start position and end position of each video segment in the video to be processed;

[0216] a first change value detection unit configured to shift a first preset time period forward from a starting position to intercept a first detection interval of the video to be processed, and calculate a change value of a color histogram between all adjacent video frames within the first detection interval;

[0217] a first video segment updating unit configured to, when a change value of a color histogram between adjacent video frames within a first detection interval exceeds a first preset value, use a later video frame among the adjacent video frames as a new starting position of the video segment to update the video segment;

[0218] a second change value detection unit configured to shift the end position back by a second preset time period to intercept a second detection interval of the video to be processed, and calculate a change value of the color histogram between all adjacent video frames in the second detection interval;

[0219] The second video segment updating unit is configured to update the video segment by using the preceding video frame among the adjacent video frames as the re-determined end position of the video segment when the change value of the color histogram between the adjacent video frames in the second detection interval exceeds a second preset value.

[0220] In some embodiments of the present application, based on the above embodiments, the video summary generation device further includes:

[0221] a segment interval sorting unit, configured to sort the segment intervals in descending order of sentiment scores;

[0222] a segment interval acquisition unit configured to acquire a preset target duration, and acquire one or more segment intervals from each segment interval in descending order of sentiment scores based on the target duration, so that a total duration of video segments obtained by intercepting the video to be processed according to the video time period corresponding to the acquired segment intervals is less than or equal to the target duration;

[0223] The preset value determining unit is configured to use the lowest emotion score among the acquired emotion scores of the segment intervals as the preset value.

[0224] The specific details of the video summary generation device provided in each embodiment of the present application have been described in detail in the corresponding method embodiments and will not be repeated here.

[0225] Figure 12 The structural block diagram of an electronic device for implementing an embodiment of the present application is schematically shown.

[0226] It should be noted that Figure 12 The electronic device 1200 shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0227] like Figure 12 As shown, the electronic device 1200 includes a central processing unit 1201 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 1202 (ROM) or the program loaded from the storage part 1208 into the random access memory 1203 (RAM). Various programs and data required for system operation are also stored in the random access memory 1203. The central processing unit 1201, the read-only memory 1202 and the random access memory 1203 are connected to each other via a bus 1204. An input / output interface 1205 (i.e., an I / O interface) is also connected to the bus 1204.

[0228] The following components are connected to the input / output interface 1205: an input section 1206 including a keyboard, a mouse, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1208 including a hard disk; and a communication section 1209 including a network interface card such as a local area network card or a modem. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the input / output interface 1205 as needed. Removable media 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1210 as needed, so that computer programs read therefrom can be installed into the storage section 1208 as needed.

[0229] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1209 and / or installed from a removable medium 1211. When the computer program is executed by the central processing unit 1201, the various functions defined in the system of the present application are performed.

[0230] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0231] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of the boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0232] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0233] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0234] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0235] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A video summary generation method, characterized in that: The method comprises: Acquire multiple frames to be processed from the video to be processed, and perform text recognition on each of the frames to be processed to obtain text content corresponding to the frames to be processed; Merging the frames to be processed according to the text contents corresponding to the frames to be processed to obtain a plurality of segment intervals, wherein the text contents of the segment intervals are determined by the text contents of the frames to be processed that are merged to form the segment intervals; Extracting features from the text content of the frames to be processed in each segment interval to obtain text features, and determining a first emotion intensity of the segment interval based on the text features, wherein the first emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval in terms of the text; Performing face detection on each of the frames to be processed to obtain a face image corresponding to the frame to be processed; Extracting features from the facial image to obtain facial features corresponding to the frame to be processed; Determining the expression type and expression confidence corresponding to the frame to be processed according to the expression feature, wherein the expression confidence is the confidence that the text cluster interval is the expression type; determining a first preset number of frames to be processed adjacent to a target frame to be processed as a reference interval of the target frame to be processed; Counting the number of frames to be processed corresponding to each expression type in the reference interval of the target frame to be processed; When the number of frames to be processed corresponding to the same expression type in the reference interval is greater than or equal to a second preset number, correcting the expression type of the target frame to be processed to the same expression type, the second preset number being less than the first preset number; Determining a second emotion intensity of the segment interval according to the expression confidence of the frames to be processed whose expression types belong to a preset expression type in the segment interval, wherein the second emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval in terms of expression; Performing a weighted sum operation on the first emotion intensity and the second emotion intensity to obtain an emotion score for the segment interval; According to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value, the video to be processed is intercepted to obtain video segments, and the video segments are spliced ​​into a video summary.

2. The method according to claim 1, characterized in that The steps of merging the frames to be processed according to the text contents corresponding to the frames to be processed to obtain a plurality of segment intervals include: When two adjacent frames to be processed both have text content, calculating the text similarity of the text contents of the two adjacent frames to be processed, and when the text similarity is greater than a preset threshold, merging the two adjacent frames to be processed into a single sentence interval, where the text content of the single sentence interval is determined by the text content of the two adjacent frames to be processed; When the time interval between two adjacent single sentence intervals is less than a first preset interval, merging the two adjacent single sentence intervals into one text cluster interval; When the time interval between two adjacent text cluster intervals is less than a second preset interval, the two adjacent text cluster intervals are merged into one segment interval.

3. The method according to claim 2, characterized in that The step of extracting features from the text content of the frames to be processed in each segment interval to obtain text features, and determining the first emotion intensity of the segment interval according to the text features, includes: Extracting features from the text content of each text cluster interval in each of the segment intervals to obtain text features; Determining the emotion type and emotion confidence corresponding to the text cluster interval according to the text feature, wherein the emotion confidence is the confidence that the text cluster interval is the emotion type; The first emotion intensity of the segment interval is determined according to the emotion confidence of the text cluster interval whose emotion type in the segment interval belongs to the preset emotion type.

4. The method according to claim 3, characterized in that The determining the first emotion intensity of the segment interval according to the emotion confidence of the text cluster interval whose emotion type in the segment interval belongs to the preset emotion type includes: Determining a text cluster interval whose emotion type in the segment interval belongs to a preset emotion type as a target text cluster interval; An average operation is performed on the sentiment confidences of the target text cluster intervals in the segment interval to obtain a first sentiment intensity of the segment interval.

5. The method according to claim 1, wherein The determining the second emotion intensity of the segment interval according to the expression confidence of the to-be-processed frame whose expression type in the segment interval belongs to a preset expression type includes: Determine the to-be-processed frame in the segment interval whose expression type belongs to a preset expression type as a target frame; An average operation is performed on the expression confidences of the target frames in the segment interval to obtain a second emotion intensity of the segment interval.

6. The method according to any one of claims 1 to 5, characterized in that After intercepting the video to be processed to obtain video segments according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to the preset value, the method further includes: Obtaining the starting position and ending position of each video segment in the video to be processed; According to the starting position, a first preset time period is shifted forward to intercept a first detection interval of the video to be processed, and a change value of a color histogram between all adjacent video frames in the first detection interval is calculated; When a change value of a color histogram between adjacent video frames within the first detection interval exceeds a first preset value, using a later video frame among the adjacent video frames as a new starting position of the video segment, and updating the video segment; According to the end position, a second detection interval of the video to be processed is intercepted by a second preset time period, and a change value of a color histogram between all adjacent video frames in the second detection interval is calculated; When a change value of a color histogram between adjacent video frames within the second detection interval exceeds a second preset value, the preceding video frame among the adjacent video frames is used as the re-determined end position of the video segment, and the video segment is updated.

7. The method according to any one of claims 1 to 5, characterized in that Before intercepting the video to be processed to obtain video segments according to the video time periods corresponding to the segment intervals whose emotion scores are greater than or equal to the preset value and splicing the video segments into a video summary, the method further includes: Sorting the segment intervals in descending order according to the sentiment score; Obtaining a preset target duration, and obtaining one or more segment intervals from each of the segment intervals according to the target duration and in descending order of the emotion scores, so that the total duration of the video segments obtained by intercepting the video to be processed according to the video time period corresponding to the obtained segment intervals is less than or equal to the target duration; The lowest emotion score among the acquired emotion scores of the segment intervals is used as the preset value.

8. A video summary generation device, characterized in that: The device comprises: A text recognition module is configured to obtain a plurality of frames to be processed from the video to be processed, and perform text recognition on each of the frames to be processed to obtain text content corresponding to the frames to be processed; a segment interval acquisition module configured to merge the frames to be processed according to the text contents corresponding to the frames to be processed to obtain a plurality of segment intervals, wherein the text contents of the segment intervals are determined by the text contents of the frames to be processed that are merged into the segment intervals; a first emotion intensity acquisition module configured to extract features from the text content of the frames to be processed in each segment interval to obtain text features, and determine a first emotion intensity of the segment interval based on the text features, wherein the first emotion intensity is used to represent the intensity of the emotion of each frame to be processed in the segment interval in terms of text; a second emotion intensity acquisition module configured to extract features of expressions in frames to be processed in each of the segment intervals to obtain expression features, and determine a second emotion intensity of the segment interval based on the expression features, wherein the second emotion intensity is used to represent the intensity of the emotion in the expression of each frame to be processed in the segment interval; an emotion score calculation module, configured to perform a weighted sum operation on the first emotion intensity and the second emotion intensity to obtain an emotion score for the segment interval; a video segment splicing module configured to intercept the video to be processed to obtain video segments according to the video time period corresponding to the segment interval whose emotion score is greater than or equal to a preset value, and to splice the video segments into a video summary; The second emotion intensity acquisition module includes: A face detection unit is configured to perform face detection on each of the frames to be processed to obtain a face image corresponding to the frame to be processed; An expression feature extraction unit is configured to extract features from the face image to obtain expression features corresponding to the frame to be processed; An expression confidence determination unit is configured to determine an expression type and an expression confidence corresponding to the frame to be processed according to the expression feature, wherein the expression confidence is a confidence that the text cluster interval is the expression type; A second emotion intensity determining unit is configured to determine a second emotion intensity of the segment interval according to the expression confidence of the to-be-processed frame in the segment interval whose expression type belongs to a preset expression type; The video summary generating device further includes: a reference interval determining unit configured to determine a first preset number of frames to be processed adjacent to a target frame to be processed as a reference interval of the target frame to be processed; a frame number counting unit configured to count the number of frames to be processed corresponding to each expression type in a reference interval of the target frame to be processed; The expression type correction unit is configured to correct the expression type of the target frame to be processed to the same expression type when the number of frames to be processed corresponding to the same expression type in the reference interval is greater than or equal to a second preset number, and the second preset number is less than the first preset number.

9. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video summary generation method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory for storing executable instructions of the processor; wherein the processor is configured to perform the video summary generation method according to any one of claims 1 to 7 by executing the executable instructions.

11. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the video summary generation method according to any one of claims 1 to 7.

Images