Method, device, equipment, and computer-readable storage medium for generating video summaries

Abstract

According to exemplary embodiments of this disclosure, a method, apparatus, device, and computer-readable storage medium for generating video summaries are provided, relating to the field of artificial intelligence, and particularly to the fields of knowledge graphs, deep learning, and computer vision. The specific implementation involves: acquiring at least one frame of an image from a video; determining content information of the at least one frame, where the content information represents the main content contained in the at least one frame; performing structured processing on the content information to generate structured information; and generating a video summary based on the structured information. Embodiments of this disclosure can efficiently and automatically generate video summaries for various types of videos, saving significant manpower.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a method, device, electronic device, computer-readable storage medium, and computer program product for generating a video summary. Background Art

[0002] In many cases, it is necessary to describe the content of a video to generate a summary of the video for subsequent retrieval or other applications. Taking the more popular variety videos as an example, when the entertainment industry splits, retrieves or re-uses variety videos, it needs to first know the main content of the video. For example, on video websites, it is necessary to introduce the key content of the video to attract users to click to browse the video, so a video summary will be provided to describe the video content. The video summary is of great significance to the number of video views and subsequent secondary use.

[0003] Currently, most of them rely on manual methods to watch each video and then write video summaries manually, which is time-consuming and labor-intensive. Summary of the invention

[0004] The present disclosure provides a method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product for generating a video summary.

[0005] According to a first aspect of the present disclosure, a method for generating a video summary is provided. The method comprises: obtaining at least one frame of image in a video; determining content information of at least one frame of image, wherein the content information represents the main content contained in at least one frame of image; performing structured processing on the content information to generate structured information; and generating a video summary based on the structured information.

[0006] In a second aspect of the present disclosure, a device for generating a video summary is provided. The device includes: an image acquisition module configured to acquire at least one frame of image in a video; a content information determination module configured to determine content information of at least one frame of image, wherein the content information represents the main content contained in at least one frame of image; a structured information generation module configured to perform structured processing on the content information to generate structured information; and a video summary generation module, which generates a summary of the video based on the structured information.

[0007] In a third aspect of the present disclosure, an electronic device is provided, comprising one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0009] In a fifth aspect of the present disclosure, a computer program product is provided. When the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0012] Figure 1 A schematic diagram illustrating an example environment in which various embodiments of the present disclosure can be implemented;

[0013] Figure 2 A flowchart for generating a video summary according to an embodiment of the present disclosure is shown;

[0014] Figure 3 A block diagram of a device for generating a video summary according to an embodiment of the present disclosure is shown;

[0015] Figure 4 A block diagram showing an apparatus for generating a video summary according to another embodiment of the present disclosure; and

[0016] Figure 5 A block diagram of a computing device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0019] As mentioned above, it is necessary to describe the content of the video to generate a summary of the video for subsequent retrieval or other applications. Currently, most people watch each video manually, then manually describe each video and generate a video summary. This is time-consuming and labor-intensive. In addition, for variety shows, since there are many people involved, how to organize the video content to obtain an appropriate description of the content is also a problem.

[0020] Moreover, for the same video, different people watching it will get different information, and even the information will be very different due to various reasons. Even if they get basically the same information, the video summary they describe will be very different due to different people's personal preferences, habits, etc.

[0021] In addition, different users have different preferences. For the same video, different users want to obtain different information. However, manually produced video summaries are less targeted and cannot provide targeted relevant video information for each user.

[0022] The above difficulties increase the difficulty of efficiently generating an ideal video summary. In order to solve the above problems, an embodiment of the present disclosure provides an improved scheme for generating a video summary. In this scheme, a computing device obtains at least one frame of image in a video. The computing device then analyzes the at least one frame of image obtained, and specifically determines the content information of the at least one frame of image, where the content information represents the main content contained in the at least one frame of image. Afterwards, the computing device performs structured processing on the content information to generate structured information. By generating structured information, it is convenient to perform various subsequent processing such as extraction of the information. The computing device generates a video summary based on the structured information. Through this scheme, the present invention can automatically generate video summaries for various types of videos efficiently, thereby saving a lot of manpower investment. At the same time, video summaries can be generated in a targeted manner according to user needs.

[0023] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0024] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Figure 11 is a schematic diagram of an example environment 100 in which various embodiments of the present disclosure can be implemented. Figure 1 As shown, the example environment 100 includes a video 110 , a computing device 120 , and a video summary 130 generated by the computing device 120 .

[0025] In some embodiments, the video 110 may be any video. For example, the video 110 may be a video stored locally on the computing device, or may be an externally input video, such as a video downloaded from the Internet, such as a variety show video, etc. The computing device 120 processes the video 110 to generate a video summary 130.

[0026] In some embodiments, the computing device 120 may have the function of extracting pictures from the video. For example, pictures are extracted from the video by the screenshot function of the video software in the computing device 120. In addition, in some embodiments, the computing device 120 may also use other related software to directly extract the required pictures from the video. In some embodiments, the computing device 120 may also be connected to an external image acquisition device to obtain images in the video (also referred to as pictures in this article). The computing device 120 then analyzes the extracted pictures, determines the main content in the pictures, and performs structured processing on the content information to generate structured information. Afterwards, based on the predetermined configuration information, the required structured information items can be obtained to generate the video summary 130.

[0027] In some embodiments, the computing device 120 may include, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), a media player, etc.), consumer electronics, a minicomputer, a mainframe computer, cloud computing resources, etc.

[0028] It should be understood that the structure and functionality of the example environment 100 is described for exemplary purposes only and is not intended to limit the scope of the subject matter described herein. The subject matter described herein may be implemented in different structures and / or functions.

[0029] The technical solutions described above are only for example and are not intended to limit the present invention. It should be understood that the example environment 100 may also have other time-sharing modes. Figure 2 Let’s describe the model training process in more detail.

[0030] Figure 2 FIG. 2 is a flowchart of a process 200 for generating a video summary according to an embodiment of the present disclosure. In some embodiments, the process 200 may be performed as follows: Figure 1 Implementation in the computing device 120. Figure 2 Combined with Figure 1The process 200 for generating a video summary according to an embodiment of the present disclosure is described. For ease of understanding, the specific examples mentioned in the following description are exemplary and are not intended to limit the protection scope of the present disclosure.

[0031] At 202, the computing device 120 obtains at least one frame of image in the video. For example, the computing device 120 may extract multiple frames of image (or referred to as pictures) in the video from the video 110 stored locally or from an external source. As mentioned above, the computing device 120 may extract images from the video 110 in a variety of ways. Automatically extracting images from the video 110 by the computing device 120 eliminates the need for manual viewing of the video 110, and significantly improves efficiency compared to manual viewing of the video 110.

[0032] In some embodiments, the video 110 may be frame-recognized at a predetermined period, that is, the screen of the video 110 may be extracted to obtain at least one frame of image. The acquired image is used for subsequent operations such as image recognition. In some embodiments, frame recognition (i.e., frame extraction) is performed at a second level. Frame recognition may be performed at different time levels according to actual needs. The higher the frequency of frame recognition, the more images are obtained, and more information about the video 110 can be obtained, which facilitates a more accurate description of the video 110.

[0033] In addition, in some embodiments, at least one frame of the acquired image is normalized to generate normalized images of the same size. For example, the size of all videos 110 can be 1920x1080. This is only an example, and the size of the video 110 can be set as needed. By normalizing different videos 110 to videos 110 of the same size, subsequent processing can be facilitated. For example, since all images have the same size, the same judgment criteria can be used in subsequent image recognition to improve the accuracy of recognition.

[0034] At 204, the computing device 120 may determine content information of at least one frame of image, where the content information represents the main content contained in at least one frame of image. The main content is the part of the image that is easily noticed or interesting to the viewer. For example, a person or object in a prominent position in the image, a person or animal in motion, a title or subtitle in the image, and so on. The computing device 120 may perceive the content in the image, thereby obtaining content information, i.e., various information displayed in the image. For example, objects such as people, houses, cars, and various actions, scenes, and so on in the image may all be perceived by the computing device 120.

[0035] In some embodiments, determining the content information of at least one frame of image includes: based on a deep learning method, identifying information of an object in at least one frame of image; and using the information of the identified object as content information. For example, the computing device 120 may have a target detection function for sensing objects in a picture, and analyzing and circling them. For example, location information of a person, and other object information: the location of a car, a house, and the location of some other notable objects. By identifying the information of an object in at least one frame of image by the computing device 120, important information in the video 110 is provided for the subsequent generation of a summary of the video 110. In this way, the need for manual viewing of the video 110 and obtaining information therefrom is eliminated.

[0036] In some embodiments, the computing device 120 has an OCR (Optical Character Recognition) function for recognizing text information in the image, including titles, subtitles, key descriptions, etc. For example, variety shows 110 often appear in the form of dialogues, and obtaining subtitle information is the key to ensuring a complete subsequent description.

[0037] In some embodiments, the computing device 120 has a celebrity recognition function, which is mainly used to identify celebrities in pictures. For variety shows, well-known figures usually participate. Therefore, it becomes meaningful to identify celebrities in pictures and describe them in the video summary. In some embodiments, for low-confidence recognition results, the title of the video can be combined to further determine whether to retain them. For example, in some embodiments, due to various reasons (such as insufficient image clarity and face angle), the credibility of the recognized celebrity is lower than a predetermined threshold, which can be further determined in combination with the title of the video. Because the title of the video usually gives some information. By combining with the title, the reliability of the low-confidence result can be further determined, thereby determining whether to retain or discard the recognition result.

[0038] In some embodiments, the computing device 120 has a face recognition function, which is mainly used to identify facial information in pictures. For low-confidence faces, further judgment can be made in combination with other information, such as the time, frequency, and location information of the face appearance.

[0039] In some embodiments, identifying an object in at least one frame of an image includes identifying a face in the image. When the confidence of the identified face is higher than a threshold, the face is determined to be a face. When the confidence of the identified face is lower than a threshold, it is determined whether the identified face is determined to be a face based on other information about the identified face in the content information. In this way, by combining the confidence of face recognition with other information in the image, faces can be recognized more accurately and the possibility of misjudgment can be reduced.

[0040] In some embodiments, determining the content information of at least one frame of image includes: identifying a face in at least one frame of image, generating a face seed set, the face seed set including unique faces corresponding to non-repeating persons appearing in at least one frame of image, and determining a timeline of each non-repeating person appearing in the video based on the time point at which each face in the face seed set appears.

[0041] In some embodiments, the computing device 120 may track the timeline information of each character appearing in the video 110. In some embodiments, by tracking the timeline information of each character appearing in the video 110, the activity context of each character in the scene of the video 110 may be understood, thereby facilitating description of information such as the plot of the video 110.

[0042] In some embodiments, in the process of tracking the timeline information of each person appearing in the video 110, a face seed set is generated. Generating the face seed set includes: identifying the first face in at least one frame of the image as the first seed in the face seed set. For example, first start from the beginning of the video 110 (it can be other ways), and obtain the first face as the first seed face. After that, the second face identified subsequently is matched with the first face; in the case of matching, if the match is successful, the second face is marked as the first face. In some embodiments, in order to generate a high-quality face seed set, the face with high recognizability between the second face and the first face is used as the first face seed (further described below). That is, in the process of generating the seed set, each seed in the seed set can be dynamically updated instead of being fixed. In the case of mismatch, the second face is added to the seed set as the second seed. Then the third face is identified and matched. By analogy, the above process is repeated, and the faces identified subsequently are matched with all the seed faces in the face seed set in turn. By matching the faces recognized at different time points in pairs, we will eventually obtain a face seed set. These faces are the only faces that appear in the video, that is, there is only one recognized face in the seed set for different people.

[0043] As mentioned earlier, in order to generate a high-quality face seed set, the highly recognizable faces among the different faces of the same person are used as face seeds. For high-quality face recognition, the seed face should be a representative face, such as a frontal face, a face of appropriate size, etc. Therefore, it is necessary to combine and weight information such as the angle at which the face appears, the proportion of the video, whether it is complete, and the location of the video as a score for the high-quality face. Then the faces are sorted according to the scores, and the face at the front, that is, the face with the highest score, is used as a face seed. In this way, high-quality face seeds can be obtained. By using high-quality face seeds, the newly identified face can be easily compared with the high-quality face seeds in subsequent face recognition (i.e., matching operation), thereby improving the efficiency and reliability of recognition.

[0044] In some embodiments, the recognized face is aligned with the celebrity recognition, that is, the recognized face is matched with the recognized celebrity. In the case where the recognized face is not a celebrity, the face is also matched with the person to determine its corresponding timeline and other information. In some embodiments, the time points of each face appearance are integrated to form a timeline of the person's appearance.

[0045] In some embodiments, the method for generating a video summary also includes extracting audio from the video, and performing speech recognition on the audio to generate text information of the speech. For example, the computing device 120 separates the audio portion from the video 110, and simultaneously segments the audio for the longer speech in the video 110. In some embodiments, the segmentation is performed using a time-sharing method. The time-sharing method is simply to segment the audio in the video 110 at a predetermined period. For example, the audio is segmented once every 5 minutes, that is, the audio is segmented into 5-minute segments. The advantage of the time-sharing method is that the operation is simple and fast.

[0046] In some embodiments, the spectral method is used for segmentation. The spectral method refers to a method for determining the performance of a signal by observing the change of the amplitude of the signal with the frequency characteristic (amplitude-frequency characteristic). In some cases, it also includes the change of the phase of the signal with the frequency characteristic (phase-frequency characteristic). Therefore, in some embodiments, the spectral method is used to segment the audio signal by observing the characteristics of the audio signal, which can ensure that the segmented audio is a complete sentence.

[0047] In some embodiments, the results of speech recognition are preliminarily screened, counted, and organized. For example, irrelevant information such as advertising information in the speech recognition text is filtered. The speech recognition text is segmented, keywords, and entities are recognized; that is, the words, keywords, and entities in the speech recognition text are distinguished. For example, people, animals, cars, houses, and other various objects in the video 110 are recognized.

[0048] In some embodiments, the texts are arranged in time order, and the speech text is aligned with the time position of the speech in the video 110. That is, the speech is consistent with the original video 110 in time.

[0049] In this way, by extracting audio from video 110 and performing speech recognition on the audio to generate text information of the speech, the text information obtained from the audio can be combined with the text information obtained through the image of video 110 in subsequent processing, thereby more accurately identifying the content in video 110.

[0050] In some embodiments, determining the content information of at least one frame of image further includes: matching the text information with the content information in a corresponding time period; and determining the successfully matched portion as valid content information. In this way, by matching the text information obtained from the audio with the text information obtained through the video image, the accuracy of the recognized content can be ensured.

[0051] In some embodiments, the result of OCR recognition is matched with the result of speech recognition. If the text information at the corresponding OCR time point happens to be in the speech text, that is, the speech recognized text and the OCR recognized text can correspond, then the subtitle is considered valid in this case. Otherwise, the result can be discarded. In this way, the reliability of the recognized subtitles can be ensured. In some embodiments, even if the speech recognized text and the OCR recognized text cannot completely correspond, the result may not be simply discarded. Instead, for example, the text recognized by OCR can be used as the standard, and a certain confidence level can be assigned to it, such as 80%, that is, the credibility of the recognition result is 89%. At this point, the recognition solution can be further verified in combination with other information. The above-described method is only an example, and the embodiments of the present disclosure are not limited to the above-mentioned method, but may have a variety of variations.

[0052] At 206, the computing device 120 may perform structured processing on the content information to generate structured information. The structured information may be structured information in a predetermined format to represent content information of various modalities. Structured information may also be referred to as structured data, which is generally data logically expressed and implemented by a two-dimensional table structure, strictly following data format and length specifications, and is mainly stored and managed through a relational database. The identified content information is originally unordered. In some embodiments, by generating structured information from the identified content, the corresponding structured information may be extracted as needed in the subsequent process of generating a video summary. In this way, it is possible to conveniently generate a video summary later.

[0053] In some embodiments, generating structured information based on content information includes: structuring text information to generate structured text information; and aligning the structured text information with a corresponding time period of the video. In this way, the text obtained by speech recognition can be associated with the timeline in the video, thereby facilitating the subsequent generation of a video summary.

[0054] In some embodiments, generating structured information includes: obtaining the speech content of the person in the video 110 based on the text information and the content information; matching the obtained speech content with the key information in the content information to determine the association relationship between each key information and the person; and combining the key information based on the association relationship and the timeline to obtain the structured information of the video 110. In this way, the text information obtained by speech recognition can be combined with the text information obtained by image recognition, thereby improving the reliability of recognition.

[0055] At 208, the computing device 120 can generate a summary of the video 110 based on the structured information. For example, after obtaining relatively complete structured information of the video 110, the data can be extracted and organized according to established standards as needed. In some embodiments, basic standard information for generating a summary is obtained from the configuration information of the video type (pre-set configuration information, such as a configuration template, which contains various fields, such as video name, main characters, video category, year, etc.). For example, for a variety show video of a competition, it may be desired that the fields of the video description contain information such as participants, competition name, competition rules, and results. Corresponding information is extracted from the structured information based on the fields of these video descriptions. If there is no pre-specified configuration information, only the title or all the information in the video can be given:

[0056] In some embodiments, generating a summary of the video 110 based on structured information includes: obtaining corresponding information from the structured information based on predetermined configuration information, wherein the predetermined configuration information specifies the type of structured information to be obtained. For example, different configuration information can be set to specify different types of structured information according to different user needs. For example, for variety shows, celebrity information can be particularly highlighted. For detective videos, the twists and turns of the plot can be particularly highlighted. Afterwards, the corresponding information is organized based on a predetermined template or through a language model based on deep learning (e.g., a language model such as gpt3) to form a video summary.

[0057] In some embodiments, the relevant description is extracted from the structured data according to the standard information mentioned above. The extracted results are organized to form a summary of the video. In some embodiments, the structured data generated previously can be used directly. In some embodiments, if you want to generate a paragraph of text separately, you can also use the following scheme: a template-based method, such as the first paragraph of the text introduces the contestants, the second paragraph introduces the rules of the game, and the third paragraph introduces the performance. Or you can use a deep learning method, such as using a statistical language model, such as gpt3, to generate controllable text. For example, the structured information is "Zhang San", "highway" and "car". Then the controllable text generation can be, for example: "Zhang San is driving a car on the road". In this way, video summaries can be generated flexibly and efficiently.

[0058] Through the above embodiments, a method for generating video summaries is proposed. On the one hand, the present disclosure can efficiently and automatically generate video summaries for various videos, avoiding or significantly reducing manual participation, thereby reducing a large amount of labor costs. On the other hand, the present disclosure can generate video summaries in a targeted and flexible manner according to user needs, thereby improving user satisfaction.

[0059] Figure 3 FIG. 1 is a block diagram of an apparatus for generating a video summary according to an embodiment of the present disclosure. Figure 3 As shown, the device 300 may include: an image acquisition module 310, configured to acquire at least one frame of image in a video; a content information determination module 320, configured to determine content information of at least one frame of image, wherein the content information represents main content contained in at least one frame of image; a structured information generation module 330, configured to perform structured processing on the content information to generate structured information; and a video summary generation module 340, generating a summary of the video based on the structured information.

[0060] In some embodiments, the image acquisition module 310 may include: an image recognition module, configured to perform frame recognition on the video at a predetermined period to acquire at least one frame of image; and an image normalization module, configured to perform normalization processing on the acquired at least one frame of image to generate a normalized image with the same size.

[0061] In some embodiments, the content information determination module 320 may include: an object information recognition module configured to recognize information of an object in at least one frame of image based on a deep learning method; and an object information determination module configured to use the recognized object information as content information.

[0062] In some embodiments, the object information recognition module may include: a face recognition module configured to recognize a face in an image, wherein if the confidence of the recognized face is higher than a threshold, the face is determined to be a face; and if the confidence of the recognized face is lower than the threshold, it is determined whether to determine the recognized face as a face based on other information about the recognized face in the content information.

[0063] In some embodiments, the content information determination module may include: a face seed set generation module, configured to identify faces in at least one frame of image and generate a face seed set, the face seed set including unique faces corresponding to non-repeating persons appearing in at least one frame of image; and a timeline determination module, configured to determine the timeline of each non-repeating person appearing in the video based on the time point when each face in the face seed set appears.

[0064] In some embodiments, a face seed set generation module may include: a face recognition module, configured to identify a first face in at least one frame of an image as a first seed in a face seed set; a face matching module, configured to match a subsequently identified second face with the first face; in the event of a match, using the more recognizable face between the second face and the first face as the first face seed; in the event of a mismatch, adding the second face as a second seed to the seed set; and repeating the above process to match the subsequently identified faces with all seed faces in the face seed set in turn to generate a face seed set.

[0065] In some embodiments, the apparatus for generating a video summary further includes: an audio extraction module configured to extract audio from the video; and a speech recognition module configured to perform speech recognition on the audio to generate text information of the speech.

[0066] In some embodiments, the content information determination module may include: a matching module configured to match text information with content information in a corresponding time period; and a valid content information determination module configured to determine a successfully matched portion as valid content information.

[0067] In some embodiments, the structured information generation module may include: a structured text information generation module configured to perform structural processing on text information to generate structured text information; and an alignment module configured to align the structured text information with a corresponding time period of the video.

[0068] In some embodiments, the structured information generation module may include: a speech content acquisition module, which is configured to acquire the speech content of a character in a video based on text information and content information; a key information matching module, which matches the acquired speech content with the key information in the content information to determine the association relationship between each key information and the character; and a key information combination module, which is configured to combine the key information based on the association relationship and timeline to obtain the structured information of the video.

[0069] In some embodiments, the video summary generation module includes: a structured information acquisition module, configured to acquire corresponding information from the structured information based on predetermined configuration information, wherein the predetermined configuration information specifies the type of structured information to be acquired; and a structured information organization module, configured to organize the corresponding information based on a predetermined template or through a language model based on deep learning to form a video summary.

[0070] Through the above embodiments, a device for generating a video summary is proposed. On the one hand, the device for generating a video summary of the embodiment of the present disclosure can efficiently and automatically generate video summaries for various videos, avoiding or significantly reducing manual participation, thereby reducing a large amount of labor costs. On the other hand, the present disclosure can generate video summaries in a targeted and flexible manner according to user needs, thereby improving user satisfaction.

[0071] In order to more clearly demonstrate the technical solution of the present disclosure, one specific implementation method of the present disclosure will be described below with reference to the drawings. Figure 4 A schematic block diagram of an apparatus for generating a video summary according to an embodiment of the present disclosure is shown.

[0072] like Figure 4As shown, in some embodiments, the apparatus 400 for generating a video summary may include: an image acquisition module 310, configured to acquire at least one frame of image in a video. A perception module 440, configured to perceive content information in at least one frame of image. In some embodiments, the perception module 440 includes a target detection unit 442, which is configured to detect targets in the image, such as people, animals and other various objects. The perception module 440 also includes an OCR unit, and the OCR unit 444 is configured to recognize text information in the image, such as subtitle information. In some embodiments, the perception module 440 also includes a celebrity recognition unit 446, which is configured to recognize celebrities in the image, which is particularly suitable for the recognition of variety shows. In some embodiments, the perception module 440 also includes a face recognition unit 448, which is configured to recognize faces in the image. In some embodiments, the apparatus 400 may include a speech extraction module 410, which is configured to extract audio from the video. In some embodiments, the device 400 may include a speech text generation module 420, which is configured to perform speech recognition on the audio extracted by the speech extraction module 410 to generate text information of the speech. The key content acquisition module 430 is configured to obtain key content information from the speech text generated by the speech text generation module 420 and from the content information perceived by the perception module 440. In some embodiments, the device 400 may include a character tracking module 450, which is configured to track the timeline information of the characters in the video. The character content matching module 460 is configured to merge the characters with the key information previously acquired. In some embodiments, the device 400 may include a video information configuration module 470, which is configured to pre-set the configuration information of the video summary. For example, the various fields contained in the configuration template, such as the video name, main characters, video category, year, etc. In some embodiments, the apparatus 400 may include a content organization module 480, which is configured to organize the previously generated structured information, for example, extract relevant descriptions from the structured data according to the information configured by the video information configuration module 470, and arrange them in a predetermined format, etc. In some embodiments, the apparatus 400 may include a video summary generation module 340, which is configured to generate a video summary based on the structured information output from the content organization module 480.

[0073] In some embodiments, the perception module 440 is configured to achieve character-OCR-voice matching, combining the time point of the character's appearance with the voice, and identifying the sound through a deep learning model to determine the voice information of the character. Then, the character's speech content is determined based on the recognized voice and the subtitle information recognized by OCR. The determined speech content is matched with the extracted key information to determine the ownership of each key information, that is, to determine which person or people each key information belongs to. In addition, the key information is recombined according to the time and character lines to obtain all the structured information of the video content. By setting the structured information, the corresponding structured information can be conveniently extracted as needed in the subsequent process of generating the visual summary.

[0074] Figure 5 A block diagram of a computing device 500 capable of implementing multiple embodiments of the present disclosure is shown. Electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0075] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0076] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0077] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as processes 200 and 300. For example, in some embodiments, processes 200 and 300 may be implemented as computer software programs that are tangibly included in machine-readable media, such as storage units 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the processes 200 and 300 described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the processes 200 , 300 in any other appropriate manner (eg, by means of firmware).

[0078] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0079] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0080] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0081] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0082] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0083] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0084] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0085] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a video summary, comprising: Acquire at least one frame of image in the video; Determining content information of the at least one frame of image, the content information indicating main content contained in the at least one frame of image, the main content including text content, objects, scenes, and action information; Performing structural processing on the content information to generate structured information; Generating a video summary in text form based on the structured information; The structured information is structured information in a predetermined format to represent content information of various modes; extracting audio from the video; Performing speech recognition on the audio to generate text information of the speech; Performing structural processing on the text information to generate structured text information; Aligning the structured text information with a corresponding time period of the video; Generating structured information includes: obtaining the speech content of the characters in the video based on the text information and the content information; Matching the acquired speech content with key information in the content information to determine the association relationship between each key information and the character; and Based on the association relationship and the timeline, the key information is combined to obtain the structured information of the video.

2. The method according to claim 1, wherein acquiring at least one frame of image in the video comprises: Performing frame recognition on the video at a predetermined period to obtain the at least one frame of image; as well as The at least one frame of image acquired is normalized to generate normalized images with the same size.

3. The method according to claim 1, wherein determining the content information of the at least one frame of image comprises: Based on a deep learning method, identifying information of an object in the at least one frame of image; as well as The information of the identified object is used as the content information.

4. The method according to claim 3, wherein identifying the object in the at least one frame of image comprises: identifying a face in the image; When the confidence level of the recognized human face is higher than a threshold, determining the human face as a human face; as well as When the confidence level of the recognized face is lower than a threshold, it is determined whether to determine the recognized face as a human face based on other information about the recognized face in the content information.

5. The method according to claim 1, wherein determining the content information of the at least one frame of image comprises: Identifying a human face in the at least one frame of image, and generating a human face seed set, wherein the human face seed set includes unique human faces corresponding to non-repeated persons appearing in the at least one frame of image; and Based on the time points at which each face in the face seed set appears, a timeline at which each non-repetitive person appears in the video is determined.

6. The method according to claim 5, wherein generating a face seed set comprises: Identify a first human face in the at least one frame of image as a first seed in the human face seed set; matching a second face identified subsequently with the first face; In case of a match, taking the second face and the first face, whichever is more recognizable, as the first face seed; In case of mismatch, adding the second face as a second seed to the seed set; as well as The above process is repeated to match subsequently recognized faces with all seed faces in the face seed set in turn to generate the face seed set.

7. The method according to claim 1, wherein determining the content information of the at least one frame of image comprises: Matching the text information with the content information in a corresponding time period; as well as The successfully matched portion is determined as the effective content information.

8. The method of claim 1, wherein generating a summary of the video based on the structured information comprises: Based on predetermined configuration information, obtaining corresponding information from the structured information, wherein the predetermined configuration information specifies the type of structured information to be obtained; as well as The corresponding information is organized based on a predetermined template or through a language model based on deep learning to form the video summary.

9. A device for generating a video summary, comprising: An image acquisition module, configured to acquire at least one frame of image in a video; A content information determination module is configured to determine content information of the at least one frame of image, wherein the content information represents main content contained in the at least one frame of image, and the main content includes text content, objects, scenes, and action information; A structured information generating module, configured to perform structured processing on the content information to generate structured information; A video summary generation module, which generates a summary of the video in text form based on the structured information; The structured information is structured information in a predetermined format to represent content information of various modes; An audio extraction module, configured to extract audio from the video; A speech recognition module, configured to perform speech recognition on the audio and generate text information of the speech; A structured text information generating module is configured to perform structural processing on the text information to generate structured text information; an alignment module, configured to align the structured text information with a corresponding time period of the video; The structured information generation module includes: A speech content acquisition module, configured to acquire the speech content of the character in the video based on the text information and the content information; A key information matching module matches the acquired speech content with the key information in the content information to determine the association relationship between each key information and the character; and The key information combination module is configured to combine the key information based on the association relationship and the timeline to obtain the structured information of the video.

10. The device according to claim 9, wherein the image acquisition module comprises: An image recognition module is configured to perform frame recognition on the video at a predetermined period to obtain the at least one frame of image; as well as The image normalization module is configured to perform normalization processing on the acquired at least one frame of image to generate a normalized image with the same size.

11. The apparatus according to claim 9, wherein the content information determination module comprises: An object information recognition module is configured to recognize information of an object in the at least one frame of image based on a deep learning method; as well as The object information determination module is configured to use the identified information of the object as the content information.

12. The apparatus according to claim 11, wherein the object information recognition module comprises: A face recognition module is configured to recognize a face in the image; wherein, when the confidence of the recognized face is higher than a threshold, the face is determined to be a human face; and when the confidence of the recognized face is lower than a threshold, it is determined whether to determine the recognized face as a human face based on other information about the recognized face in the content information.

13. The device according to claim 9, wherein the content information determination module comprises: a face seed set generating module, configured to identify faces in the at least one frame of image, and generate a face seed set, wherein the face seed set includes unique faces corresponding to non-repeated persons appearing in the at least one frame of image; and The timeline determination module is configured to determine the timeline at which the non-repeating persons appear in the video based on the time points at which the respective faces in the face seed set appear.

14. The apparatus according to claim 13, wherein the face seed set generation module comprises: A face recognition module, configured to recognize a first face in the at least one frame of image as a first seed in the face seed set; as well as The face matching module is configured to match a second face identified subsequently with the first face; in the case of a match, use the second face and the first face with a higher recognizability as the first face seed; in the case of a mismatch, add the second face as a second seed to the seed set; and repeat the above process to match the subsequently identified face with all the seed faces in the face seed set in turn to generate the face seed set.

15. The apparatus according to claim 9, wherein the content information determination module comprises: a matching module, configured to match the text information with the content information in a corresponding time period; as well as The effective content information determination module is configured to determine the successfully matched part as the effective content information.

16. The apparatus according to claim 9, wherein the video summary generation module comprises: A structured information acquisition module, configured to acquire corresponding information from the structured information based on predetermined configuration information, wherein the predetermined configuration information specifies the type of structured information to be acquired; as well as The structured information organization module is configured to organize the corresponding information based on a predetermined template or through a language model based on deep learning to form the video summary.

17. An electronic device, comprising: one or more processors; as well as A storage device, used for storing one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 8.

18. A computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images