Video processing method and device

Abstract

The embodiment of the invention provides a video processing method and device, computer equipment, a computer readable storage medium and a computer program product, and relates to the technical field of computers. The video processing method comprises the following steps: acquiring a target video material, and performing voice recognition on the target video material; segmenting the target video material according to a recognition result of the voice recognition to obtain a plurality of video slices; and generating video slice index information according to the plurality of video slices, wherein the video slice index information is used for video slice indexing. According to the technical scheme provided by the embodiment of the invention, a video identification segmentation and information extraction technology can be implemented on an end side (local client).

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a video processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In today's era of digital information explosion, video data is experiencing massive growth. Video editing plays a crucial role in numerous fields, from personal creation and media dissemination to education and teaching. The video editing process often involves video recognition and slicing, as well as extracting video information (video index information), which requires significant computing power. Existing technologies typically use large cloud-based models for video recognition, segmentation, and information extraction. Therefore, there is a need for video recognition, segmentation, and information extraction technologies that can be implemented on-device.

[0003] It should be noted that the above content is not necessarily prior art, nor is it intended to limit the scope of patent protection of this application. Summary of the Invention

[0004] The embodiments of the present application provide a video processing method, apparatus, computer device, computer-readable storage medium, and computer program product to solve or alleviate one or more of the technical problems raised above.

[0005] One aspect of an embodiment of the present application provides a video processing method for a client, the method comprising: Acquire target video material and perform speech recognition on the target video material; According to the recognition result of the speech recognition, the target video material is segmented to obtain a plurality of video slices; and Video slice index information is generated according to the multiple video slices, and the video slice index information is used for video slice indexing.

[0006] Optionally, the multiple video slices include multiple primary video slices and multiple secondary video slices; Segmenting the target video material to obtain a plurality of video slices according to a recognition result obtained by the speech recognition, including: In the case where a dialogue text is obtained through speech recognition, determining the content outline of the target video material based on the semantic understanding of the dialogue text; According to the content outline, the target video material is divided into a plurality of first-level video slices; According to the sentence semantic understanding and sentence timestamps in the dialogue text, the multiple first-level video slices are respectively segmented to obtain multiple second video slices; wherein each of the first-level video slices includes one or more second-level video slices.

[0007] Optionally, the plurality of second video slices include a plurality of conversation video slices and a plurality of non-conversation video slices; the plurality of video slices further include a plurality of tertiary video slices; and the method further includes: Local object recognition is performed on each dialogue video slice through the character's face, and each dialogue video slice is divided according to the recognition result to obtain multiple third video slices; wherein each dialogue video slice includes one or more third-level video slices, and the video slice index information of each third-level video slice can be configured with the character information in the slice.

[0008] Optionally, generating video slice index information according to the multiple video slices includes: Generate slice description information for each conversation video slice or the three-level video slices obtained by segmenting it, wherein the slice description information includes position information; Generate video understanding information for each dialogue video slice or its three-level video slices using the timestamp of the character's face; Corresponding video slice index information is generated based on each dialogue video slice or the three-level video slice obtained by segmenting it, as well as the slice description information and the video understanding information; the corresponding video slice index information is stored through multiple fields of the index table.

[0009] Optionally, generating video slice index information according to the multiple video slices includes: Extracting frames from each non-dialogue video slice or the third-level video slices obtained by segmenting the non-dialogue video slice to obtain multiple target frames; and Through the local recognition model, image recognition is performed on each target frame to obtain the corresponding video slice index information.

[0010] Optionally, the multiple video slices include multiple first-level video slices and multiple third-level video slices; Segmenting the target video material to obtain a plurality of video slices according to a recognition result obtained by the speech recognition, including: In the case where the speech recognition obtains a dialogue text, determining the content outline of the target video material according to the dialogue text; According to the content outline, the target video material is divided into a plurality of first-level video slices; Taking the shot change as the segmentation point, each first-level video slice is divided into multiple third-level video slices; Each of the first-level video slices includes multiple third-level video slices.

[0011] Optionally, the plurality of video slices include a plurality of tertiary video slices; Segmenting the target video material to obtain a plurality of video slices according to a recognition result obtained by the speech recognition, including: In the case where voice recognition obtains narration text or blank text, the target video material is divided into the plurality of three-level video slices based on shot changes.

[0012] Optionally, generating video slice index information according to the multiple video slices includes: Extracting frames from each three-level video slice to obtain multiple target frames; and Through the local recognition model, image recognition is performed on each target frame to obtain the corresponding video slice index information.

[0013] Optionally, the multiple video slices include primary video slices, secondary video slices, and shot-level tertiary video slices; the video index information includes video index information of the primary video slices, video index information of the secondary video slices, and video index information of the tertiary video slices; the method further includes: Determine the storyboard description information of the shot to be indexed; According to the storyboard description information, a target first-level video slice is obtained by indexing from a plurality of first-level video slices of the target video material; According to the storyboard description information, a target secondary video slice is obtained by indexing from multiple secondary video slices of the target primary video slice; According to the shot description information, a target tertiary video slice is obtained by indexing from multiple tertiary video slices of the target secondary video slice; wherein the target tertiary video slice is a video slice corresponding to the shot to be indexed.

[0014] Another aspect of an embodiment of the present application provides a video processing device for a client, the device comprising: An acquisition module, configured to acquire target video material and perform speech recognition on the target video material; a segmentation module, configured to segment the video material to obtain a plurality of video slices according to the recognition result of the speech recognition; and A generating module is used to generate video slice index information according to the multiple video slices, and the video slice index information is used for video slice indexing.

[0015] Another aspect of an embodiment of the present application provides a computer device, including: at least one processor; and a memory communicatively coupled to the at least one processor; Wherein: the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0016] Another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method described above is implemented.

[0017] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned method when executed by a processor.

[0018] The embodiments of this application employing the above-mentioned technical solution offer the following advantages: By performing speech recognition on the target video material, the target video material can be segmented based on the speech recognition results. Compared to direct video recognition and segmentation solutions, video segmentation based on speech recognition results requires less computing power and can be implemented on the client side. Furthermore, video slice index information can be generated on the client side based on the segmented video slices. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings illustrate exemplary embodiments and constitute a part of the specification. Together with the description of the specification, they serve to explain exemplary implementation of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals designate similar, but not necessarily identical, elements.

[0020] Figure 1 The following schematically shows an operating environment diagram of the video processing method according to the first embodiment of the present application; Figure 2 The flowchart of the video processing method according to the first embodiment of the present application is schematically shown; Figure 3 Schematically shows the Figure 2 Sub-flowchart of step S202; Figure 4 Schematically shows the Figure 2 Sub-flowchart of step S202; Figure 5 Schematically shows the Figure 2 Sub-flowchart of step S204; Figure 6 Schematically shows the Figure 2 Sub-flowchart of step S204; Figure 7 Schematically shows the Figure 2 Sub-flowchart of step S204; Figure 8 The following schematically shows a newly added flow chart of the video processing method according to the first embodiment of the present application; Figure 9 Schematically shows an exemplary application diagram of the video processing method according to the first embodiment of the present application; Figure 10 A block diagram schematically shows a video processing device according to a second embodiment of the present application; and Figure 11 The following schematically shows a hardware architecture diagram of a computer device according to the third embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of this application more clear, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] It should be noted that the descriptions of "first", "second", etc. in the embodiments of the present application are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0023] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order in which the steps are executed. They are only used to facilitate the description of this application and to distinguish each step. Therefore, they cannot be understood as limitations on this application.

[0024] First, an explanation of the terms involved in this application is provided: ASR (Automatic Speech Recognition): Automatic speech recognition, used to convert human speech into text.

[0025] The Large Language Model (LLM) is an AI model trained on large-scale datasets, boasting powerful language understanding and generation capabilities. By learning patterns and regularities from massive amounts of text data, it can accurately analyze input text and generate responses. In terms of video understanding, the LLM can be combined with a visual encoder to convert video frames into textual information, understanding the content, events, and semantics of the video. It can also infer, summarize, and answer questions based on the video content, achieving in-depth understanding and analysis of the video.

[0026] This embodiment of the present application provides a video processing technology solution. In this technology solution, videos can be automatically identified and segmented on the client side (locally), and corresponding video index information can be extracted from the segmented video slices. Details are described below.

[0027] Finally, for ease of understanding, an exemplary operating environment is provided below.

[0028] like Figure 1 As shown, client 10000 includes various types of electronic devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, smart screen devices, self-service kiosk devices, gaming systems, or other terminal devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (such as Google Chrome OS); or various mobile operating systems, such as Microsoft Windows, MobileOS, iOS, Windows Phone, and Android. Portable handheld devices can include cellular phones, smartphones, tablets, personal digital assistants, etc. Wearable devices can include head-mounted displays (such as smart glasses), etc. Gaming systems can include various handheld gaming devices and internet-enabled gaming devices. Client devices are capable of executing a variety of different applications, such as various internet-related applications, communication applications (such as email applications), and short message service (SMS) applications, and can use various communication protocols.

[0029] The client 10000 may include an input / output interface. The input interface may include a touch pad, a touch screen, a mouse, a keyboard, or other sensing elements. The input interface may be configured to receive user instructions, which may cause the client to perform various operations, such as Video Editing Etc. The output interface is used to output information to the user, such as display information.

[0030] Based on the operating environment described above, the client 10000 may also be installed with one or more application programs, such as a video editing program.

[0031] The following describes the technical solutions of the present application through multiple embodiments, taking the client 10000 as the execution subject. It should be noted that these embodiments can be implemented in many different forms and should not be construed as being limited to the embodiments described here.

[0032] Example 1 Figure 2 The flowchart of the video processing method according to the first embodiment of the present application is schematically shown.

[0033] like Figure 2 As shown, the video processing method may include steps S200~S200, wherein: Step S200: Acquire target video material and perform speech recognition on the target video material.

[0034] Step S202: Segment the target video material to obtain a plurality of video slices according to the recognition result of the speech recognition.

[0035] Step S204: Generate video slice index information according to the multiple video slices, where the video slice index information is used for video slice indexing.

[0036] The video processing method provided in this embodiment performs speech recognition on target video material and then segments the target video material based on the speech recognition results. Compared to direct video recognition and segmentation solutions, video segmentation based on speech recognition results requires less computing power and can be implemented on the client side. Furthermore, video slice index information can be generated on the client side based on the segmented video slices.

[0037] According to the video slice index information, it is possible to quickly locate and search for the required video slice in the local resources.

[0038] The following combination Figure 2 , each step in steps S200~S204 and other optional steps are described in detail.

[0039] Step S200 , obtain target video material, and perform speech recognition on the target video material.

[0040] The target video material may refer to the original video used to create the target video. The target video material may be obtained locally or downloaded from a server, without limitation. Speech recognition can transcribe speech content into corresponding text, which can then be analyzed to segment the target video material. In some embodiments, ASR can be used to perform speech recognition on the video material to convert the audio content in the video material into text.

[0041] Step S202 , according to the recognition result of the speech recognition, the target video material is segmented to obtain multiple video slices.

[0042] The recognition result can be the text corresponding to the speech content of the target video material. To better segment the target video material, the text corresponding to the recognition result can be marked, further divided into speech-containing portions and blank portions. The text corresponding to the speech-containing portions can then be divided into dialogue-type text and narration-type text, and the text corresponding to the blank portions can be classified as blank text. It should be noted that the text corresponding to the target video material can include one or more text types, such as dialogue between characters (corresponding to dialogue-type text), monologues of individual characters (corresponding to narration-type text), and close-ups of scenes without speech (corresponding to blank text). The target video material corresponding to different text types can be segmented differently. In some embodiments, the text corresponding to the recognition result can also be segmented into dialogue-rich text, dialogue-sparse text, and non-dialogue text. Dialogue-rich text can correspond to dialogue-type video slices, dialogue-sparse text can correspond to narration-type video slices, and non-dialogue text can correspond to non-speech video slices.

[0043] The following will provide further exemplary introductions to segmenting target video materials corresponding to different text types in different ways through more embodiments.

[0044] In an optional embodiment, the plurality of video slices include a plurality of primary video slices and a plurality of secondary video slices; Figure 3 As shown, step S202 includes: Step S300: When a dialogue text is obtained through speech recognition, the content outline of the target video material is determined based on the semantic understanding of the dialogue text.

[0045] Step S302: dividing the target video material into a plurality of first-level video slices according to the content outline.

[0046] Step S304: segment the plurality of first-level video slices respectively according to the sentence semantic understanding and sentence timestamps in the dialogue text to obtain a plurality of second video slices; wherein each of the first-level video slices includes one or more second-level video slices.

[0047] Based on the dialogue text, a content outline can be generated in the dimension of advancing events, and each first-level video slice after segmentation can correspond to an event. Based on the semantic understanding of sentences and sentence timestamps in the dialogue text, the content outline can be further segmented into detailed outlines in the narrative main mirror dimension, and then each first-level video slice can be segmented to obtain multiple second-level video slices. Taking the video material authorized for identification by the user as an example, the lines in the video material can be analyzed to divide the series into multiple events, and the video material can be segmented according to these multiple events, with each event corresponding to a first-level video slice. Subsequently, the first-level video slice can be segmented into multiple second-level video slices based on the lines of each character in the video material.

[0048] In this embodiment, based on the semantic understanding of sentences in dialogue texts, the target video material can be effectively segmented with low computing power requirements.

[0049] Secondary video slices can be divided in the dimension of storyboards.

[0050] In an optional embodiment, the multiple second video slices include multiple dialogue video slices and multiple non-dialogue video slices; the multiple video slices also include multiple tertiary video slices; the method also includes: performing local object recognition on each dialogue video slice through the character's face, and dividing each of the dialogue video slices according to the recognition results to obtain multiple third video slices; wherein each of the dialogue video slices includes one or more tertiary video slices, and the video slice index information of each tertiary video slice can be configured with the character information in the slice.

[0051] Character faces can be facial image information of characters appearing in the target video footage. Since conversation video slices may include cuts between multiple characters, the conversation video slices can be further segmented based on character faces. To facilitate local object recognition, character faces can be pre-delivered, meaning the character facial image information is pre-matched with the corresponding character information. In some embodiments, local object recognition can be achieved using a pre-deployed facial recognition model. The facial recognition model is used to detect faces from images or videos, extract features, and perform identity recognition or verification.

[0052] In this embodiment, effective segmentation points can be found in the dialogue video slices quickly and with low computational cost through the character's face.

[0053] The above describes a three-level splitting mode. This mode first splits the video into primary video slices, then splits the primary video slices into secondary video slices, and then splits the secondary video slices into tertiary video slices. The following describes another splitting method.

[0054] In an optional embodiment, the plurality of video slices include a plurality of primary video slices and a plurality of tertiary video slices. Figure 4 As shown, step S202 further includes: Step S400: When a dialogue text is obtained through speech recognition, a content outline of the target video material is determined according to the dialogue text.

[0055] Step S402: dividing the target video material into a plurality of first-level video slices according to the content outline.

[0056] Step S404 : dividing each first-level video slice into a plurality of third-level video slices based on shot changes as segmentation points.

[0057] After segmenting the target video material into multiple first-level video slices using the event dimension, each first-level video slice can be segmented directly using the storyboard dimension, using shot changes as the segmentation points. Shot changes can include screen cuts, changes in scene size, and so on. In some embodiments, the first-level video slices can be segmented into multiple third-level video slices using a local video shot cut detection model. The local video shot cut detection model is a deep learning model that can accurately identify shot boundaries in a video (such as scene cuts and transition effects).

[0058] In this embodiment, the target video material is segmented using the content outline to ensure the content coherence of each first-level video slice. Furthermore, the first-level video slices can also be locally segmented based on shot changes to obtain third-level video slices based on single continuous shots, facilitating subsequent accurate indexing of the video slices through multi-layer indexing.

[0059] The above describes a method for video segmentation in the case of dialogue-type text. The following describes a method for video segmentation in the case of narration-type text or blank text. In an optional embodiment, the multiple video slices include multiple three-level video slices. Step S202 may also include: when voice recognition obtains narration-type text or blank text, segmenting the target video material into the multiple three-level video slices based on shot changes as segmentation points.

[0060] The target video material corresponding to voice-over text is primarily based on visual information, so it can be segmented based on camera changes. Since it's impossible to obtain the target video material's content information from blank text, such videos are segmented directly based on camera changes. For example, for videos shot using stream-of-consciousness filming techniques, it's difficult to obtain valid video content information from the text obtained through speech recognition, so they can be segmented directly based on camera changes. Stream-of-consciousness filming techniques are a filmmaking method that mimics the fluid nature of human thought. Through non-linear narratives, jumpy cuts, and subjective lenses, they break through traditional temporal and spatial logic, directly presenting a character's inner world, subconscious activities, or abstract concepts.

[0061] In this embodiment, when the target video material cannot be directly segmented according to voice recognition, the target video material can be directly segmented locally through lens changes, and local video segmentation can also be achieved efficiently.

[0062] Step S204 , generating video slice index information based on the multiple video slices, and the video slice index information is used for video slice indexing.

[0063] Video slice index information can be a collection of metadata for multiple video slices. Metadata describes data and provides additional information about the structure, content, source, and context of other data. Video slice index information includes key attributes of the video slice, such as the video slice identifier, content tag, time range, and frame description.

[0064] The following will provide a further exemplary introduction to generating video slice index information based on multiple video slices through more embodiments.

[0065] In order to generate index information more effectively and better, the video slice index information methods of conversation-type video slices and non-conversation-type video slices are different.

[0066] For conversational video slices, such as Figure 5 As shown, step S204 includes: Step S500: Generate slice description information for each conversation video slice or the three-level video slices obtained by segmenting the conversation video slice. The slice description information includes position information.

[0067] Step S502 , generating video understanding information for each dialogue video slice or the three-level video slices obtained by segmenting the dialogue video slice using the timestamp of the character's face.

[0068] Step S504: Generate corresponding video slice index information according to each dialogue video slice or the three-level video slice obtained by segmentation thereof, the slice description information, and the video understanding information; the corresponding video slice index information is stored through multiple fields of the index table.

[0069] Location information can be the physical location of the video slice, such as a file path and database identifier. Description information can also include a shot number (i.e., video slice identifier), start and end timestamps, etc. Video understanding information can be a visual description of the video slice, such as scene, action, emotion, dialogue, and screen content. The description information and video understanding information can be populated into predefined fields of an index table to generate video slice index information. In some embodiments, the index table can be a metadata wide table, a structured data table used to store metadata for multimedia content (such as video, audio, and images). In some embodiments, the index table can include a strip field, a role field, a comprehension field, and a dialogue field. The strip field can be used to record the shot number, start time (i.e., start timestamp), end time (i.e., end timestamp), and keyframes. The role field can be used to record the character or subject. The comprehension field can be used to record the action (e.g., swinging on a swing), time (e.g., daytime or nighttime), scene (e.g., imperial garden), emotion (e.g., seriousness, sadness), shot type (e.g., long shot, close-up, medium shot, and close-up), and screen content. The lines field can be used to record dialogue or narration. It should be noted that according to actual needs, blank fields can also be set in the index table, and the index table can also be customized according to the type of different video materials.

[0070] In this embodiment, video slice index information is generated according to the video understanding information and the slice description information, so that an indexing method that supports video understanding capability can be formed through the video slice index information.

[0071] For non-dialogue video slices, such as Figure 6 As shown, step S204 includes: Step S600 , extracting frames from each non-dialogue video slice or the third-level video slices obtained by segmenting the non-dialogue video slice to obtain a plurality of target frames.

[0072] Step S602: Perform image recognition on each target frame through a local recognition model to obtain corresponding video slice index information.

[0073] Since the video content of non-dialogue video slices or the third-level video slices obtained by segmenting them is mainly reflected in the video screen, the video content can be understood based on the video frames of such video slices. The local recognition model can be used to perform image recognition on each target frame to obtain an image description of each target frame. A non-dialogue video slice or the third-level video slices obtained by segmenting it can correspond to one or more target video frames. On this basis, the recognition results obtained by performing image recognition on each target frame can be classified to further determine the video content of each video slice. In some embodiments, the frame extraction method can be a fixed frame extraction, for example, extracting one frame per second while maintaining the original resolution. In some embodiments, the local recognition model can be a lightweight large language model pre-deployed locally, which can be used for resource-constrained end-side devices or scenarios.

[0074] In this embodiment, a small number of frames are identified and understood by a local recognition model by extracting frames, thereby generating corresponding video slice index information to solve the problem of video content that cannot be understood by voice recognition.

[0075] Similarly, regarding the third-level video slices corresponding to the narration text or blank text, in an optional embodiment, such as Figure 7 As shown, step S204 includes: Step S700: extract frames from each level 3 video slice to obtain multiple target frames.

[0076] Step S702: Perform image recognition on each target frame through a local recognition model to obtain corresponding video slice index information.

[0077] Regarding the indexing process, in an optional embodiment, the multiple video slices include first-level video slices, second-level video slices and third-level video slices at the lens level; the video slice index information includes video index information of first-level video slices, video index information of second-level video slices and video index information of third-level video slices.

[0078] like Figure 8 As shown, the method further includes: Step S800: Determine the shot description information of the shot to be indexed.

[0079] Step S802: According to the storyboard description information, a target first-level video slice is obtained by indexing from multiple first-level video slices of the target video material.

[0080] Step S804: According to the storyboard description information, a target secondary video slice is obtained by indexing from multiple secondary video slices of the target primary video slice.

[0081] Step S806: According to the shot description information, index a target tertiary video slice from multiple tertiary video slices of the target secondary video slice; wherein the target tertiary video slice is a video slice corresponding to the shot to be indexed.

[0082] The storyboard description information may be determined by the index conditions input by the user. It should be noted that the number of the target first-level video slices, target second-level video slices and target third-level video slices obtained through indexing may be one or more, that is, the storyboard description information may match multiple video slices. For example, if the description information includes "conference room", all first-level video slices marked as "conference room scene" may be indexed. If the target first-level video slice is "conference opening", the corresponding multiple second-level video slices may include "host speech" and "PPT opening animation", etc. If the storyboard description information also includes "PPT display clips", the third-level video slices containing "PPT screen" may be further determined.

[0083] In this embodiment, the video slices are hierarchically indexed using the storyboard description information to gradually narrow the index range, thereby improving the indexing efficiency and accuracy.

[0084] In order to make this application easier to understand, the following Figure 9 An exemplary application is provided.

[0085] 1. Taking dialogue video material as an example (i.e. Route 2, which can be used for semantic understanding), the operation process is as follows: S1. Obtain video material and perform speech recognition on the video to determine that the video is a narrative video (i.e., speech recognition obtains dialogue text). Semantic understanding can be performed to obtain the dialogue text of the video.

[0086] S2. Use the local LLM model to break down the dialogue text of the video material into outlines to obtain Event 1, Event 2...Event N (i.e., determine the content outline and divide the target video material into multiple first-level video slices based on the content outline).

[0087] Among them, outline-level indexing (ie, obtaining a target first-level video slice by indexing from multiple first-level video slices of the target video material).

[0088] S3. Use the local LLM model to break down the dialogue text into segments to obtain slice 1, slice 2, slice 3, etc. (i.e., based on sentence semantic understanding and sentence timestamps, multiple first-level video slices are segmented separately to obtain multiple second-level video slices).

[0089] Among them, the detailed level index (i.e., indexing the target secondary video slice from multiple secondary video slices of the target primary video slice) S4. Generate role 1, role 2, role 3, etc. based on the face calibration role and the timestamp corresponding to the role face (i.e., perform local object recognition on each dialogue video slice through the role face, and segment each dialogue video slice according to the recognition result to obtain multiple third video slices).

[0090] Among them, the strip-level index is split (ie, the target third-level video slice is obtained by indexing from multiple third-level video slices of the target second-level video slice).

[0091] 2. Taking narration video materials as an example (i.e. Route 3, which can be used for visual comprehension), the operation process is as follows: S1. Obtain video material and perform voice recognition on the video to determine whether the video is realistic (i.e., voice recognition obtains narration text or blank text), so that the picture can be understood.

[0092] S2. Through hard shot segmentation, we get shot a, shot b, shot c, etc. (i.e., we divide the target video material into three-level video slices based on shot changes).

[0093] S3. Extract frames of each storyboard obtained (i.e., obtain video frames of the storyboard for image understanding) to obtain a picture description of each storyboard.

[0094] S4. Group each storyboard at the outline level (i.e., cluster and group each storyboard according to the screen description of each storyboard to obtain multiple groups; one group can correspond to an event and a first-level slice video).

[0095] In this exemplary application, based on the understanding of the content of the target video material, a multi-layer segmentation strategy is adopted to locally segment the target video material.

[0096] Example 2 Figure 10 The block diagram of the video processing device according to the second embodiment of the present application is schematically shown. The device can be divided into one or more program modules, one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiment of the present application. The program module referred to in the embodiment of the present application refers to a series of computer program instruction segments that can complete specific functions. The following description will specifically introduce the functions of each program module in this embodiment. Figure 10 As shown, the apparatus 1000 may include: an acquisition module 1010, a segmentation module 1020, and a generation module 1030, wherein: An acquisition module 1010 is configured to acquire a target video material and perform speech recognition on the target video material; A segmentation module 1020 is configured to segment the video material to obtain a plurality of video slices according to the recognition result of the speech recognition; and The generating module 1030 is configured to generate video slice index information according to the multiple video slices, where the video slice index information is used for video slice indexing.

[0097] As an optional embodiment, the multiple video slices include multiple primary video slices and multiple secondary video slices; the slicing module 1020 is further configured to: In the case where a dialogue text is obtained through speech recognition, determining the content outline of the target video material based on the semantic understanding of the dialogue text; According to the content outline, the target video material is divided into a plurality of first-level video slices; According to the sentence semantic understanding and sentence timestamps in the dialogue text, the multiple first-level video slices are respectively segmented to obtain multiple second video slices; wherein each of the first-level video slices includes one or more second-level video slices.

[0098] As an optional embodiment, it is characterized in that the multiple second video slices include multiple conversation video slices and multiple non-conversation video slices; the multiple video slices also include multiple third-level video slices; the segmentation module 1020 is further used to: Local object recognition is performed on each dialogue video slice through the character's face, and each dialogue video slice is divided according to the recognition result to obtain multiple third video slices; wherein each dialogue video slice includes one or more third-level video slices, and the video slice index information of each third-level video slice can be configured with the character information in the slice.

[0099] As an optional embodiment, the generating module 1030 is further configured to: Generate slice description information for each conversation video slice or the three-level video slices obtained by segmenting it, wherein the slice description information includes position information; Generate video understanding information for each dialogue video slice or its three-level video slices using the timestamp of the character's face; Corresponding video slice index information is generated based on each dialogue video slice or the three-level video slice obtained by segmenting it, as well as the slice description information and the video understanding information; the corresponding video slice index information is stored through multiple fields of the index table.

[0100] As an optional embodiment, the generating module 1030 is further configured to: Extracting frames from each non-dialogue video slice or the third-level video slices obtained by segmenting the non-dialogue video slice to obtain multiple target frames; and Through the local recognition model, image recognition is performed on each target frame to obtain the corresponding video slice index information.

[0101] As an optional embodiment, the multiple video slices include multiple first-level video slices and multiple third-level video slices; the segmentation module 1020 is further configured to: In the case where the speech recognition obtains a dialogue text, determining the content outline of the target video material according to the dialogue text; According to the content outline, the target video material is divided into a plurality of first-level video slices; Taking the shot change as the segmentation point, each first-level video slice is divided into multiple third-level video slices; Each of the first-level video slices includes multiple third-level video slices.

[0102] As an optional embodiment, the multiple video slices include multiple three-level video slices; the segmentation module 1020 is further configured to: In the case where voice recognition obtains narration text or blank text, the target video material is divided into the plurality of three-level video slices based on shot changes.

[0103] As an optional embodiment, the generating module 1030 is further configured to: Extracting frames from each three-level video slice to obtain multiple target frames; and Through the local recognition model, image recognition is performed on each target frame to obtain the corresponding video slice index information.

[0104] As an optional embodiment, the multiple video slices include primary video slices, secondary video slices, and shot-level tertiary video slices; the video index information includes video index information of the primary video slices, video index information of the secondary video slices, and video index information of the tertiary video slices; the apparatus 1000 further includes an indexing module, the indexing module being configured to: Determine the storyboard description information of the shot to be indexed; According to the storyboard description information, a target first-level video slice is obtained by indexing from a plurality of first-level video slices of the target video material; According to the storyboard description information, a target secondary video slice is obtained by indexing from multiple secondary video slices of the target primary video slice; According to the shot description information, a target tertiary video slice is obtained by indexing from multiple tertiary video slices of the target secondary video slice; wherein the target tertiary video slice is a video slice corresponding to the shot to be indexed.

[0105] Example 3 Figure 11The following schematically shows a hardware architecture diagram of a computer device 10000 suitable for implementing a video processing method according to the third embodiment of the present application. In some embodiments, the computer device 10000 can be a terminal device such as a smart phone, a wearable device, a tablet computer, a personal computer, a vehicle-mounted terminal, a game console, a virtual device, a workbench, a digital assistant, a set-top box, a robot, etc. Figure 11 As shown, the computer device 10000 includes but is not limited to: a memory 10010, a processor 10020, and a network interface 10030 that can communicate with each other via a system bus. Memory 10010 includes at least one type of computer-readable storage medium, including flash memory, a hard disk, a multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, and the like. In some embodiments, memory 10010 may be an internal storage module of computer device 10000, such as a hard disk or memory of computer device 10000. In other embodiments, memory 10010 may also be an external storage device of computer device 10000, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like equipped on computer device 10000. Of course, memory 10010 may also include both internal storage modules and external storage devices of computer device 10000. In this embodiment, the memory 10010 is generally used to store an operating system and various application software installed on the computer device 10000, such as program codes of a video processing method, etc. In addition, the memory 10010 can also be used to temporarily store various data that has been output or is to be output.

[0106] In some embodiments, processor 10020 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other chips. Processor 10020 is typically used to control the overall operation of computer device 10000, such as performing control and processing related to data exchange or communication with computer device 10000. In this embodiment, processor 10020 is used to execute program code stored in memory 10010 or process data.

[0107] Network interface 10030 may include a wireless network interface or a wired network interface. Network interface 10030 is typically used to establish a communication link between computer device 10000 and other computer devices. For example, network interface 10030 is used to connect computer device 10000 to an external terminal via a network, establishing a data transmission channel and a communication link between computer device 10000 and the external terminal. The network may be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communications (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0108] It should be pointed out that Figure 11 Only a computer device having components 10010 - 10030 is shown, but it should be understood that implementing all of the shown components is not a requirement, and more or fewer components may alternatively be implemented.

[0109] In this embodiment, the video processing method stored in the memory 10010 can also be divided into one or more program modules and executed by one or more processors (such as processor 10020) to complete the embodiment of the present application.

[0110] Example 4 An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the video processing method in the embodiment are implemented.

[0111] In this embodiment, computer-readable storage media include flash memory, hard disks, multimedia cards, card-type memories (e.g., SD or DX memories), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disks, optical disks, and the like. In some embodiments, the computer-readable storage medium may be an internal storage unit of a computer device, such as the computer device's hard disk or memory. In other embodiments, the computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, and the like. Of course, the computer-readable storage medium may also include both the internal storage unit and external storage devices of the computer device. In this embodiment, the computer-readable storage medium is typically used to store the operating system and various application software installed on the computer device, such as the program code of the video processing method described in the embodiment. In addition, the computer-readable storage medium may also be used to temporarily store various types of data that has been output or is about to be output.

[0112] Example 5 An embodiment of the present application further provides a computer program product, including a computer program, which implements the method in the above embodiment when executed by a processor.

[0113] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present application can be implemented using general-purpose computer devices. They can be concentrated on a single computer device or distributed on a network composed of multiple computer devices. Alternatively, they can be implemented using program codes executable by the computer device, so that they can be stored in a storage device and executed by the computer device. In some cases, the steps shown or described can be performed in a different order than herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0114] It should be noted that the above are only preferred embodiments of the present application and do not limit the scope of patent protection of the present application. Any equivalent structure or equivalent process transformation made using the contents of the description and drawings of this application, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of the present application.

Claims

1. A video processing method, characterized in that: For a client, the method includes: Acquire target video material and perform speech recognition on the target video material; According to the recognition result of the speech recognition, the target video material is segmented to obtain a plurality of video slices; and Video slice index information is generated according to the multiple video slices, and the video slice index information is used for video slice indexing.

2. The method according to claim 1, characterized in that The multiple video slices include multiple primary video slices and multiple secondary video slices; Segmenting the target video material to obtain a plurality of video slices according to a recognition result obtained by the speech recognition, including: In the case where a dialogue text is obtained through speech recognition, determining the content outline of the target video material based on the semantic understanding of the dialogue text; According to the content outline, the target video material is divided into a plurality of first-level video slices; According to the sentence semantic understanding and sentence timestamps in the dialogue text, the multiple first-level video slices are respectively segmented to obtain multiple second video slices; wherein each of the first-level video slices includes one or more second-level video slices.

3. The method according to claim 2, characterized in that The plurality of second video slices include a plurality of conversation video slices and a plurality of non-conversation video slices; the plurality of video slices also include a plurality of tertiary video slices; the method further includes: Local object recognition is performed on each dialogue video slice through the character's face, and each dialogue video slice is divided according to the recognition result to obtain multiple third video slices; wherein each dialogue video slice includes one or more third-level video slices, and the video slice index information of each third-level video slice can be configured with the character information in the slice.

4. The method according to claim 3, characterized in that Generating video slice index information according to the multiple video slices includes: Generate slice description information for each conversation video slice or the three-level video slices obtained by segmenting it, wherein the slice description information includes position information; Generate video understanding information for each dialogue video slice or its three-level video slices using the timestamp of the character's face; Corresponding video slice index information is generated based on each dialogue video slice or the three-level video slice obtained by segmenting it, as well as the slice description information and the video understanding information; the corresponding video slice index information is stored through multiple fields of the index table.

5. The method according to claim 3, wherein Generating video slice index information according to the multiple video slices includes: Extracting frames from each non-dialogue video slice or the third-level video slices obtained by segmenting the non-dialogue video slice to obtain multiple target frames; and Through the local recognition model, image recognition is performed on each target frame to obtain the corresponding video slice index information.

6. The method according to claim 1, characterized in that The multiple video slices include multiple first-level video slices and multiple third-level video slices; Segmenting the target video material to obtain a plurality of video slices according to a recognition result obtained by the speech recognition, including: In the case where the speech recognition obtains a dialogue text, determining the content outline of the target video material according to the dialogue text; According to the content outline, the target video material is divided into a plurality of first-level video slices; Taking the shot change as the segmentation point, each first-level video slice is divided into multiple third-level video slices; Each of the first-level video slices includes multiple third-level video slices.

7. The method according to claim 1, characterized in that The plurality of video slices includes a plurality of tertiary video slices; Segmenting the target video material to obtain a plurality of video slices according to a recognition result obtained by the speech recognition, including: In the case where voice recognition obtains narration text or blank text, the target video material is divided into the plurality of three-level video slices based on shot changes.

8. The method according to claim 7, wherein Generating video slice index information according to the multiple video slices includes: Extracting frames from each three-level video slice to obtain multiple target frames; and Through the local recognition model, image recognition is performed on each target frame to obtain the corresponding video slice index information.

9. The method according to claim 1, characterized in that The multiple video slices include a first-level video slice, a second-level video slice, and a third-level video slice at a shot level; the video slice index information includes video index information of the first-level video slice, video index information of the second-level video slice, and video index information of the third-level video slice; the method further includes: Determine the storyboard description information of the shot to be indexed; According to the storyboard description information, a target first-level video slice is obtained by indexing from a plurality of first-level video slices of the target video material; According to the storyboard description information, a target secondary video slice is obtained by indexing from multiple secondary video slices of the target primary video slice; According to the shot description information, a target tertiary video slice is obtained by indexing from multiple tertiary video slices of the target secondary video slice; wherein the target tertiary video slice is a video slice corresponding to the shot to be indexed.

10. A video processing device, characterized in that: For a client, the device includes: An acquisition module, configured to acquire target video material and perform speech recognition on the target video material; a segmentation module, configured to segment the video material to obtain a plurality of video slices according to the recognition result of the speech recognition; and A generating module is used to generate video slice index information according to the multiple video slices, and the video slice index information is used for video slice indexing.

11. A computer device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; wherein: The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to claims 1 to 9 are implemented.

Images