Video processing method and related device

Abstract

The invention provides a video processing method and a related device. The embodiment of the invention can be applied to the technical field of computer vision. According to the video processing method provided by the embodiment of the invention, all the segments in the target audio and video generate corresponding semantic information, picture contents in the corresponding segments are represented through the semantic information, voice information extraction is performed on the segments containing the target object in the target audio and video, and the voice information of each segment is obtained. And splicing the semantic information corresponding to the fragment containing the target object with the voice information to obtain audio and video information corresponding to the fragment containing the target object, and taking the semantic information corresponding to the fragment not containing the target object as the audio and video information of the fragment. And finally, splicing the audio and video information corresponding to all the segments according to the time sequence tags carried by the segments to obtain the audio and video information of the target audio and video, and taking the audio and video information of the target audio and video as the target audio and video for dictating the copywriting of the image, thereby improving the accuracy and authenticity of the copywriting of the dictated image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video processing method and related devices. Background Art

[0002] Audio Description is a service that converts visual information that the visually impaired cannot receive into verbal symbols through oral or written narration, so that the visually impaired can overcome visual impairments and help them overcome various visual barriers in their daily lives, studies and employment environments. Simply put, audio description is to "say" what you "see".

[0003] At present, the text of oral description is usually obtained by manual text editing based on the text information provided by the script of the film or TV series. However, the video platform usually does not obtain the script of the film or TV series, which increases the difficulty of manual editing of the text of oral description. In addition, the text of oral description that relies on manual editing will contain the editor's understanding of the image and personal emotions, resulting in poor accuracy of the text of manually edited oral description. The visually impaired cannot accurately understand the real content of the image by listening to the text of manually edited oral description. Summary of the invention

[0004] The embodiments of the present application provide a video processing method and related devices, which solve the problem in the prior art that the accuracy and authenticity of oral descriptions are poor due to manual editing of the text of oral descriptions.

[0005] One aspect of the present application provides a video processing method, comprising:

[0006] Get the target audio and video;

[0007] The target audio and video are segmented to generate M segments, where each segment in the M segments carries a time sequence label, M ≥ 1;

[0008] Analyze the picture of each of the M segments to generate M segment semantic information corresponding to the M segments, wherein the segment semantic information is used to represent the picture content in the corresponding segment;

[0009] Performing target object recognition on the picture of each of the M segments, and extracting voice information on each of the K segments containing the target object, to obtain voice information of each of the K segments, wherein the voice information is generated according to a voice audio signal matching the target object, 1≤K≤M;

[0010] Splicing the semantic information and the speech information corresponding to each of the K segments to generate the audio-visual information corresponding to each of the K segments;

[0011] The semantic information corresponding to each segment in the MK segments that do not contain the target object is used as the audio-visual information corresponding to each segment in the MK segments, wherein the MK segments that do not contain the target object refer to the remaining segments from the M segments minus the K segments that contain the target object;

[0012] According to the timing label carried by each of the M segments, the audio and video information corresponding to each of the K segments and the audio and video information corresponding to each of the MK segments are spliced ​​to generate the audio and video information of the target audio and video.

[0013] Another aspect of the present application provides a video processing device, including: a target audio and video acquisition module, an audio and video segmentation module, a segment semantic information generation module, a voice information generation module, an audio and video information first generation module, an audio and video information second generation module and a target audio and video information generation module; specifically:

[0014] A target audio and video acquisition module is used to acquire target audio and video;

[0015] The audio and video segmentation module is used to segment the target audio and video to generate M segments, where each segment in the M segments carries a timing label, M≥1;

[0016] A segment semantic information generation module, used for analyzing the picture of each segment of the M segments, and generating M segment semantic information corresponding to the M segments, wherein the segment semantic information is used for representing the picture content in the corresponding segment;

[0017] A voice information generation module is used to identify the target object in each of the M segments, extract voice information from each of the K segments containing the target object, and obtain voice information of each of the K segments, wherein the voice information is generated according to the voice audio signal matching the target object, 1≤K≤M;

[0018] A first audio-visual information generation module, used for splicing semantic information and speech information corresponding to each of the K segments to generate audio-visual information corresponding to each of the K segments;

[0019] The second audio-visual information generation module is used to use the semantic information corresponding to each segment in the MK segments that do not contain the target object as the audio-visual information corresponding to each segment in the MK segments, wherein the MK segments that do not contain the target object refer to the remaining segments from the M segments minus the K segments that contain the target object;

[0020] The target audio and video information generation module is used to splice the audio and video information corresponding to each of the K segments and the audio and video information corresponding to each of the MK segments according to the timing labels carried by each of the M segments, so as to generate the audio and video information of the target audio and video.

[0021] In another implementation of the embodiment of the present application, the voice information generation module is further used to:

[0022] Perform target object recognition on the picture of each of the M segments to determine K segments containing the target object;

[0023] Acquire a speech audio signal of each of the K segments, and acquire subtitle information of each of the K segments;

[0024] Determine subtitle information matching the target object from the subtitle information of each of the K segments;

[0025] According to the subtitle information matching the target object in each of the K segments, determining the speech audio signal matching the target object from the speech audio signal of the segment;

[0026] According to the speech audio signal matching the target object in each of the K segments, speech information matching the target object is generated.

[0027] In another implementation of the embodiment of the present application, the voice information generation module is further used to:

[0028] According to the subtitle information of each of the K segments, each of the K segments is segmented to obtain at least one first video frame corresponding to each of the K segments, wherein one first video frame includes one subtitle information;

[0029] Subtitle information matching the target object is determined from a first video frame corresponding to each of the K segments.

[0030] In another implementation of the embodiment of the present application, the voice information generation module is further used to:

[0031] According to the subtitle information matching the target object in each of the K segments, abnormal speech and audio signal detection is performed on the speech and audio signal matching the target object to eliminate the speech and audio signal that does not correspond to the subtitle information in the speech and audio signal matching the target object.

[0032] In another implementation of the embodiment of the present application, the voice information generation module is further used to:

[0033] Obtain subtitle information for each of the M segments;

[0034] According to the subtitle information of each of the M segments, each of the M segments is segmented to obtain at least one second video frame corresponding to each of the M segments;

[0035] Target object recognition is performed on the second video frame corresponding to each of the M segments to obtain K segments containing the target object.

[0036] In another implementation of the embodiment of the present application, the voice information generation module is further used to:

[0037] Performing face detection on the second video frame corresponding to each of the M segments to obtain a third video frame corresponding to each of the M segments and containing a face detection frame;

[0038] Performing face recognition on the third video frame containing the face detection frame corresponding to each of the M segments, to obtain a face recognition result for each of the M segments;

[0039] According to the face recognition result of each of the M segments, K segments with the same face recognition result are determined from the M segments.

[0040] In another implementation of the embodiment of the present application, the voice information generation module is further used to:

[0041] Obtaining a mapping table of faces and roles of the target audio and video, wherein the mapping table of faces and roles is used to represent the corresponding relationship between role names and face information;

[0042] Obtaining a speech audio signal corresponding to each segment of the M segments;

[0043] According to the mapping table between human faces and roles, a corresponding relationship between role names and voice audio signals is established.

[0044] In another implementation of the embodiment of the present application, the voice information generation module is further used to:

[0045] Get the audio of each of the K segments;

[0046] Perform frame-by-frame windowing processing on the audio of each of the K segments to obtain K time domain signals corresponding to the K segments;

[0047] Through fast Fourier transform, K time domain signals are converted into K frequency domain signals;

[0048] The K frequency domain signals are processed by the Mel filter to obtain K energy information corresponding to the K frequency domain signals;

[0049] Taking the logarithm of K energy information, obtaining K logarithmic energies corresponding to the K energy information;

[0050] Performing discrete cosine transform on the logarithmic energy sequence composed of K logarithmic energies to obtain a frequency coefficient signal, wherein the logarithmic energy sequence is spliced ​​according to the K logarithmic energies;

[0051] The frequency coefficient signal is denoised to generate a speech audio signal.

[0052] In another implementation of the embodiment of the present application, the audio and video segmentation module is further used to:

[0053] The target audio and video are used as the input of the video shot boundary detection model, and the target audio and video are subjected to feature extraction through the video feature extraction network in the video shot boundary detection model to obtain the target audio and video feature sequence;

[0054] The target audio and video feature sequence is segmented through the temporal segmentation network in the video shot boundary detection model to obtain M segments.

[0055] In another implementation of the embodiment of the present application, the audio and video segmentation module is further used to:

[0056] Performing action detection on the target object in each of the M segments to obtain action information of the target object in each of the M segments, wherein the action information includes a start timestamp when the target object starts to perform the action and an end timestamp when the target object ends the action;

[0057] According to the action information of the target object in each of the M segments, the M segments are classified to obtain M labels corresponding to the M segments, wherein the labels are used to represent the action information of the target object in the segment.

[0058] In another implementation of the embodiment of the present application, the segment semantic information generation module is further used to:

[0059] Taking the M clips as the input of the video semantic model, encoding the M clips through the dual visual encoding network in the video semantic model to generate M encoding information corresponding to the M clips;

[0060] The M encoding information is text-encoded through the text encoder network in the video semantic model to generate M segment semantic information corresponding to the M segments.

[0061] Another aspect of the present application provides a computer device, comprising:

[0062] memories, transceivers, processors, and bus systems;

[0063] Wherein, the memory is used to store programs;

[0064] The processor is used to execute the program in the memory, including executing the above-mentioned methods;

[0065] The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

[0066] Another aspect of the present application provides a computer-readable storage medium, in which instructions are stored. When the computer-readable storage medium is run on a computer, the computer is enabled to execute the above-mentioned methods.

[0067] Another aspect of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided by the above aspects.

[0068] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0069] The present application provides a video processing method and a related device, which includes: obtaining target audio and video; dividing the target audio and video to generate M segments carrying timing tags; analyzing the picture of each segment in the M segments to generate M segment semantic information corresponding to the M segments, wherein the segment semantic information is used to characterize the picture content in the corresponding segment; identifying the target object on the picture of each segment in the M segments, extracting voice information from each segment in the K segments containing the target object, and obtaining voice information of each segment in the K segments, wherein the voice information is generated based on a voice audio signal matching the target object; splicing the semantic information corresponding to each segment in the K segments with the voice information to generate audio and video information corresponding to each segment in the K segments; using the semantic information corresponding to each segment in MK segments that do not contain the target object as the audio and video information corresponding to each segment in the MK segments; splicing the audio and video information corresponding to each segment in the K segments and the audio and video information corresponding to each segment in the MK segments according to the timing tags carried by each segment in the M segments to generate audio and video information of the target audio and video. The video processing method provided by the embodiment of the present application generates corresponding semantic information for all the clips in the target audio and video, represents the picture content in the corresponding clip by the semantic information, extracts voice information from the clip containing the target object in the target audio and video to obtain the voice information of each clip, splices the semantic information and voice information corresponding to the clip containing the target object to obtain the audio and video information corresponding to the clip containing the target object, uses the semantic information corresponding to the clip not containing the target object as the audio and video information of the clip, and finally splices the audio and video information corresponding to all the clips according to the timing tags carried by the clips to obtain the audio and video information of the target audio and video, and uses the semantic information corresponding to the picture content in the target audio and video and the voice information of the target object to form audio and video information as the text of the target audio and video for oral description, which improves the accuracy and authenticity of the text of the oral description compared with the text of the oral description manually edited. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 A schematic diagram of the architecture of a video processing system provided in one embodiment of the present application;

[0071] Figure 2 A flowchart of a video processing method provided in one embodiment of the present application;

[0072] Figure 3 A flowchart of a video processing method provided by another embodiment of the present application;

[0073] Figure 4 A flowchart of a video processing method provided by another embodiment of the present application;

[0074] Figure 5A flowchart of a video processing method provided by another embodiment of the present application;

[0075] Figure 6 A flowchart of a video processing method provided by another embodiment of the present application;

[0076] Figure 7 A flowchart of a video processing method provided by another embodiment of the present application;

[0077] Figure 8 A flowchart of a video processing method provided by another embodiment of the present application;

[0078] Fig. 9 A flowchart of a video processing method provided by another embodiment of the present application;

[0079] Fig.10 A flowchart of a video processing method provided by another embodiment of the present application;

[0080] Fig.11 A flowchart of a video processing method provided by another embodiment of the present application;

[0081] Fig.12 A flowchart of a video processing method provided by another embodiment of the present application;

[0082] Fig.13 A schematic diagram of a video processing method flow provided in an embodiment of the present application;

[0083] Fig.14 A schematic diagram of the structure of a video shot segmentation module provided in one embodiment of the present application;

[0084] Fig.15 A schematic diagram of a training of a fragment summary description module provided for a certain embodiment of the present application;

[0085] Fig.16 A schematic diagram of the structure of a video processing device provided in one embodiment of the present application;

[0086] Fig.17 A schematic diagram of a server structure provided for a certain embodiment of the present application. DETAILED DESCRIPTION

[0087] The embodiment of the present application provides a video processing method, which generates corresponding semantic information for all clips in a target audio and video, represents the picture content in the corresponding clip by the semantic information, extracts voice information from the clip containing the target object in the target audio and video to obtain the voice information of each clip, splices the semantic information corresponding to the clip containing the target object with the voice information to obtain the audio and video information corresponding to the clip containing the target object, uses the semantic information corresponding to the clip not containing the target object as the audio and video information of the clip, and finally splices the audio and video information corresponding to all clips according to the timing tags carried by the clips to obtain the audio and video information of the target audio and video, uses the audio and video information of the target audio and video as the target audio and video for the text of the oral description, thereby improving the accuracy and authenticity of the text of the oral description.

[0088] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "corresponding to" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0089] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0090] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0091] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing so that computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0092] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0093] Audio Description is a service that converts visual information that the visually impaired cannot receive into verbal symbols through oral or written narration, so that the visually impaired can overcome visual barriers and help them overcome various visual barriers in their lives, studies and employment environments. Simply put, audio description is to "say" what you "see". Taking movies as an example, audio description is a technology that uses language to explain and describe the visual elements of the film, such as spatial settings, scenes, character expressions, and actions, without interfering with the sound information and dialogue of the normal program.

[0094] However, the current oral description is usually written manually, and the automated oral description solution proposed in the related art relies on the script of the film and television drama to provide text information. The film and television script is owned by the producer of the film and television production. As one of the core copyrights, it is usually not provided to the video platform. The platform needs to obtain the script through negotiation and purchase, which is costly. In addition, the accuracy of the oral description that relies on manual editing is poor. The visually impaired cannot accurately understand the real content of the image by listening to the manual editing of the oral description. The manual editing of the oral description will carry the editor's personal understanding and emotional color, resulting in the poor authenticity of the manual editing of the oral description.

[0095] The video processing method provided by the embodiment of the present application generates corresponding semantic information for all the clips in the target audio and video, represents the picture content in the corresponding clip by the semantic information, extracts voice information from the clip containing the target object in the target audio and video to obtain the voice information of each clip, splices the semantic information corresponding to the clip containing the target object with the voice information to obtain the audio and video information corresponding to the clip containing the target object, uses the semantic information corresponding to the clip not containing the target object as the audio and video information of the clip, and finally splices the audio and video information corresponding to all the clips according to the timing tags carried by the clips to obtain the audio and video information of the target audio and video, and uses the audio and video information of the target audio and video as the target audio and video for the text of the oral description, thereby improving the accuracy and authenticity of the text of the oral description.

[0096] For easier understanding, see Figure 1 , Figure 1 FIG. 1 is an application environment diagram of the video processing method in the embodiment of the present application, such as Figure 1 As shown, the video processing method in the embodiment of the present application is applied to a video processing system. The video processing system includes: a server and a terminal device; wherein the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and the embodiments of the present application are not limited here.

[0097] The server first obtains the target audio and video; secondly, the server divides the target audio and video into M segments, wherein each of the M segments carries a time sequence tag, M ≥ 1; thirdly, the server analyzes the image of each of the M segments to generate M segment semantic information corresponding to the M segments, wherein the segment semantic information is used to characterize the image content in the corresponding segment; then, the server performs target object recognition on the image of each of the M segments, and extracts voice information from each of the K segments containing the target object, and obtains voice information of each of the K segments, wherein the voice information is It is generated according to the speech audio signal matching the target object, 1≤K≤M; then, the server splices the semantic information and speech information corresponding to each of the K segments to generate the audio-visual information corresponding to each of the K segments; and, the server uses the semantic information corresponding to each of the MK segments that do not contain the target object as the audio-visual information corresponding to each of the MK segments; finally, the server splices the audio-visual information corresponding to each of the K segments and the audio-visual information corresponding to each of the MK segments according to the timing tags carried by each of the M segments to generate the audio-visual information of the target audio and video.

[0098] The video processing method provided by the embodiment of the present application generates corresponding semantic information for all the clips in the target audio and video, represents the picture content in the corresponding clip by the semantic information, extracts voice information from the clip containing the target object in the target audio and video to obtain the voice information of each clip, splices the semantic information corresponding to the clip containing the target object with the voice information to obtain the audio and video information corresponding to the clip containing the target object, uses the semantic information corresponding to the clip not containing the target object as the audio and video information of the clip, and finally splices the audio and video information corresponding to all the clips according to the timing tags carried by the clips to obtain the audio and video information of the target audio and video, and uses the audio and video information of the target audio and video as the target audio and video for the text of the oral description, thereby improving the accuracy and authenticity of the text of the oral description.

[0099] The following is an introduction to the video processing method in this application from the perspective of the server. Figure 2 The video processing method provided in the embodiment of the present application includes: step S110 to step S170. Specifically:

[0100] S110, obtaining target audio and video.

[0101] It can be understood that the audio and video are composed of audio and video, and the target audio and video can be a long video (such as a film and television drama, variety show video, etc.) or a short video.

[0102] S120: Segment the target audio and video to generate M segments.

[0103] Each of the M segments carries a temporal label, M ≥ 1.

[0104] It is understandable that the target audio and video can be segmented by using the video shot segmentation method, or the complete target audio and video can be segmented by using a video timing segmentation algorithm (such as the TransNetV2 model), or the complete target audio and video can be segmented at the video frame of the shot conversion to generate M segments. Each of the M segments obtained by segmentation carries a timing label, and the M segments can be restored to the complete target audio and video through the timing label. The timing label is used to characterize the timing information of each segment in the target audio and video.

[0105] For ease of understanding, the embodiment of the present application uses the TransNetV2 model to segment the complete target audio and video. TransNetV2 is an algorithm model for video temporal segmentation, which can divide a video sequence into multiple action segments and classify and locate each segment. The input of the TransNetV2 model can be a video of any length. In the embodiment of the present application, the input of the TransNetV2 model is the target audio and video, and the processing process of the TransNetV2 model specifically includes the following steps:

[0106] 1. Feature extraction: Take the target audio and video as the input of the TransNetV2 model, and extract the features of the target audio and video through the video feature extraction network in the TransNetV2 model to obtain the target audio and video feature sequence. The video feature extraction network can be a convolutional neural network such as ResNet.

[0107] 2. Time segmentation: The target audio and video feature sequence is segmented through the time segmentation network in the TransNetV2 model to obtain M segments, each of which contains a series of continuous feature vectors. The action of the target object is detected in each of the M segments to obtain the action information of the target object in each of the M segments, that is, each segment represents an action in the target audio and video.

[0108] 3. Fragment classification: Through the classification module in the TransNetV2 model, the action information of the target object in each of the M fragments is classified to obtain M labels corresponding to the M fragments, where the labels are used to represent the action information of the target object in the fragment.

[0109] 4. Fragment positioning: Each of the M fragments is positioned through the positioning module in the TransNetV2 model, and the start timestamp of the target object’s action and the end timestamp of the target object’s action are output.

[0110] 5. Result output: The M labels corresponding to the M segments generated by the segment classification step, as well as the start timestamp of the target object's action and the end timestamp of the target object's action in each segment generated by the segment positioning step, are used as the video timing segmentation results.

[0111] In summary, the TransNetV2 model achieves video temporal segmentation by dividing the target audio and video into multiple action segments and classifying and locating each segment.

[0112] S130 , analyzing the picture of each of the M segments to generate M segment semantic information corresponding to the M segments.

[0113] The segment semantic information is used to represent the picture content in the corresponding segment.

[0114] It can be understood that, by performing picture analysis on each of the M segments through a video semantic model (e.g., an mPLUG-2 model), semantic information of the picture of each segment is obtained, and the semantic information of the picture of each segment is summarized to obtain segment semantic information corresponding to each segment. The segment semantic information is used to characterize the picture content in the corresponding segment, such as character information, scene information, environment information, etc.

[0115] For ease of understanding, the embodiment of the present application analyzes the picture content of each fragment through the mPLUG-2 model to obtain a summary description of the semantic information of the picture of each fragment. The mPLUG-2 model is a large model that unifies image, video, and text modalities. The model introduces a multi-module combination network. Different modules can be flexibly selected for different understanding and generation tasks of all modes such as text, image, and video. The input of the mPLUG-2 model is M fragments, and the specific implementation process is:

[0116] 1. Take the M clips as the input of the mPLUG-2 model, encode the M clips through the dual visual encoding network in the mPLUG-2 model, and generate M encoding information corresponding to the M clips.

[0117] 2. The M encoded information is text-encoded through the text encoder network in the mPLUG-2 model to generate M fragment semantic information corresponding to the M fragments.

[0118] The mPLUG-2 model consists of a dual encoder module for images and videos, a text encoder module, and a general layer module. The mPLUG-2 model uses two unimodal encoders to encode images / videos and texts respectively to represent the inherent information of a single modality. mPLUG-2 designs a unified dual visual encoder module, where the video input shares a standard Transformer module with the image input to model spatial information, and an additional local temporal modeling module is used to model temporal relationships for video-related tasks. A new general layer module is then introduced to project the visual and language modalities into a common language-guided semantic space through a shared self-attention module. In addition, an additional cross-attention module is used to fuse the general visual representation with the original fine-grained visual representation. Finally, the unimodal and cross-modal representations can be used as input to a shared Transformer decoder for various generation tasks.

[0119] S140 , performing target object recognition on the picture of each of the M segments, and extracting voice information on each of the K segments containing the target object, to obtain voice information of each of the K segments.

[0120] The voice information is generated based on the voice audio signal matching the target object, 1≤K≤M.

[0121] It is understandable that the target object recognition module is used to perform target object recognition on the screen of each of the M segments. The target object can be a person, an animal, etc. If the target audio and video is a film or TV series, the target object is the person in the screen of the film or TV series. If the target audio and video is an animation or cartoon, the target object is an animal or a person in the animation. The target object recognition is performed on the screen of each of the M segments, and K segments containing the target object are determined from the M segments. Voice information is extracted from each of the determined K segments to obtain voice information of each of the K segments. The voice information is generated based on the voice audio signal matching the target object, for example, it can be the text or lines contained in the audio information of the target object's speech.

[0122] S150, concatenating the semantic information and the speech information corresponding to each of the K segments to generate audio-visual information corresponding to each of the K segments.

[0123] It can be understood that for the K segments containing the target object, the semantic information corresponding to each segment is spliced ​​with the voice information to generate the audio-visual information corresponding to each segment, and the audio-visual information can reflect the picture content of the segment (such as character information, scene information, environmental information, etc.) and the lines of the target object in the segment. For example, the picture content of a certain segment is that the target object is sitting in the stands of the stadium, and the voice of the target object is "Next year you will be a college student, this summer you will go to xxxx to participate in their training camp this year, I heard it is very interesting", and the semantic information generated according to the picture of the segment is: the target object is sitting in the stands of the stadium, and the voice information generated according to the lines of the target object in the segment is: Next year you will be a college student, this summer you will go to xxxx to participate in their training camp this year, I heard it is very interesting, and the semantic information of the segment is spliced ​​with the voice information, and the audio-visual information of the segment is generated: the target object is sitting in the stands of the stadium, and she said "Next year you will be a college student, this summer you will go to xxxx to participate in their training camp this year, I heard it is very interesting".

[0124] For ease of understanding, the embodiment of the present application uses the DeepVoice model to splice the semantic information and voice information corresponding to the segment to generate the audio-visual information corresponding to the segment. The core of the DeepVoice algorithm model is a sequence-to-sequence model based on a recurrent neural network (RNN). It takes the input text sequence as input and generates the corresponding speech waveform sequence through a neural network model. The speech generated by DeepVoice is natural, fluent, and close to human-like timbre.

[0125] The training process of the DeepVoice model includes two stages: training the model from text to acoustic features and training the model from acoustic features to speech waveforms. In the first stage, the input text sequence is converted into an intermediate high-level semantic representation through the encoder model. Then, the decoder model converts these semantic representations into acoustic feature sequences, such as phonemes, tones, etc. In the second stage, the acoustic feature sequence is converted into the final speech waveform sequence through another model, the acoustic model. The core of the DeepVoice algorithm is a sequence-to-sequence model based on recurrent neural networks (RNN). It takes the input text sequence as input and generates the corresponding speech waveform sequence through the neural network model.

[0126] The DeepVoice model uses a progressive training approach, which is to first train a coarse model from a large-scale annotated dataset, then use the generated synthetic speech data to enhance the dataset, and use this data to continue training a more refined model. This approach helps improve the quality and naturalness of speech synthesis. Another notable feature is the use of the Attention Mechanism, which is used to weight the semantic representation of the input in the decoder to better capture relevant contextual information. This helps improve the fluency and coherence of speech synthesis.

[0127] S160. Use the semantic information corresponding to each of the MK segments that do not contain the target object as the audio-visual information corresponding to each of the MK segments.

[0128] Among them, the MK segments that do not contain the target object refer to the remaining segments after subtracting the K segments that contain the target object from the M segments.

[0129] It can be understood that for the K segments that do not contain the target object, the semantic information corresponding to each segment is used as the audio-visual information corresponding to the segment, and the audio-visual information only reflects the picture content of the segment (such as character information, scene information, environmental information, etc.).

[0130] S170. According to the timing label carried by each of the M segments, the audio and video information corresponding to each of the K segments and the audio and video information corresponding to each of the MK segments are spliced ​​to generate the audio and video information of the target audio and video.

[0131] It can be understood that, according to the timing tag carried by each of the M segments, the audio and video information corresponding to each of the M segments is spliced ​​to generate complete audio and video information corresponding to the target audio and video.

[0132] The video processing method provided by the embodiment of the present application generates corresponding semantic information for all the clips in the target audio and video, represents the picture content in the corresponding clip by the semantic information, extracts voice information from the clip containing the target object in the target audio and video to obtain the voice information of each clip, splices the semantic information corresponding to the clip containing the target object with the voice information to obtain the audio and video information corresponding to the clip containing the target object, uses the semantic information corresponding to the clip not containing the target object as the audio and video information of the clip, and finally splices the audio and video information corresponding to all the clips according to the timing tags carried by the clips to obtain the audio and video information of the target audio and video, and uses the audio and video information of the target audio and video as the target audio and video for the text of the oral description, thereby improving the accuracy and authenticity of the text of the oral description.

[0133] In this application Figure 2In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Figure 3 , step S140 further includes sub-steps S141 to S145. Specifically:

[0134] S141 . Perform target object recognition on the picture of each of the M segments to determine K segments containing the target object.

[0135] It can be understood that the target object recognition module is used to perform target object recognition on the screen of each of the M segments. The target object can be a person, an animal, etc. If the target audio and video is a film or TV series, the target object is the person in the screen of the film or TV series. If the target audio and video is an animation or cartoon, the target object is an animal or a person in the animation. The target object recognition is performed on the screen of each of the M segments, and K segments containing the target object are determined from the M segments.

[0136] For ease of understanding, taking the target object as a person as an example, the target object recognition module provided in the embodiment of the present application includes an MTCNN model (Multi-task Cascaded Convolutional Networks) and ArcFace. Specifically, the MTCNN model is first used to detect the face area appearing in the current picture, and then the ArcFace model is used to perform person recognition on each face area.

[0137] The MTCNN model (Multi-task Cascaded Convolutional Networks) is a deep learning algorithm for face detection and facial key point detection. The MTCNN model uses a cascade approach and divides face detection and facial key point detection into three stages: Proposal Network (P-Net): used to generate candidate face frames; RefineNetwork (R-Net): used to screen candidate face frames and remove unqualified frames; Output Network (O-Net): used to detect facial key points, and finally predict the position and confidence of the face frame through regression model and classification model.

[0138] The ArcFace model is a deep learning algorithm for face recognition. The core of the ArcFace algorithm is a new loss function that adds a cosine similarity constraint to the traditional Softmax classifier, making faces of the same category closer in the feature space and the distance between faces of different categories more obvious. Based on the trained ArcFace model, the facial feature vector of each face region can be extracted.

[0139] First, obtain a training image containing facial information (also called a reference image). Then, extract the facial features of the reference image based on the MTCNN model and the ArcFace model. Then, summarize the extracted facial features into a face library. The purpose of the face library is to establish a database containing multiple facial feature vectors for comparison and recognition in subsequent face recognition tasks. Specifically, the face library contains multiple facial images of known identities, and each facial image corresponds to a feature vector. When performing face recognition, it is necessary to extract the feature vector of the face image to be identified and compare it with the feature vector in the face library to find the most similar face. If the similarity with the face exceeds a preset threshold, its identity information will be output as the information of the face to be predicted.

[0140] S142: Acquire a speech audio signal of each of the K segments, and acquire subtitle information of each of the K segments.

[0141] It can be understood that the speech audio signal of each segment in the K segments and the subtitle information of each segment are obtained respectively. Wherein, the speech audio signal of each segment in the K segments can be obtained by processing according to the speech feature extraction module, including:

[0142] 1. Audio acquisition: Get the audio of each of the K segments.

[0143] 2. Preprocessing: Perform frame and window processing on the audio of each of the K segments to obtain K time domain signals corresponding to the K segments. Specifically: divide the audio signal into frames, each frame is 20-30ms long, and perform window processing. Commonly used window functions include Hanning window, Hamming window, etc.

[0144] 3. Fourier transform: Through fast Fourier transform, K time domain signals are converted into K frequency domain signals. Specifically: Fast Fourier transform (FFT) is performed on each frame of audio signal to convert the time domain signal into a frequency domain signal.

[0145] 4. Mel filter bank: Process K frequency domain signals through Mel filters to obtain K energy information corresponding to the K frequency domain signals. Specifically: Pass the frequency domain signals through a set of Mel filters to obtain the energy value of each frequency band. The center frequency of the Mel filter bank is set according to the Mel scale, and generally 20-40 filters are set.

[0146] 5. Logarithmic energy: Take the logarithm of K energy information to obtain K logarithmic energies corresponding to the K energy information. Specifically: Take the logarithm of the energy value of each frequency band to obtain the logarithmic energy.

[0147] 6. DCT transformation: Perform discrete cosine transformation on the logarithmic energy sequence composed of K logarithmic energies to obtain a frequency coefficient signal, wherein the logarithmic energy sequence is spliced ​​based on the K logarithmic energies. Specifically: Perform discrete cosine transformation (DCT) on the logarithmic energy sequence to obtain a set of frequency coefficients, namely MFCC coefficients.

[0148] 7. Remove DC components, truncate high frequencies, and normalize: De-noise the frequency coefficient signal to generate a speech audio signal. Specifically: First, remove the first item in the MFCC coefficient because it represents the average energy of the signal and does not have the function of distinguishing features; then, truncate the high-frequency part of the MFCC coefficient because the energy of the high-frequency part is small and has little effect on distinguishing signal features. Finally, normalize the MFCC coefficients to make them have the same scale range to generate a speech audio signal.

[0149] S143 . Determine subtitle information matching the target object from the subtitle information of each of the K segments.

[0150] It can be understood that the subtitle information that matches the target object refers to the lines spoken by the target object. Since each segment may include multiple subtitles, and the subtitles may correspond to the lines spoken by different target objects, it is necessary to determine the subtitle information that matches the target object from the subtitle information of each segment.

[0151] For ease of understanding, the subtitle information that matches the target object is determined from the subtitle information of each of the K segments through the line and title mapping module. Based on the speech feature extraction module in step S142, the MFCC features of the speech corresponding to each segment can be obtained. Based on the target object recognition module in step S141, the character recognition information corresponding to each segment can be obtained. The MFCC features of the speech corresponding to each segment are integrated with the corresponding character recognition information, and the MFCC features and voice list of each character can be obtained respectively. Usually, the characters appearing in the video screen correspond to the characters who speak the lines, and the voice list of each character (role) can be roughly obtained through step S143.

[0152] S144. Determine a speech audio signal matching the target object from the speech audio signals of the segment according to the subtitle information matching the target object in each of the K segments.

[0153] It can be understood that, according to the subtitle information matching the target object, the speech audio signal matching the target object is determined from the speech audio signal of the segment, that is, the speech audio signal corresponding to the target object is obtained.

[0154] S145. Generate speech information matching the target object according to the speech audio signal matching the target object in each of the K segments.

[0155] It is understandable that the voice information matching the target object is generated based on the voice audio signal corresponding to the target object. The voice information may be the text corresponding to the complete lines spoken by the target object or the summary information of the text corresponding to the lines spoken by the target object.

[0156] The video processing method provided in the embodiment of the present application identifies the target object for each segment, and then screens out the segments containing the target object, determines the subtitle information matching the target object and the speech and audio signal corresponding to the subtitle information for each segment containing the target object, and then generates speech information matching the target object, thereby improving the accuracy of the speech information matching of the target object, and thereby improving the accuracy and authenticity of the generated audio-visual information containing the speech information.

[0157] In this application Figure 3 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Figure 4 , step S143 further includes sub-steps S1431 to S1432. Specifically:

[0158] S1431. Segment each of the K segments according to subtitle information of each of the K segments to obtain at least one first video frame corresponding to each of the K segments.

[0159] Wherein, a first video frame includes a subtitle information.

[0160] It is understandable that since a segment often contains multiple subtitles, in order to ensure the accuracy of the subtitle information matching the target object, the segment needs to be segmented according to the video shot segmentation module so that each first video frame after segmentation includes a subtitle information of the segment.

[0161] The input of the video shot segmentation module is K segments. Since each segment is usually several seconds long and contains multiple lines of video dialogue, in order to match the lines with the characters, the video shot segmentation module further segments the segments by sentence to obtain at least one first video frame corresponding to each segment.

[0162] For a complete target audio and video, it usually contains two parts of information. One part is the video image and audio, and the other part is the subtitle information in the video. The commonly used subtitle file format is srt. The subtitle srt file contains the start and end time of each sentence in the video and the video text content. The video shot segmentation module splits the clips based on the start and end time of each sentence. Each frame corresponds to a line in the video and contains three pieces of information: video image, line, and voice.

[0163] S1432: Determine subtitle information matching the target object from the first video frame corresponding to each of the K segments.

[0164] It can be understood that, for each of the segmented video frames, the target object obtained by the target object recognition module is matched with the subtitle information in each video frame to determine the subtitle information matching the target object in each segment.

[0165] The video processing method provided in the embodiment of the present application divides the fragments to obtain the video frames corresponding to each subtitle, and then determines the subtitle information matching the target object according to the subtitle information in the video frame, thereby improving the accuracy of matching the subtitle information for the target object.

[0166] In this application Figure 3 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Figure 5 , sub-step S144 also includes sub-step S146. Specifically:

[0167] S146. According to the subtitle information matching the target object in each of the K segments, abnormal speech and audio signal detection is performed on the speech and audio signal matching the target object to eliminate the speech and audio signal that does not correspond to the subtitle information in the speech and audio signal matching the target object.

[0168] It is understandable that in most cases, the characters appearing in the video screen and the characters speaking the lines correspond, and the above method can roughly obtain the voice list of each character (role), but due to the fact that there are a small number of speakers whose screen and voice are not synchronized, in order to improve the data quality of the character's voice, the embodiment of the present application filters such voice data through anomaly detection algorithm based on MFCC features. Anomaly detection refers to finding values ​​or outliers that are different from other values ​​in a data set.

[0169] The embodiment of the present application performs anomaly detection through the LOF algorithm. The basic idea of ​​the algorithm is to compare each sample point with the sample points around it, and to determine whether the sample point is an outlier by calculating the distance relationship between its local density and its neighbors. The specific steps are as follows:

[0170] 1. For each sample point, calculate the distance between it and its k nearest neighbor points, and arrange these distance values ​​in ascending order, and take the kth distance value as the reachability distance of the point.

[0171] 2. For each sample point, calculate the local density of its neighborhood (i.e., the set of sample points whose distance to the point does not exceed a given radius r), that is, the number of points in the neighborhood. At the same time, for each point in the neighborhood, calculate the maximum value of the reachable distance to the point as the local reachability density of the point.

[0172] 3. For each sample point, calculate its LOF value (local outlier factor), which is the average of the ratio of the local reachability density of the point to the local reachability density of its neighbors. The larger the LOF value, the more likely the point is an outlier.

[0173] Based on the results of the above anomaly detection algorithm, some non-corresponding voice features can be screened out, and some voice features that cannot be matched with the role can also be selected. The voice features retained after anomaly detection are collected to form a feature library. For the above non-corresponding and non-matching voice features, the nearest neighbor search is performed in the feature library to calculate the similarity of the Top-1 vector. If it is higher than a given threshold, it is considered that the voice feature and the Top-1 voice feature are voices emitted by the same role.

[0174] The video processing method provided in the embodiment of the present application detects abnormal speech and audio signals to eliminate speech and audio signals that do not correspond to subtitle information in speech and audio signals that match the target object, thereby improving the accuracy of determining the speech and audio signals that match the target object.

[0175] In this application Figure 3 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Figure 6 , step S141 further includes sub-steps S1411 to S1413. Specifically:

[0176] S1411. Obtain subtitle information of each of the M segments.

[0177] S1412. Segment each of the M segments according to the subtitle information of each of the M segments to obtain at least one second video frame corresponding to each of the M segments.

[0178] It is understandable that since a segment often contains multiple subtitles, in order to ensure the accuracy of the subtitle information matching the target object, the segment needs to be segmented according to the video shot segmentation module so that each second video frame after segmentation includes a subtitle information of the segment.

[0179] S1413 . Perform target object recognition on the second video frame corresponding to each of the M segments to obtain K segments containing the target object.

[0180] It is understandable that the target object is identified for the second video frame corresponding to each segment, and the second video frames containing the same target object are spliced ​​to obtain the corresponding segment. For example, an original segment is 4 seconds long, and contains a subtitle information every half second. The 4-second segment is segmented by the subtitle information to obtain 8 second video frames; the target object is identified for the 8 second video frames, and it is determined that only the 2nd, 3rd, 4th, 5th, and 6th second video frames contain the target object. A new segment containing the target object is generated based on the 2nd, 3rd, 4th, 5th, and 6th second video frames. The new segment is shorter than the original segment, and the video frames that do not contain the target object are eliminated.

[0181] For another example, an original clip is 3 seconds long and contains a subtitle information every half second. The 3-second clip is divided according to the subtitle information to obtain 6 second video frames. The target object is identified on the 6 second video frames, and it is determined that none of the 6 second video frames contain the target object, then the original clip is directly deleted.

[0182] The video processing method provided in the embodiment of the present application divides each fragment into frames and identifies the target object on the obtained video frames to determine the fragments that only contain the target object, thereby improving the accuracy of target object identification on the fragments, and reducing the time for subsequent processing of the fragments by eliminating invalid video frames that do not contain the target object in the fragments, thereby improving the processing efficiency of the fragments.

[0183] In this application Figure 6 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Figure 7 Sub-step S1413 further includes sub-steps S14131 to S14133. It should be noted that the present application embodiment is described by taking the target object as a person as an example. Specifically:

[0184] S14131. Perform face detection on the second video frame corresponding to each of the M segments to obtain a third video frame corresponding to each of the M segments and containing a face detection frame.

[0185] It is understandable that face detection is first performed on the second video frame corresponding to each segment to determine the third video frame containing the face detection frame corresponding to each segment. The second video frame can be detected by the MTCNN model. The MTCNN model (Multi-task Cascaded Convolutional Networks) is a deep learning algorithm for face detection and facial key point detection. The MTCNN model uses a cascaded approach to divide face detection and facial key point detection into three stages: Proposal Network (P-Net): used to generate candidate face frames; Refine Network (R-Net): used to screen candidate face frames and remove unqualified frames; Output Network (O-Net): used to detect facial key points, and finally the regression model and classification model are used to predict the position and confidence of the face frame.

[0186] S14132. Perform face recognition on the third video frame containing the face detection frame corresponding to each of the M segments, and obtain a face recognition result for each of the M segments.

[0187] It is understandable that the ArcFace model is used to perform character recognition on each face region to obtain the face recognition result of each segment. The ArcFace model is a deep learning algorithm for face recognition. The core of the ArcFace algorithm is a new loss function. On the basis of the traditional Softmax classifier, this loss function adds a cosine similarity constraint, which makes the faces of the same category closer in the feature space and the distance between faces of different categories more obvious. Based on the trained ArcFace model, the face feature vector of each face region can be extracted. When performing face recognition, it is necessary to extract the feature vector of the face image to be identified and compare it with the feature vector in the face library to find the most similar face. If the similarity with the face exceeds a preset threshold, its identity information will be output as the information of the face to be predicted.

[0188] S14133. According to the face recognition result of each of the M segments, determine K segments with the same face recognition results from the M segments.

[0189] It can be understood that K segments belonging to the same person (with the same face recognition result) are determined from the M segments.

[0190] The video processing method provided in the embodiment of the present application determines K segments belonging to the same person (with the same face recognition results) from M segments through face detection and face recognition, thereby improving the accuracy of determining the segments belonging to the same person.

[0191] In this application Figure 7 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Figure 8 , after sub-step S14132, sub-steps S14134 to S14136 are also included. Specifically:

[0192] S14134. Obtain a mapping table of faces and roles in the target audio and video.

[0193] Among them, the face and role mapping table is used to represent the corresponding relationship between the role name and the face information.

[0194] It is understandable that when the target audio or video is a film or TV series, a mapping table of faces and characters can be obtained, that is, a mapping table that corresponds facial information to the names of characters in the series.

[0195] S14135. Obtain the speech audio signal corresponding to each of the M segments.

[0196] It can be understood that the speech audio signal corresponding to each segment in the M segments is obtained.

[0197] S14136. According to the mapping table between human faces and roles, a corresponding relationship between the role name and the voice audio signal is established.

[0198] It can be understood that the correspondence between the character name and the voice audio signal is established by corresponding the face information and the corresponding voice audio signal (i.e., the voice spoken by the face information) with the character name corresponding to the face information. For example, face information A plays the role of role a in the play. The voice audio signal of face information A has been determined through the above embodiment, and the correspondence between role a and the voice audio signal is established.

[0199] The video processing method provided in the embodiment of the present application establishes a corresponding relationship between the role name and the voice audio signal, thereby facilitating the determination of the role name corresponding to the line information in the subsequent generation of audio-visual information, thereby improving the accuracy and authenticity of the audio-visual information.

[0200] In this application Figure 3 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Fig. 9 , sub-step S142 further includes sub-steps S1421 to S1427. Specifically:

[0201] S1421. Obtain the audio of each of the K segments.

[0202] S1422. Perform frame-by-frame windowing processing on the audio of each of the K segments to obtain K time domain signals corresponding to the K segments.

[0203] It can be understood that the audio signal is divided into frames, each frame length is 20-30ms, and windowing is performed. Commonly used window functions include Hanning window, Hamming window, etc.

[0204] S1423. Convert the K time domain signals into K frequency domain signals through fast Fourier transform.

[0205] It can be understood that a fast Fourier transform (FFT) is performed on each frame of the audio signal to convert the time domain signal into a frequency domain signal.

[0206] S1424. Process the K frequency domain signals through a Mel filter to obtain K energy information corresponding to the K frequency domain signals.

[0207] It can be understood that the frequency domain signal is passed through a set of Mel filters to obtain the energy value of each frequency band. The center frequency of the Mel filter bank is set according to the Mel scale, and generally 20-40 filters are set.

[0208] S1425. Take the logarithm of the K pieces of energy information to obtain K logarithmic energies corresponding to the K pieces of energy information.

[0209] It can be understood that the logarithm of the energy value of each frequency band is taken to obtain the logarithmic energy.

[0210] S1426. Perform discrete cosine transform on the logarithmic energy sequence composed of K logarithmic energies to obtain a frequency coefficient signal.

[0211] The logarithmic energy sequence is formed by concatenating K logarithmic energies.

[0212] It can be understood that a set of frequency coefficients, namely, MFCC coefficients, is obtained by performing discrete cosine transform (DCT) on the logarithmic energy sequence.

[0213] S1427. Perform denoising on the frequency coefficient signal to generate a speech audio signal.

[0214] It can be understood that, first, the first item in the MFCC coefficient is removed because it represents the average energy of the signal and has no function of distinguishing features; then, the high-frequency part of the MFCC coefficient is truncated because the energy of the high-frequency part is small and has little effect on distinguishing signal features; finally, the MFCC coefficient is normalized to have the same scale range to generate a speech audio signal.

[0215] The video processing method provided in the embodiment of the present application generates a speech audio signal by extracting speech features from the audio of the clip, thereby improving the accuracy of the speech audio signal generation process.

[0216] In this application Figure 2In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Fig.10 , step S120 further includes sub-steps S121 to S122. Specifically:

[0217] S121. Using the target audio and video as the input of the video shot boundary detection model, extracting features of the target audio and video through the video feature extraction network in the video shot boundary detection model to obtain a target audio and video feature sequence.

[0218] S122. Segment the target audio and video feature sequence through the temporal segmentation network in the video shot boundary detection model to obtain M segments.

[0219] It is understandable that the target audio and video can be segmented by using the video shot segmentation method, or the complete target audio and video can be segmented by using a video timing segmentation algorithm (such as the TransNetV2 model), or the complete target audio and video can be segmented at the video frame of the shot conversion to generate M segments. Each of the M segments obtained by segmentation carries a timing label, and the M segments can be restored to the complete target audio and video through the timing label. The timing label is used to characterize the timing information of each segment in the target audio and video.

[0220] For ease of understanding, the embodiment of the present application uses the TransNetV2 model to segment the complete target audio and video. TransNetV2 is an algorithm model for video temporal segmentation, which can divide a video sequence into multiple action segments and classify and locate each segment. The input of the TransNetV2 model can be a video of any length, and specifically includes the following steps:

[0221] 1. Feature extraction: Take the target audio and video as the input of the TransNetV2 model, and extract the features of the target audio and video through the video feature extraction network in the TransNetV2 model to obtain the target audio and video feature sequence. The video feature extraction network can be a convolutional neural network such as ResNet.

[0222] 2. Time segmentation: The target audio and video feature sequence is segmented through the time segmentation network in the TransNetV2 model to obtain M segments, each of which contains a series of continuous feature vectors. The action of the target object is detected in each of the M segments to obtain the action information of the target object in each of the M segments, that is, each segment represents an action in the target audio and video.

[0223] The video processing method provided in the embodiment of the present application segments the target audio and video through a video shot boundary detection model, thereby improving the accuracy of segmenting the target audio and video.

[0224] In this application Fig.10 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Fig.11 , sub-step S122 also includes sub-steps S123 to S124. Specifically:

[0225] S123 . Perform action detection on the target object in each of the M segments to obtain action information of the target object in each of the M segments.

[0226] The action information includes a start timestamp when the target object starts to perform the action and an end timestamp when the target object ends the action.

[0227] S124. Classify the M segments according to the action information of the target object in each of the M segments to obtain M labels corresponding to the M segments.

[0228] Among them, the label is used to represent the action information of the target object in the clip.

[0229] It is understandable that the embodiment of the present application realizes the segmentation of the complete target audio and video through the TransNetV2 model. TransNetV2 is an algorithm model for video time segmentation, which can divide the video sequence into multiple action segments and classify and locate each segment. The input of the TransNetV2 model can be a video of any length, and specifically includes the following steps:

[0230] 1. Feature extraction: Take the target audio and video as the input of the TransNetV2 model, and extract the features of the target audio and video through the video feature extraction network in the TransNetV2 model to obtain the target audio and video feature sequence. The video feature extraction network can be a convolutional neural network such as ResNet.

[0231] 2. Time segmentation: The target audio and video feature sequence is segmented through the time segmentation network in the TransNetV2 model to obtain M segments, each of which contains a series of continuous feature vectors. The action of the target object is detected in each of the M segments to obtain the action information of the target object in each of the M segments, that is, each segment represents an action in the target audio and video.

[0232] 3. Fragment classification: Through the classification module in the TransNetV2 model, the action information of the target object in each of the M fragments is classified to obtain M labels corresponding to the M fragments, where the labels are used to represent the action information of the target object in the fragment.

[0233] 4. Fragment positioning: Each of the M fragments is positioned through the positioning module in the TransNetV2 model, and the start timestamp of the target object’s action and the end timestamp of the target object’s action are output.

[0234] 5. Result output: The M labels corresponding to the M segments generated by the segment classification step, as well as the start timestamp of the target object's action and the end timestamp of the target object's action in each segment generated by the segment positioning step, are used as the video timing segmentation results.

[0235] The TransNetV2 algorithm achieves video temporal segmentation by dividing the target audio and video into multiple action segments and classifying and locating each segment.

[0236] The video processing method provided in the embodiment of the present application segments the target audio and video through a video shot boundary detection model, thereby improving the accuracy of segmenting the target audio and video.

[0237] In this application Figure 2 In an optional embodiment of the video processing method provided in the corresponding embodiment, please refer to Fig.12 , step S130 also includes sub-steps S131 to S132. Specifically:

[0238] S131. Use the M segments as input of a video semantic model, encode the M segments through a dual visual coding network in the video semantic model, and generate M coding information corresponding to the M segments.

[0239] S132. Perform text encoding on the M encoding information through the text encoder network in the video semantic model to generate M fragment semantic information corresponding to the M fragments.

[0240] It can be understood that, by performing picture analysis on each of the M segments through a video semantic model (e.g., an mPLUG-2 model), semantic information of the picture of each segment is obtained, and the semantic information of the picture of each segment is summarized to obtain segment semantic information corresponding to each segment. The segment semantic information is used to characterize the picture content in the corresponding segment, such as character information, scene information, environment information, etc.

[0241] For ease of understanding, the embodiment of the present application analyzes the picture content of each fragment through the mPLUG-2 model to obtain a summary description of the semantic information of the picture of each fragment. The mPLUG-2 model is a large model that unifies image, video, and text modalities. The model introduces a multi-module combination network. Different modules can be flexibly selected for different understanding and generation tasks of all modes such as text, image, and video. The input of the mPLUG-2 model is M fragments, and the specific implementation process is:

[0242] 1. Take the M clips as the input of the mPLUG-2 model, encode the M clips through the dual visual encoding network in the mPLUG-2 model, and generate M encoding information corresponding to the M clips.

[0243] 2. The M encoded information is text-encoded through the text encoder network in the mPLUG-2 model to generate M fragment semantic information corresponding to the M fragments.

[0244] The mPLUG-2 model consists of a dual encoder module for images and videos, a text encoder module, and a general layer module. The mPLUG-2 model uses two unimodal encoders to encode images / videos and texts respectively to represent the inherent information of a single modality. mPLUG-2 designs a unified dual visual encoder module, where the video input shares a standard Transformer module with the image input to model spatial information, and an additional local temporal modeling module is used to model temporal relationships for video-related tasks. A new general layer module is then introduced to project the visual and language modalities into a common language-guided semantic space through a shared self-attention module. In addition, an additional cross-attention module is used to fuse the general visual representation with the original fine-grained visual representation. Finally, the unimodal and cross-modal representations can be used as input to a shared Transformer decoder for various generation tasks.

[0245] The video processing method provided in the embodiment of the present application generates semantic information of each segment through a video semantic model, thereby improving the accuracy and authenticity of voice information.

[0246] For ease of understanding, the following will combine Fig.13 A video processing method is introduced. First, the complete target audio and video is segmented into M segments based on a video shot segmentation module; then a summary description of each segment is generated based on a segment summary description module; each segment is further segmented by sentence, and each sentence corresponds to a line; for each video frame of the above segmentation, the characters appearing in the video screen are identified based on a video character recognition module, and the character name corresponding to each character is captured through an encyclopedia, the MFCC features of the segment are extracted based on a voice feature extraction module, and the characters and voices are matched; in order to improve the matching accuracy, the voice features of each character are detected for anomalies based on the lines and drama names mapping module, and the matching relationship is cleaned based on the nearest neighbor retrieval; finally, the audio and video information of the target audio and video is generated based on the audio and video information result generation module.

[0247] First, the video shot segmentation module is introduced:

[0248] For a film or TV series, the first step is to divide the complete long video into multiple video clips according to its shots and storylines. This application embodiment uses the TransNetV2 algorithm model to divide the video clips. Fig.14 , Fig.14 It is a structural diagram of the video shot segmentation module provided in the embodiment of the present application. TransNetV2 is an algorithm for video temporal segmentation, which can divide a video sequence into multiple action segments and classify and locate each segment. The input of the TransNetV2 algorithm can be a video of any length. In the embodiment of the present application, the input of the TransNetV2 algorithm is the target audio and video, and the dimension of the target audio and video is N×48×27×3, where N represents the duration of the target audio and video. The overall structure of the TransNetV2 algorithm network is as follows: Fig.14As shown in the figure, it includes 6 deep discrete convolutional neural networks (DDCNN modules). After the input is forwarded to obtain features, they are concatenated with RGB histogram features and learnable similarity features, and then some fully connected operations are performed to obtain the probability of each frame being a shot boundary frame. The features of the target audio and video after passing through the first deep discrete convolutional neural network (DDCNN V2 cell, 64 filters) and the second deep discrete convolutional neural network (DDCNN V2 cell, 64 filters) are concatenated to obtain a concatenated vector of dimension N×48×27×64. The concatenated vector of dimension N×48×27×64 is processed by an average pooling layer (Avg pooling, 1×2×2) of dimension 1×2×2 to obtain a vector of dimension N×48×13×64. The features obtained by processing through the third deep discrete convolutional neural network (DDCNN V2 cell, 128 filters) composed of 128 filters and the features obtained by processing through the fourth deep discrete convolutional neural network (DDCNN V2 cell, 128 filters) composed of 128 filters and the second deep discrete convolutional neural network (DDCNN V2 cell, 128 filters) composed of 64 filters are concatenated. The features obtained after processing by the 64 filters (V2 cell, 64 filters) are spliced ​​to obtain features of dimension N×24×13×128, and the features of dimension N×24×13×128 are processed again by the average pooling layer (Avg pooling, 1×2×2) of dimension 1×2×2 to obtain a vector of dimension N×12×6×128. The vector of dimension N×12×6×128 is processed by the sixth deep discrete convolutional neural network (DDCNN V2 cell, 256 filters) composed of 256 filters. The features obtained are processed by a learnable similarity network, an average pooling layer (Avg pooling, 1×2×2) of dimension 1×2×2, and RGB histogram features. After compression and splicing, the probability of whether each frame is a shot boundary frame is obtained. The specific steps include:

[0249] 1. Feature extraction: Take the target audio and video as the input of the TransNetV2 model, and extract the features of the target audio and video through the video feature extraction network in the TransNetV2 model to obtain the target audio and video feature sequence. The video feature extraction network can be a convolutional neural network such as ResNet.

[0250] 2. Time segmentation: The target audio and video feature sequence is segmented through the time segmentation network in the TransNetV2 model to obtain M segments, each of which contains a series of continuous feature vectors. The action of the target object is detected in each of the M segments to obtain the action information of the target object in each of the M segments, that is, each segment represents an action in the target audio and video.

[0251] 3. Fragment classification: Through the classification module in the TransNetV2 model, the action information of the target object in each of the M fragments is classified to obtain M labels corresponding to the M fragments, where the labels are used to represent the action information of the target object in the fragment.

[0252] 4. Fragment positioning: Each of the M fragments is positioned through the positioning module in the TransNetV2 model, and the start timestamp of the target object’s action and the end timestamp of the target object’s action are output.

[0253] 5. Result output: The M labels corresponding to the M segments generated by the segment classification step, as well as the start timestamp of the target object's action and the end timestamp of the target object's action in each segment generated by the segment positioning step, are used as the video timing segmentation results.

[0254] In summary, the TransNetV2 algorithm achieves video temporal segmentation by dividing the target audio and video into multiple action segments and classifying and locating each segment.

[0255] Secondly, the fragment summary description module is introduced:

[0256] Based on the video shot segmentation results of the video shot segmentation module, the role of the segment summary description module is to provide a summary description of the semantic information appearing in each segment. The mPLUG-2 algorithm model is a large model that unifies image, video, and text modalities. The model introduces a multi-module combination network. Different modules can be flexibly selected for different understanding and generation tasks of all modes such as text, images, and videos. The input of the mPLUG-2 model is M segments, and the specific implementation process is:

[0257] 1. Take the M clips as the input of the mPLUG-2 model, encode the M clips through the dual visual encoding network in the mPLUG-2 model, and generate M encoding information corresponding to the M clips.

[0258] 2. The M encoded information is text-encoded through the text encoder network in the mPLUG-2 model to generate M fragment semantic information corresponding to the M fragments.

[0259] The mPLUG-2 model consists of a dual encoder module for images and videos, a text encoder module, and a general layer module. The mPLUG-2 model uses two unimodal encoders to encode images / videos and texts, respectively, to represent the inherent information of a single modality. mPLUG-2 designs a unified dual visual encoder module, where the video input shares a standard Transformer module with the image input to model spatial information, and an additional local temporal modeling module is used to model temporal relationships for video-related tasks. A new general layer module is then introduced to project the visual and language modalities into a common language-guided semantic space through a shared self-attention module. In addition, an additional cross-attention module is used to fuse the general visual representation with the original fine-grained visual representation. Finally, the unimodal and cross-modal representations can be used as input to a shared Transformer decoder for various generation tasks. This facilitates multi-task pre-training and transfer learning.

[0260] like Fig.15 As shown, Fig.15 The diagram is a diagram of training the segment summary description module. The dataset used to train the segment summary description module is the MSR-VTT2016 dataset, which uses 20 categories and contains 257 common topics to search for related videos. It contains 10,000 videos with a total length of about 41.2 hours, and each video corresponds to 20 reference sentences. Through training, the segment summary description module can provide a summary description of the semantic information in the video segment.

[0261] Next, the module for segmenting fragments into sentences is introduced:

[0262] The result of the video shot segmentation module is a number of video clips, each of which is usually a few seconds long and contains multiple lines of video dialogue. In order to match the lines with the characters, the sentence segmentation module further segments the video clips by sentence to obtain at least one first video frame corresponding to each clip. For a complete long video source medium, it usually contains two parts of information, one part of which is the video picture and audio, and the other part is the subtitle information in the video. The commonly used subtitle file format is srt. The subtitle srt file contains the start and end time of each sentence in the video and the video text content. The sentence segmentation module splits the video clips into strips based on the start and end time of each sentence. Each first video frame corresponds to a line in the video and contains three pieces of information: video picture, lines, and voice.

[0263] Next, the character recognition module is introduced:

[0264] For each of the above-split video frames, the character recognition module is used to identify the characters appearing in the current video frame. Specifically, the MTCNN algorithm model is first used to detect the face area appearing in the current picture, and then the ArcFace algorithm model is used to identify the celebrity characters in each face area.

[0265] MTCNN (Multi-task Cascaded Convolutional Networks) is a deep learning algorithm for face detection and facial key point detection. The MTCNN algorithm uses a cascade approach and divides face detection and facial key point detection into three stages: Proposal Network (P-Net): used to generate candidate face frames; Refine Network (R-Net): used to screen candidate face frames and remove unqualified frames; Output Network (O-Net): used to detect facial key points, and finally predict the position and confidence of the face frame through regression model and classification model.

[0266] The ArcFace algorithm is a deep learning algorithm for face recognition. The core of the ArcFace algorithm is a new loss function that adds a cosine similarity constraint to the traditional Softmax classifier, making faces of the same category closer in the feature space and the distance between faces of different categories more obvious. This can improve the accuracy and robustness of face recognition.

[0267] Based on the trained ArcFace model, the facial feature vector of each facial region can be extracted. We collect a batch of people and their corresponding pictures (called reference pictures) in advance, and extract the facial features of the reference pictures based on MTCNN and ArcFace, and summarize them into the face library. The purpose of the face library is to establish a database containing multiple facial feature vectors for comparison and recognition in subsequent face recognition tasks. Specifically, the face library contains multiple facial images of known identities, and each facial image corresponds to a feature vector. When performing face recognition, it is necessary to extract the feature vector of the face image to be identified and compare it with the feature vector in the face library to find the most similar face. If the similarity with the face exceeds a preset threshold, its identity information will be output as the information of the face to be predicted.

[0268] Based on the above, the character information appearing in the video frame can be identified, and then the corresponding relationship between the characters in the play and the characters in the encyclopedia can be captured according to the play title request capture interface, so as to establish a mapping relationship between the characters and the character names.

[0269] Then, the speech feature extraction module is introduced:

[0270] For each of the above-divided video segments, the MFCC features of the speech of each segment are extracted. Through these steps, the audio signal can be converted into a set of MFCC coefficients, which can be used to analyze the characteristics of the audio signal. The specific steps include:

[0271] 1. Audio acquisition: Get the audio of each of the K segments.

[0272] 2. Preprocessing: Perform frame and window processing on the audio of each of the K segments to obtain K time domain signals corresponding to the K segments. Specifically: divide the audio signal into frames, each frame is 20-30ms long, and perform window processing. Commonly used window functions include Hanning window, Hamming window, etc.

[0273] 3. Fourier transform: Through fast Fourier transform, K time domain signals are converted into K frequency domain signals. Specifically: Fast Fourier transform (FFT) is performed on each frame of audio signal to convert the time domain signal into a frequency domain signal.

[0274] 4. Mel filter bank: Process K frequency domain signals through Mel filters to obtain K energy information corresponding to the K frequency domain signals. Specifically: Pass the frequency domain signals through a set of Mel filters to obtain the energy value of each frequency band. The center frequency of the Mel filter bank is set according to the Mel scale, and generally 20-40 filters are set.

[0275] 5. Logarithmic energy: Take the logarithm of K energy information to obtain K logarithmic energies corresponding to the K energy information. Specifically: Take the logarithm of the energy value of each frequency band to obtain the logarithmic energy.

[0276] 6. DCT transformation: Perform discrete cosine transformation on the logarithmic energy sequence composed of K logarithmic energies to obtain a frequency coefficient signal, wherein the logarithmic energy sequence is spliced ​​based on the K logarithmic energies. Specifically: Perform discrete cosine transformation (DCT) on the logarithmic energy sequence to obtain a set of frequency coefficients, namely MFCC coefficients.

[0277] 7. Remove DC components, truncate high frequencies, and normalize: De-noise the frequency coefficient signal to generate a speech audio signal. Specifically: First, remove the first item in the MFCC coefficient because it represents the average energy of the signal and does not have the function of distinguishing features; then, truncate the high-frequency part of the MFCC coefficient because the energy of the high-frequency part is small and has little effect on distinguishing signal features. Finally, normalize the MFCC coefficients to make them have the same scale range to generate a speech audio signal.

[0278] Then, the role voice mapping module is introduced:

[0279] Based on the speech feature extraction module, we can obtain the MFCC features of the speech corresponding to each clip. Based on the results of the character recognition module, we can obtain the character information corresponding to each clip. By integrating these two results, we can get the MFCC features and speech list of each character respectively.

[0280] In most cases, the characters appearing in the video and the characters speaking the lines correspond to each other. The above method can roughly obtain the voice list of each character. However, due to the fact that the screen and voice of a small number of speakers are not synchronized, in order to improve the data quality of the character voice, the character voice mapping module filters this voice data through anomaly detection algorithm based on MFCC features. Anomaly detection refers to finding values ​​or outliers that are different from other values ​​in the data set.

[0281] The embodiment of the present application uses the LOF algorithm for anomaly detection. The basic idea of ​​the algorithm is to compare each sample point with the sample points around it, and to determine whether the sample point is an outlier by calculating the distance relationship between its local density and its neighbors. The specific steps are as follows:

[0282] 1. For each sample point, calculate the distance between it and its k nearest neighbor points, and arrange these distance values ​​in ascending order, taking the kth distance value as the reachability distance of the point.

[0283] 2. For each sample point, calculate the local density of its neighborhood (i.e., the set of sample points that are no more than a given radius r from the point), that is, the number of points in the neighborhood. At the same time, for each point in the neighborhood, calculate the maximum reachable distance from the point to the point, as the local reachability density of the point.

[0284] 3. For each sample point, calculate its LOF value (local outlier factor), which is the average of the ratio of the local reachability density of the point to the local reachability density of its neighbors. The larger the LOF value, the more likely the point is an outlier.

[0285] Based on the results of the above anomaly detection algorithm, some non-corresponding voice features can be screened out, and some voice features that cannot be matched with the role can also be selected. The voice features retained after anomaly detection are collected to form a feature library. For the above non-corresponding and non-matching voice features, the nearest neighbor search is performed in the feature library to calculate the similarity of the Top-1 vector. If it is higher than a given threshold, it is considered that the voice feature and the Top-1 voice feature are voices emitted by the same role.

[0286] Finally, the audio and video information result generation module is introduced:

[0287] The audio-visual information result generation module combines the outputs of the above modules to generate the final "listening to the video" result, that is, the audio-visual information of the target audio and video. For each video clip, a summary description of the clip is first generated based on the clip summary description module. After that, for each line of dialogue, the character corresponding to the line can be obtained based on the line title mapping module, and the character name information is inserted before the audio of the corresponding line is played. When inserting, the character name is not inserted for every line of dialogue. For multiple consecutive lines spoken by the same character, the character information is only inserted before the first line of dialogue. For a small number of lines that the title mapping module cannot map to the character's lines, no additional information is inserted, and the audio is played directly.

[0288] For auxiliary information such as video summary descriptions and speaking roles, the embodiment of the present application uses the DeepVoice algorithm model for speech synthesis. The core of the DeepVoice algorithm is a sequence-to-sequence model based on a recurrent neural network (RNN). It takes the input text sequence as input and generates the corresponding speech waveform sequence through a neural network model. The speech generated by DeepVoice is natural, fluent, and close to human-like timbre.

[0289] The training process of DeepVoice consists of two stages: training a model from text to acoustic features and training a model from acoustic features to speech waveforms. In the first stage, the input text sequence is converted into an intermediate high-level semantic representation through the encoder model. Then, the decoder model converts these semantic representations into acoustic feature sequences, such as phonemes, tones, etc. In the second stage, the acoustic feature sequence is converted into the final speech waveform sequence through another model, the acoustic model. The core of the DeepVoice algorithm is a sequence-to-sequence model based on recurrent neural networks (RNN). It takes the input text sequence as input and generates the corresponding speech waveform sequence through the neural network model.

[0290] DeepVoice uses a progressive training approach, which is to first train a coarse model from a large-scale annotated dataset, then use the generated synthetic speech data to enhance the dataset, and use this data to continue training a more refined model. This approach helps improve the quality and naturalness of speech synthesis. Another noteworthy feature is the use of the Attention Mechanism, which is used to weight the semantic representation of the input in the decoder to better capture relevant contextual information. This helps improve the fluency and coherence of speech synthesis.

[0291] For a certain video, after the user starts the "listen to video" function, the audio and video information generated by the method provided in the embodiment of the present application will be extracted, and the audio and video information includes both the voice in the original video and the semantic information used to describe the picture. The audio and video information generated by the method provided in the embodiment of the present application allows the user to better understand the video content and thus obtain a better listening experience.

[0292] The method provided by the embodiments of the present application can help the visually impaired to better understand the content of movies and TV series when listening to them, and can also help the elderly to reduce the burden on their eyes when listening to movies and TV series. In addition, by listening to videos in a car terminal, the driver's attention is distracted less than by watching videos, while providing in-vehicle entertainment during the driving process.

[0293] Visual impairment is not only a perceptual obstacle, but also affects the development of abilities such as "understanding, thinking, memory, creation, and communication". Because of the lack of image information, they miss out on the information conveyed by the colorful visual world, and it also increases the difficulty for them to integrate into family and social life. The entertainment of the visually impaired is mainly listening to the radio, audio books, etc., and the content is relatively low. The video processing method provided in the embodiment of the present application can give the visually impaired the ability to "listen to videos", which helps to enrich the entertainment experience of the visually impaired and better integrate into society.

[0294] With the acceleration of population aging in my country, the number of people suffering from cataracts is increasing year by year. The incidence of cataracts in people over 60 years old in my country is about 80%, and the incidence of presbyopia after 52 years old is close to 100%. It is estimated that there are more than 200 million elderly people in my country who suffer from both presbyopia and cataracts. These eye diseases will affect the experience of watching videos for the elderly, and they are prone to discomfort such as eye fatigue and dryness. At the same time, watching electronic screens too much will easily further deteriorate the vision of the elderly. The video processing method provided by the embodiment of the present application can give the elderly the ability to "listen to videos" and reduce the burden on their eyes.

[0295] The Ministry of Industry and Information Technology's "General Design Specifications for Internet Websites Adapted to the Aged" and "General Design Specifications for Mobile Internet Applications (APPs) Adapted to the Aged" clearly point out that relevant Internet websites and APPs need to complete the oral web page structure service for aging-friendly and barrier-free transformation. This includes oral web page structure services: providing voice reading services for the overall description of the information structure, regional composition, and service functions of the current web page. Through the video processing method provided in the embodiment of this application, the video content can be structured and orally output to meet the needs of the elderly.

[0296] Users' demands for in-car entertainment systems are constantly increasing. In-car entertainment systems usually include audio, video and other functions to provide drivers and passengers with a richer and more convenient in-car entertainment experience. At present, some in-car entertainment systems support video playback, but drivers and passengers also need to pay attention to safety when using in-car entertainment systems. Watching videos can easily distract the driver and affect driving safety. The video processing method provided in the embodiment of the present application can "listen to videos" while the driver is driving, which will not distract the driver's visual attention and provide in-car entertainment for the driving process.

[0297] The video processing device in this application is described in detail below. Fig.16 . Fig.16 1 is a schematic diagram of an embodiment of a video processing device 10 in an embodiment of the present application. The video processing device 10 includes: a target audio and video acquisition module 110, an audio and video segmentation module 120, a segment semantic information generation module 130, a voice information generation module 140, an audio and video information first generation module 150, an audio and video information second generation module 160 and a target audio and video information generation module 170; specifically:

[0298] The target audio and video acquisition module 110 is used to acquire the target audio and video.

[0299] The audio and video segmentation module 120 is used to segment the target audio and video to generate M segments.

[0300] Each of the M segments carries a temporal label, M ≥ 1.

[0301] The segment semantic information generating module 130 is used to analyze the picture of each segment of the M segments and generate M segment semantic information corresponding to the M segments.

[0302] The segment semantic information is used to represent the picture content in the corresponding segment.

[0303] The voice information generating module 140 is used to identify the target object in each of the M segments, extract the voice information in each of the K segments containing the target object, and obtain the voice information of each of the K segments.

[0304] The voice information is generated based on the voice audio signal matching the target object, 1≤K≤M.

[0305] The first audio-visual information generating module 150 is used to concatenate the semantic information and the speech information corresponding to each of the K segments to generate the audio-visual information corresponding to each of the K segments.

[0306] The second audio-visual information generation module 160 is used to use the semantic information corresponding to each of the MK segments that do not contain the target object as the audio-visual information corresponding to each of the MK segments.

[0307] Among them, the MK segments that do not contain the target object refer to the remaining segments after subtracting the K segments that contain the target object from the M segments. The target audio and video information generation module 170 is used to splice the audio and video information corresponding to each segment in the K segments and the audio and video information corresponding to each segment in the MK segments according to the timing tags carried by each segment in the M segments, so as to generate the audio and video information of the target audio and video.

[0308] The video processing device provided by the embodiment of the present application generates corresponding semantic information for all the clips in the target audio and video, represents the picture content in the corresponding clip by the semantic information, extracts voice information from the clip containing the target object in the target audio and video to obtain the voice information of each clip, splices the semantic information corresponding to the clip containing the target object with the voice information to obtain the audio and video information corresponding to the clip containing the target object, uses the semantic information corresponding to the clip not containing the target object as the audio and video information of the clip, and finally splices the audio and video information corresponding to all the clips according to the timing tags carried by the clips to obtain the audio and video information of the target audio and video, and uses the audio and video information of the target audio and video as the target audio and video for the text of the oral description, thereby improving the accuracy and authenticity of the text of the oral description.

[0309] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the voice information generating module 140 is further used for:

[0310] Target object recognition is performed on the picture of each of the M segments to determine K segments containing the target object.

[0311] A speech audio signal of each of the K segments is obtained, and subtitle information of each of the K segments is obtained.

[0312] Subtitle information matching the target object is determined from the subtitle information of each of the K segments.

[0313] According to the subtitle information matching the target object in each of the K segments, a speech audio signal matching the target object is determined from the speech audio signals of the segments.

[0314] According to the speech audio signal matching the target object in each of the K segments, speech information matching the target object is generated.

[0315] The video processing device provided in the embodiment of the present application identifies the target object for each segment, and then screens out the segments containing the target object, determines the subtitle information matching the target object and the speech and audio signal corresponding to the subtitle information for each segment containing the target object, and then generates speech information matching the target object, thereby improving the accuracy of the speech information matching of the target object, and thereby improving the accuracy and authenticity of the generated audio-visual information containing the speech information.

[0316] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the voice information generating module 140 is further used for:

[0317] Each of the K segments is segmented according to the subtitle information of each of the K segments to obtain at least one first video frame corresponding to each of the K segments.

[0318] Wherein, a first video frame includes a subtitle information.

[0319] Subtitle information matching the target object is determined from a first video frame corresponding to each of the K segments.

[0320] The video processing device provided in the embodiment of the present application divides the fragments to obtain the video frames corresponding to each subtitle, and then determines the subtitle information matching the target object according to the subtitle information in the video frame, thereby improving the accuracy of matching the subtitle information for the target object.

[0321] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the voice information generating module 140 is further used for:

[0322] According to the subtitle information matching the target object in each of the K segments, abnormal speech and audio signal detection is performed on the speech and audio signal matching the target object to eliminate the speech and audio signal that does not correspond to the subtitle information in the speech and audio signal matching the target object.

[0323] The video processing device provided in the embodiment of the present application detects abnormal speech and audio signals to eliminate speech and audio signals that do not correspond to subtitle information from speech and audio signals that match the target object, thereby improving the accuracy of determining the speech and audio signals that match the target object.

[0324] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the voice information generating module 140 is further used for:

[0325] Get subtitle information of each of the M segments.

[0326] According to the subtitle information of each of the M segments, each of the M segments is segmented to obtain at least one second video frame corresponding to each of the M segments.

[0327] Target object recognition is performed on the second video frame corresponding to each of the M segments to obtain K segments containing the target object.

[0328] The video processing device provided in the embodiment of the present application divides each fragment into frames and identifies the target object on the obtained video frames to determine the fragments that only contain the target object, thereby improving the accuracy of target object identification on the fragments, and reducing the time for subsequent processing of the fragments by eliminating invalid video frames that do not contain the target object in the fragments, thereby improving the processing efficiency of the fragments.

[0329] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the voice information generating module 140 is further used for:

[0330] Get the mapping table between faces and characters of the target audio and video.

[0331] Among them, the face and role mapping table is used to represent the corresponding relationship between the role name and the face information.

[0332] Acquire a speech audio signal corresponding to each of the M segments.

[0333] According to the mapping table between human faces and roles, a corresponding relationship between role names and voice audio signals is established.

[0334] The video processing device provided in the embodiment of the present application establishes a correspondence between the role name and the voice audio signal, thereby facilitating the determination of the role name corresponding to the dialogue information in the subsequent generation of audio-visual information, thereby improving the accuracy and authenticity of the audio-visual information.

[0335] In this application Fig.16In an optional embodiment of the video processing device provided by the corresponding embodiment, the voice information generating module 140 is further used for:

[0336] Get the audio for each of the K segments.

[0337] The audio of each of the K segments is processed by frame division and windowing to obtain K time domain signals corresponding to the K segments.

[0338] Through fast Fourier transform, K time domain signals are converted into K frequency domain signals.

[0339] The K frequency domain signals are processed by the Mel filter to obtain K energy information corresponding to the K frequency domain signals.

[0340] The logarithm of the K energy information is taken to obtain K logarithmic energies corresponding to the K energy information.

[0341] A logarithmic energy sequence composed of K logarithmic energies is subjected to discrete cosine transform to obtain a frequency coefficient signal, wherein the logarithmic energy sequence is formed by splicing the K logarithmic energies.

[0342] The frequency coefficient signal is denoised to generate a speech audio signal.

[0343] The video processing device provided in the embodiment of the present application generates a speech audio signal by extracting speech features from the audio of the clip, thereby improving the accuracy of the speech audio signal generation process.

[0344] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the audio and video segmentation module 120 is further used for:

[0345] The target audio and video are used as the input of the video shot boundary detection model. The target audio and video are feature extracted through the video feature extraction network in the video shot boundary detection model to obtain the target audio and video feature sequence.

[0346] The target audio and video feature sequence is segmented through the temporal segmentation network in the video shot boundary detection model to obtain M segments.

[0347] The video processing device provided in the embodiment of the present application segments the target audio and video through a video shot boundary detection model, thereby improving the accuracy of segmenting the target audio and video.

[0348] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the audio and video segmentation module 120 is further used for:

[0349] Action detection is performed on the target object in each of the M segments to obtain action information of the target object in each of the M segments.

[0350] The action information includes a start timestamp when the target object starts to perform the action and an end timestamp when the target object ends the action.

[0351] According to the action information of the target object in each of the M segments, the M segments are classified to obtain M labels corresponding to the M segments.

[0352] Among them, the label is used to represent the action information of the target object in the clip.

[0353] The video processing device provided in the embodiment of the present application segments the target audio and video through a video shot boundary detection model, thereby improving the accuracy of segmenting the target audio and video.

[0354] In this application Fig.16 In an optional embodiment of the video processing device provided by the corresponding embodiment, the segment semantic information generating module 130 is further used for:

[0355] The M clips are used as the input of the video semantic model, and the M clips are encoded through the dual visual encoding network in the video semantic model to generate M encoding information corresponding to the M clips.

[0356] The M encoding information is text-encoded through the text encoder network in the video semantic model to generate M segment semantic information corresponding to the M segments.

[0357] The video processing device provided in the embodiment of the present application generates semantic information for each segment through a video semantic model, thereby improving the accuracy and authenticity of the voice information.

[0358] Fig.17 : is a schematic diagram of a server structure provided in an embodiment of the present application. The server 300 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 322 (for example, one or more processors) and memory 332, and one or more storage media 330 (for example, one or more mass storage devices) storing application programs 342 or data 344. Among them, the memory 332 and the storage medium 330 can be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the server. Furthermore, the central processing unit 322 can be configured to communicate with the storage medium 330 to execute a series of instruction operations in the storage medium 330 on the server 300.

[0359] The server 300 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input and output interfaces 358, and / or one or more operating systems 341, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM Etc.

[0360] The steps performed by the server in the above embodiment can be based on the Fig.17 The server structure shown.

[0361] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0362] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0363] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0364] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0365] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0366] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video processing method, characterized in that: include: Get the target audio and video; Segment the target audio and video to generate M segments, wherein each of the M segments carries a timing tag, M≥1; Analyze the picture of each of the M segments to generate M segment semantic information corresponding to the M segments, wherein the segment semantic information is used to represent the picture content in the corresponding segment; Performing target object recognition on the picture of each of the M segments, and extracting voice information on each of the K segments containing the target object, to obtain voice information of each of the K segments, wherein the voice information is generated according to a voice audio signal matching the target object, 1≤K≤M; Splicing the semantic information and the speech information corresponding to each of the K segments to generate audio-visual information corresponding to each of the K segments; Using semantic information corresponding to each of the MK segments that do not contain the target object as audio-visual information corresponding to each of the MK segments, wherein the MK segments that do not contain the target object refer to the remaining segments after subtracting the K segments that contain the target object from the M segments; According to the timing label carried by each of the M fragments, the audio and video information corresponding to each of the K fragments and the audio and video information corresponding to each of the MK fragments are spliced ​​to generate the audio and video information of the target audio and video.

2. The video processing method according to claim 1, characterized in that: The step of performing target object recognition on the picture of each of the M segments, extracting voice information from each of the K segments containing the target object, and obtaining voice information of each of the K segments includes: Performing target object recognition on the picture of each of the M segments to determine K segments containing the target object; Acquire a speech audio signal of each of the K segments, and acquire subtitle information of each of the K segments; Determining subtitle information matching the target object from the subtitle information of each of the K segments; Determining, according to the subtitle information matching the target object in each of the K segments, a speech audio signal matching the target object from the speech audio signals of the segments; According to the speech audio signal matching the target object in each of the K segments, speech information matching the target object is generated.

3. The video processing method according to claim 2, characterized in that: The determining the subtitle information matching the target object from the subtitle information of each of the K segments includes: According to the subtitle information of each of the K segments, each of the K segments is segmented to obtain at least one first video frame corresponding to each of the K segments, wherein the first video frame includes one piece of subtitle information; Subtitle information matching the target object is determined from a first video frame corresponding to each of the K segments.

4. The video processing method according to claim 2, characterized in that: After determining the speech audio signal matching the target object from the speech audio signals of the segments according to the subtitle information matching the target object in each of the K segments, the method further includes: According to the subtitle information matching the target object in each of the K segments, abnormal speech and audio signal detection is performed on the speech and audio signal matching the target object to eliminate the speech and audio signal that does not correspond to the subtitle information in the speech and audio signal matching the target object.

5. The video processing method according to claim 2, characterized in that: The step of performing target object recognition on the picture of each of the M segments to determine K segments containing the target object includes: Obtaining subtitle information of each of the M segments; According to the subtitle information of each of the M segments, each of the M segments is segmented to obtain at least one second video frame corresponding to each of the M segments; Target object recognition is performed on the second video frame corresponding to each of the M segments to obtain K segments containing the target object.

6. The video processing method according to claim 5, characterized in that: The step of performing target object recognition on the second video frame corresponding to each of the M segments to obtain K segments containing the target object includes: Performing face detection on the second video frame corresponding to each of the M segments to obtain a third video frame corresponding to each of the M segments and containing a face detection frame; Performing face recognition on a third video frame including a face detection frame corresponding to each of the M segments to obtain a face recognition result for each of the M segments; According to the face recognition result of each of the M segments, K segments with the same face recognition result are determined from the M segments.

7. The video processing method according to claim 6, characterized in that: After performing face recognition on the third video frame containing the face detection frame corresponding to each of the M segments to obtain the face recognition result of each of the M segments, the method further includes: Obtaining a mapping table of faces and roles of the target audio and video, wherein the mapping table of faces and roles is used to represent the corresponding relationship between role names and face information; Obtaining a speech audio signal corresponding to each of the M segments; According to the mapping table between human faces and roles, a corresponding relationship between the role name and the voice audio signal is established.

8. The video processing method according to claim 2, characterized in that: The obtaining of the speech audio signal of each of the K segments comprises: Obtaining the audio of each of the K segments; Performing frame-by-frame windowing processing on the audio of each of the K segments to obtain K time domain signals corresponding to the K segments; Convert the K time domain signals into K frequency domain signals by fast Fourier transform; Processing the K frequency domain signals through a Mel filter to obtain K energy information corresponding to the K frequency domain signals; Taking logarithms of the K pieces of energy information to obtain K logarithmic energies corresponding to the K pieces of energy information; Performing discrete cosine transform on the logarithmic energy sequence composed of the K logarithmic energies to obtain a frequency coefficient signal, wherein the logarithmic energy sequence is formed by splicing the K logarithmic energies; The frequency coefficient signal is subjected to denoising processing to generate a speech audio signal.

9. The video processing method according to claim 1, characterized in that: The step of segmenting the target audio and video to generate M segments includes: The target audio and video are used as input of a video shot boundary detection model, and features of the target audio and video are extracted through a video feature extraction network in the video shot boundary detection model to obtain a target audio and video feature sequence; The target audio and video feature sequence is segmented through the temporal segmentation network in the video shot boundary detection model to obtain M segments.

10. The video processing method according to claim 9, characterized in that: After the target audio and video feature sequence is segmented by the temporal segmentation network in the video shot boundary detection model to obtain M segments, the method further includes: Performing action detection on the target object in each of the M segments to obtain action information of the target object in each of the M segments, wherein the action information includes a start timestamp when the target object starts to perform the action and an end timestamp when the target object ends the action; The M segments are classified according to the action information of the target object in each of the M segments to obtain M labels corresponding to the M segments, wherein the labels are used to characterize the action information of the target object in the segments.

11. The video processing method according to claim 1, characterized in that: The analyzing the picture of each of the M segments to generate M segment semantic information corresponding to the M segments includes: Taking the M segments as input of a video semantic model, encoding the M segments through a dual visual encoding network in the video semantic model to generate M encoding information corresponding to the M segments; The M encoding information is text-encoded by a text encoder network in the video semantic model to generate M fragment semantic information corresponding to the M fragments.

12. A video processing device, characterized in that: include: A target audio and video acquisition module is used to acquire target audio and video; An audio and video segmentation module, used to segment the target audio and video to generate M segments, wherein each of the M segments carries a timing tag, M≥1; A segment semantic information generation module, used for analyzing the picture of each segment of the M segments, and generating M segment semantic information corresponding to the M segments, wherein the segment semantic information is used to represent the picture content in the corresponding segment; A voice information generation module, used to perform target object recognition on the picture of each of the M segments, extract voice information from each of the K segments containing the target object, and obtain voice information of each of the K segments, wherein the voice information is generated according to a voice audio signal matching the target object, 1≤K≤M; A first audio-visual information generation module, used for splicing semantic information and speech information corresponding to each of the K segments to generate audio-visual information corresponding to each of the K segments; A second audio-visual information generation module is used to use the semantic information corresponding to each segment of the MK segments that do not contain the target object as the audio-visual information corresponding to each segment of the MK segments, wherein the MK segments that do not contain the target object refer to the remaining segments after subtracting the K segments that contain the target object from the M segments; The target audio and video information generation module is used to splice the audio and video information corresponding to each of the K segments and the audio and video information corresponding to each of the MK segments according to the timing labels carried by each of the M segments, so as to generate the audio and video information of the target audio and video.

13. A computer device, characterized in that: include: memories, transceivers, processors, and bus systems; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, including executing the video processing method according to any one of claims 1 to 11; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate with each other.

14. A computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the video processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that The computer program is used by a processor to execute the video processing method according to any one of claims 1 to 11.

Images