Method, device, equipment and storage medium for adding subtitles to digital human video

Abstract

The present application relates to a method, device, equipment and storage medium for adding subtitles to a digital human video. It relates to the field of artificial intelligence technology. The method comprises: generating a digital human video according to an original text, wherein the original text comprises at least two first text segments; segmenting the digital human video according to the lip features of the digital human in each image frame in the digital human video to obtain at least two audio segments with timestamps; performing text recognition on at least two audio segments with timestamps to obtain second text segments with timestamps corresponding to each audio segment; adding subtitles to the digital human video according to the original text and the second text segments with timestamps corresponding to each audio segment. The present application improves the text matching degree between the second text segment corresponding to the segmented audio segment and the first text segment in the original text, improves the accuracy of adding the timestamp of the first text segment, and ensures the accuracy of adding subtitles to the digital human video.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and storage medium for adding subtitles to a digital human video. Background Art

[0002] With the development of audio and video technology, digital human videos are becoming more and more popular among users. In the process of generating digital human videos, adding subtitles to digital human videos has become an indispensable and important part of the process.

[0003] At present, the method of adding subtitles to digital human videos is to input text into the digital human engine, automatically generate audio through the Text To Speech (TTS) component in the digital human engine, and use an open source computer program (FFMpeg, Fast Forward Mpeg) to overlay the text on the video, thereby obtaining the digital human video with subtitles added. However, since the audio automatically generated by TTS cannot confirm the text corresponding to each audio content, there is a problem of mismatch between the audio content and the text content when the text is directly overlaid on the video, which in turn affects the user's viewing experience of the digital human video.

[0004] Therefore, how to accurately add subtitles to digital human videos is a technical problem that urgently needs to be solved. Summary of the invention

[0005] Based on this, it is necessary to provide an accurate method, device, equipment and storage medium for adding subtitles to digital human videos in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for adding subtitles to a digital human video, the method comprising:

[0007] Generate a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0008] Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio clips with time stamps;

[0009] Performing text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0010] Subtitles are added to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0011] In one embodiment, segmenting the digital human video according to the digital human lip features of each image frame in the digital human video includes:

[0012] According to the lip closure of the digital human in each image frame in the digital human video, the lip features of the digital human in each image frame are determined; wherein the lip features of the digital human include lip closure features and lip opening features;

[0013] Determine a segmented frame from the image frames corresponding to the lip closure features;

[0014] The digital human video is segmented according to the segmented frames.

[0015] In one embodiment, determining a segmented frame from an image frame corresponding to a lip closure feature comprises:

[0016] According to the continuity of the image frames corresponding to the lip closure features, the image frames corresponding to the lip closure features are divided into at least one candidate segment set;

[0017] Determine at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set;

[0018] From the image frames included in each target segmentation set, a segmentation frame is selected.

[0019] In one embodiment, the digital human video is segmented according to the digital human lip features of each image frame in the digital human video to obtain at least two audio segments with time stamps, including:

[0020] According to the lip features of the digital human in each image frame in the digital human video and the phoneme interval of the audio data in the digital human video, the digital human video is segmented to obtain at least two audio segments with time stamps.

[0021] In one embodiment, performing text recognition on at least two audio segments with timestamps to obtain second text segments with timestamps corresponding to the audio segments includes:

[0022] Performing text recognition on at least two audio segments with time stamps respectively by at least two audio recognizers to obtain candidate text segments with time stamps recognized by each audio recognizer for each audio segment;

[0023] According to the similarity of the candidate text segments recognized by different audio recognizers for the same audio segment, a second text segment with a timestamp corresponding to each audio segment is determined.

[0024] In one embodiment, adding subtitles to a digital human video according to the original text and the second text segment with a time stamp corresponding to each audio segment includes:

[0025] Determine the timestamps corresponding to the first text segments in the original text according to the second text segments with timestamps corresponding to the audio segments;

[0026] Subtitles are added to the digital human video according to each first text segment and the time stamp of each first text segment.

[0027] In a second aspect, the present application also provides a device for adding subtitles to a digital human video. The device comprises:

[0028] A generating module, used for generating a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0029] A segmentation module, used to segment the digital human video according to the digital human lip features of each image frame in the digital human video, to obtain at least two audio clips with time stamps;

[0030] A recognition module, configured to perform text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to the audio segments;

[0031] The adding module is used to add subtitles to the digital human video according to the original text and the second text segment with a time stamp corresponding to each audio segment.

[0032] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0033] Generate a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0034] Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio clips with time stamps;

[0035] Performing text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0036] Subtitles are added to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0037] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0038] Generate a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0039] Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio clips with time stamps;

[0040] Performing text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0041] Subtitles are added to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0042] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0043] Generate a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0044] Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio clips with time stamps;

[0045] Performing text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0046] Subtitles are added to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0047] The above-mentioned method, device, equipment and storage medium for adding subtitles to digital human videos first generate a digital human video based on at least two first text segments included in the original text; since the digital human lip features are related to the sentence segmentation method of the text and audio, the digital human video can be accurately segmented according to the digital human lip features of each image frame in the digital human video to obtain at least two accurate audio segments with timestamps, and text recognition is performed on the at least two audio segments with timestamps to obtain a second text segment with timestamp corresponding to each audio segment; then, based on the second text segment with timestamp corresponding to each audio segment, a timestamp is accurately added to each first text segment included in the original text; finally, based on each first text segment in the original text and the accurate timestamp of each first text segment, subtitles for the digital human video are generated. In the present application, audio segmentation can be performed according to the lip features of the digital human in each image frame in the digital human video, so that the segmented audio segments can match the sentence segmentation method of the digital human during the broadcasting process, that is, the text matching degree between the second text segment corresponding to the segmented audio segment and the first text segment in the original text is improved, and based on the timestamp of the second text segment identified in the segmented audio segment, a timestamp is added to each first text segment in the original text, thereby improving the accuracy of adding the first text segment timestamp, thereby ensuring the accuracy of adding subtitles to the digital human video based on the first text segment timestamp, and improving the user's viewing experience of the digital human video. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A diagram of the application environment of a method for adding subtitles to a digital human video provided in this embodiment;

[0049] Figure 2 A schematic diagram of the flow of a first method for adding subtitles to a digital human video provided in this embodiment;

[0050] Figure 3 A schematic diagram of a process of segmenting a digital human video provided in this embodiment;

[0051] Figure 4 A schematic diagram of a process for determining a segmented frame provided in this embodiment;

[0052] Figure 5 A schematic diagram of the flow of a second method for adding subtitles to a digital human video provided in this embodiment;

[0053] Figure 6 A structural block diagram of a device for adding subtitles to a digital human video provided in this embodiment;

[0054] Figure 7 This is a diagram of the internal structure of the computer device provided in this embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The embodiment of the present application provides a computer device, which may be a terminal, and its internal structure diagram may be as follows: Figure 1As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through wireless fidelity (Wireless-Fidelity, WIFI), a mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. When the computer program is executed by the processor, a method for adding subtitles to a digital human video is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0057] In one embodiment, Figure 2 As shown, a method for adding subtitles to a digital human video is provided, and the method comprises the following steps:

[0058] S201, generating a digital human video according to the original text.

[0059] The original text includes at least two first text segments.

[0060] Optionally, the original text refers to any text including multiple text fragments. For example, it can be funny text, lines or speeches. The first text segment refers to the text segment obtained by segmenting the original text according to a preset segmentation method. For example, the original text is segmented according to punctuation marks to obtain text segments. Assuming that the original text is "Study hard and make progress every day", the original text contains two first text segments, namely "Study hard" and "Make progress every day".

[0061] Digital human video refers to a video generated by combining virtual images, sounds, movements, etc. through technologies such as artificial intelligence. A digital human video includes multiple image frames.

[0062] In this embodiment, the original text can be input into a text segmenter, and the original text can be separated into at least two first text segments according to punctuation marks. The digital human engine is called to process the at least two first text segments, thereby generating a digital human video.

[0063] For example, the original text "Study hard and make progress every day" is input into the text segmenter to obtain the two first text segments "Study hard" and "Make progress every day". The digital human engine is called to process the two first text segments "Study hard" and "Make progress every day" to generate a digital human video.

[0064] S202: Segment the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio segments with time stamps.

[0065] Optionally, the digital human lip feature is a feature that characterizes the digital human's lip state. The lip state feature may be a lip opening feature corresponding to an open lip state or a lip closing feature corresponding to a closed lip state, or may be a feature such as the position of a lip feature point. The digital human lip feature is related to the audio. In this embodiment, whether the digital human in the digital human video is in a broadcasting state may be determined based on the lip state feature.

[0066] An audio clip refers to a clip captured from the audio data of a digital human video. The corresponding playback time of each audio clip in the digital human video is the timestamp of the audio clip. For example, the first 5 seconds of the audio clip captured from the audio data of a digital human video is the audio clip with a timestamp of "00:00-00:05".

[0067] In this embodiment, for each image frame in the digital human video, the characteristics of the digital human's lip state in each image frame are analyzed according to the image recognition algorithm. Since the characteristics of the digital human's lip state correspond to the lip closure characteristics when the digital human is in a non-broadcasting state, in this embodiment, the image frame corresponding to the lip closure characteristics can be used as the segmentation position of the digital human video, thereby obtaining at least two audio clips. Alternatively, since the position of the lip feature points of the digital human is in a changing state when the digital human is in a broadcasting state, and the position of the lip feature points of the digital human is unchanged when the digital human is in a non-broadcasting state, in this embodiment, an image frame in which the position of the lip feature points remains unchanged can be selected, and the unchanged image frame can be used as the segmentation position of the digital human video, thereby obtaining at least two audio clips.

[0068] S203: Perform text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment.

[0069] Optionally, the second text segment refers to the final text recognition result obtained by performing text recognition and analysis on each audio segment with a timestamp in the digital human video. In this embodiment, each audio segment corresponds to a timestamp, and the timestamp corresponding to each audio segment can be used as the timestamp of the second text segment corresponding to the audio segment.

[0070] In this embodiment, by performing text parsing and recognition on each audio segment with a timestamp in the digital human video, the second text segment corresponding to each audio segment can be output, and the timestamp corresponding to each audio segment can be used as the timestamp of the second text segment corresponding to the audio segment.

[0071] In an optional implementation, in order to obtain the precise second text segment corresponding to each audio clip, at least two audio clips with timestamps can be respectively subjected to text recognition by at least two audio recognizers to obtain candidate text segments with timestamps recognized by each audio recognizer for each audio clip; and based on the similarity of the candidate text segments recognized by different audio recognizers for the same audio clip, the second text segment with timestamp corresponding to each audio clip is determined.

[0072] Optionally, the audio recognizer refers to a program that converts audio into text. In this embodiment, the candidate text segment refers to the recognition result obtained by performing text recognition and analysis on each audio segment with a timestamp by the audio recognizer.

[0073] In this embodiment, for each audio segment with a timestamp, multiple audio recognizers can be used to perform text recognition on the audio segment respectively, and the recognition result corresponding to the audio segment by each audio recognizer is obtained, that is, the candidate text segment is obtained. For the recognition result of an audio segment, the candidate text segment of the same text is selected, and the number of candidate text segments of the same text is determined, and the candidate text segment of the same text with the largest number is used as the second text segment. Among them, the method of adding a timestamp to the second text segment is described above, and it will not be repeated here.

[0074] For example, suppose that five audio recognizers are used to perform text recognition on each audio clip, and the recognition results of three audio recognizers for one of the audio clips are "study hard", and the recognition results of two audio recognizers for the audio clip are "study hard", then the "study hard" recognized by the three audio recognizers will be used as the second text segment.

[0075] S204: Add subtitles to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0076] In this embodiment, according to each audio segment with a timestamp in the digital human video, a second text segment with a timestamp corresponding to each audio segment is obtained. Afterwards, a timestamp can be added to the original text (i.e., each first text segment in the original text) according to the second text segment with a timestamp corresponding to each audio segment, and subtitles can be added to the digital human video based on the original text with the timestamp added.

[0077] In an optional implementation, the timestamps corresponding to the first text segments in the original text are determined based on the second text segments with timestamps corresponding to the audio clips; and subtitles are added to the digital human video based on the first text segments and the timestamps of the first text segments.

[0078] In this embodiment, after obtaining the second text segment with a timestamp corresponding to each audio segment, the second text segment corresponding to each audio segment is calculated with the first text segments included in the original text for similarity, and the second text segment and the first text segment whose similarity is greater than the text threshold are found, and the second text segment and the first text segment whose similarity is greater than the text threshold are used as a group of matching pairs. For each group of matching pairs, the timestamp of the second text segment in the matching pair is used as the timestamp of the first text segment. This embodiment ensures that the first text segment with a timestamp in the digital human video matches the second text segment with a timestamp, and further, subtitles can be accurately added to the digital human video, thereby improving the user's viewing experience of the video.

[0079] Assume that in a set of matching pairs, the first text segment is "Study hard", and the second text segment includes: timestamp: 00:00-00:05, text segment: "Study hard", then the timestamp "00:00-00:05" of the second text segment in the matching pair is used as the timestamp of the first text segment "Study hard". Then, the first text segment "Study hard" is added to the image frame of 00:00-00:05 as the subtitle corresponding to the image frame of "00:00-00:05" in the digital human video. That is, the subtitles corresponding to the image frame of "00:00-00:05" in the digital human video are all "Study hard".

[0080] In the above-mentioned method for adding subtitles to a digital human video, first, a digital human video is generated based on at least two first text segments included in the original text; since the digital human lip features are related to the sentence segmentation method of the text and audio, the digital human video can be accurately segmented according to the digital human lip features of each image frame in the digital human video to obtain at least two accurate audio segments with timestamps, and text recognition is performed on the at least two audio segments with timestamps to obtain a second text segment with timestamp corresponding to each audio segment; then, based on the second text segment with timestamp corresponding to each audio segment, a timestamp is accurately added to each first text segment included in the original text; finally, based on each first text segment in the original text and the accurate timestamp of each first text segment, subtitles for the digital human video are generated. In the present application, audio segmentation can be performed according to the lip features of the digital human in each image frame in the digital human video, so that the segmented audio segments can match the sentence segmentation method of the digital human during the broadcasting process, that is, the text matching degree between the second text segment corresponding to the segmented audio segment and the first text segment in the original text is improved, and based on the timestamp of the second text segment identified in the segmented audio segment, a timestamp is added to each first text segment in the original text, thereby improving the accuracy of adding the first text segment timestamp, thereby ensuring the accuracy of adding subtitles to the digital human video based on the first text segment timestamp, and improving the user's viewing experience of the digital human video.

[0081] On the basis of the above-mentioned embodiment, in order to understand the segmentation of the digital human video, this embodiment provides the following optional implementation method for segmenting the digital human video, specifically as follows: Figure 3 As shown, the following steps are included:

[0082] S301, determining the lip features of the digital human in each image frame according to the lip closure status of the digital human in each image frame in the digital human video.

[0083] Among them, the digital human lip features include lip closing features and lip opening features.

[0084] Optionally, the lip closure state of the digital human refers to a state describing whether the lip state of the digital human is closed. The lip features of the digital human include lip closure features corresponding to the lip closure state and lip opening features corresponding to the lip opening state. In this embodiment, the lip features of the digital human can be represented by digital symbols, for example, 0 represents the lip closure feature of the digital human, and 1 represents the lip opening feature of the digital human.

[0085] In this embodiment, each image frame in the digital human video is analyzed to obtain the lip state of the digital human in each analyzed image frame, and according to the lip state of the digital human, the lip state feature of the digital human in each image frame is determined.

[0086] Optionally, the lip state of the digital person in each image frame is obtained. One possible implementation method is to pre-train a model for determining the lip state of the digital person in the image frame, and use the model to parse the data corresponding to the lip state of the digital person in the image frame, thereby obtaining the lip state of the digital person in each image frame after parsing. Another possible implementation method is to parse the data corresponding to the lip state of the digital person in the image frame through an image recognition algorithm, thereby obtaining the lip state of the digital person in each image frame after parsing.

[0087] Optionally, in this embodiment, the numbers corresponding to the features of the lip state of each frame of the image may be recorded, thereby obtaining a digital symbol sequence corresponding to the digital human video. For example, if 0 represents the closed lip feature of the digital human, and 1 represents the open lip feature of the digital human, then the digital symbol sequence corresponding to the digital human video may be 111100110110001111...10. In this embodiment, the digital symbol sequence corresponding to the digital human video is recorded to facilitate the subsequent rapid and accurate segmentation of the digital human video based on the digital symbol sequence corresponding to the digital human video, thereby obtaining multiple segments with timestamps and sounds.

[0088] S302: Determine a segmented frame from the image frame corresponding to the lip closure feature.

[0089] Optionally, the segmented frame refers to an image frame for segmenting the digital human video, that is, the digital human video is divided once at the segmented frame position.

[0090] In an optional implementation, after determining the digital human lip features of each image frame in the digital human video, all image frames corresponding to the closed features are used as candidate segmented frames, and then the segmented frames are determined from the candidate segmented frames.

[0091] Specifically, the first closed feature in the digital human lip feature is used as the first segmented frame, and it is determined whether the time difference of the next candidate segmented frame adjacent to the first segmented frame is greater than the segmentation threshold. If it is less than, the next candidate segmented frame adjacent to the first segmented frame is deleted from the candidate segmented frames; if it is greater, the next candidate segmented frame adjacent to the first segmented frame is used as the second segmented frame. After determining the second segmented frame, it is determined whether the time difference of the next candidate segmented frame adjacent to the second segmented frame is greater than the segmentation threshold. If it is less than, the next candidate segmented frame adjacent to the second segmented frame is deleted from the candidate segmented frames; if it is greater, the next candidate segmented frame adjacent to the second segmented frame is used as the third segmented frame, and so on, until the segmented frames in the digital human video are determined to be completed.

[0092] In another optional implementation, after determining the lip features of the digital human in each image frame in the digital human video, it is determined whether the previous frame of the image frame corresponding to the closed feature is an image frame corresponding to the open lip feature. If so, the image frame corresponding to the closed feature is used as a segmented frame. Otherwise, there is no need to use the image frame corresponding to the closed feature as a segmented frame.

[0093] S303: Segment the digital human video according to the segmented frames.

[0094] Optionally, the digital human video is divided once at the segmented frame position. Specifically, the position of each segmented frame in the digital human video is determined, and the digital human video is divided based on the position to obtain multiple digital human video segments. For example, assuming that there are 2 segmented frames, the digital human video can be divided into 3 segments according to the position of each segmented frame in the digital human video.

[0095] In this embodiment, the lip features of the digital human are refined into lip closure features and lip opening features, and based on the lip closure features in the image frames in the digital human video, multiple segmented frames for segmenting the digital human video are determined, and based on each segmented frame, the digital human video is segmented quickly and accurately. Obviously, in this embodiment, the lip features of the digital human are divided in a finer granularity, that is, the lip closure features and the lip opening features are obtained, and then based on the lip closure features, the image frames corresponding to the interruptions in the digital human video corresponding to the lip closure features can be quickly and accurately obtained, and then the digital human video is segmented according to the image frames corresponding to the interruptions in the digital human video, thereby improving the segmentation efficiency and accuracy of the digital human video.

[0096] On the basis of the above embodiment, in order to reduce the error caused by determining the segmented frames based only on the lip closure features and segmenting the digital human video based on the segmented frames, in this embodiment, the above S302 can be performed as follows Figure 4 The method shown determines the segmented frames.

[0097] S401. Divide the image frames corresponding to the lip closure features into at least one candidate segment set according to the continuity of the image frames corresponding to the lip closure features.

[0098] Optionally, the continuity of image frames corresponding to lip closure features is used to describe the characteristics of the same lip closure feature appearing continuously in multiple image frames. For example, the same lip closure feature may appear continuously in a sequence of digital symbols corresponding to a digital human video. The candidate segment set refers to a set obtained by preliminarily dividing the image frames corresponding to the lip closure feature according to the continuity of the lip closure feature. In this embodiment, multiple image frames in which the same lip closure feature appears continuously can be used as a candidate segment set, and a non-continuous lip closure feature that only appears in one image frame can also be used as a candidate segment set alone.

[0099] Optionally, in this embodiment, the digital symbol sequence corresponding to the digital human video can be divided according to the continuity of the image frames corresponding to the lip closure features, so as to obtain multiple continuous lip closure features and discontinuous lip closure features; and adjacent continuous lip closure features are taken as a set, and discontinuous lip closure features are also taken as a set, so as to obtain a candidate segment set.

[0100] Assuming that the digital symbol sequence corresponding to the image frame in the digital human video is 111101100110001110000011110, based on the continuity of the feature "0", the image frame corresponding to the digital symbol can be divided into a set of candidate segments corresponding to "0", "00", "000", "000000" and "00".

[0101] S402: Determine at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set.

[0102] Optionally, the target segmentation set refers to a set that satisfies the correspondence between the number of lip closure features and sentence segmentation and is used to segment the digital human video.

[0103] In this embodiment, based on the number of image frames included in each candidate segment set and the pre-set correspondence between the number of lip closure features and sentence segments, a set that satisfies the correspondence between the number of lip closure features and sentence segments is selected from the candidate segment set, that is, the target segment set is selected.

[0104] Assume that the preset correspondence between the number of lip closure features and the sentence segmentation is that the digital human video is segmented when the number of lip closure features is not less than 3. In this embodiment, based on the number of "0", "00", "000", and "000000" corresponding to the candidate segmentation set, and the preset correspondence between the number of lip closure features and the sentence segmentation, the candidate segmentation sets "000" and "000000" can be determined as the target segmentation sets.

[0105] S404: Select a segment frame from the image frames included in each target segment set.

[0106] In this embodiment, after determining the target segmentation set, any image frame can be selected from each target segmentation set as a segmentation frame. For example, for each target segmentation set, the first frame, a middle frame, or the last frame can be selected as a segmentation frame, and the digital human video is segmented based on the segmentation frame.

[0107] In this embodiment, first, based on the continuity of the image frames corresponding to the lip closure features, a candidate segment set is selected, and multiple target segment sets are selected from the candidate segment set, and then a segment frame is selected from each target segment set. In this embodiment, in the process of determining the segmented frame, the continuity of the image frames corresponding to the lip closure features is introduced, which can reduce the segmentation error caused by determining the segmented frame based only on the lip closure features, and improve the accuracy of determining the segmented frame.

[0108] For example, in this embodiment, in the process of determining the segmented frame, the segmented frame is accurately selected based on the candidate segment sets "000" and "000000", avoiding the segmentation error caused by determining the segmented frame from the candidate segment sets "0" and "00".

[0109] Based on the above embodiment, this embodiment also provides another optional implementation method of segmenting the digital human video to obtain at least two audio segments with time stamps. Specifically, the digital human video is segmented according to the digital human lip features of each image frame in the digital human video and the phoneme interval of the audio data in the digital human video to obtain at least two audio segments with time stamps.

[0110] In this embodiment, based on the content described in the above embodiments, at least one first segmented frame for segmenting the digital human video can be determined according to the lip features of the digital human in each image frame in the digital human video, and at least one second segmented frame for segmenting the digital human video can be determined according to the phoneme interval of the audio data in the digital human video. Based on the first segmented frame and the second segmented frame, a target segmented frame for segmenting the digital human video is determined, and the digital human video is segmented based on the target segmented frame to obtain at least two audio clips with timestamps.

[0111] In an optional implementation, the first segmented frame and the second segmented frame are fused, for example, the first segmented frame and the second segmented frame are averaged to obtain a target segmented frame, and the digital human video is segmented based on the target segmented frame to obtain at least two audio segments with timestamps.

[0112] In another optional implementation, the similarity between the first segmented frame and the second segmented frame is calculated. If the similarity between the first segmented frame and the second segmented frame is greater than a frame threshold, any one of the first segmented frame or the second segmented frame is used as a target segmented frame, and the digital human video is segmented based on the target segmented frame to obtain at least two audio segments with timestamps. If the similarity between the first segmented frame and the second segmented frame is less than the frame threshold, a frame is manually selected from the first segmented frame or the second segmented frame as a target segmented frame, and the digital human video is segmented based on the target segmented frame to obtain at least two audio segments with timestamps.

[0113] In order to understand the determination of the second segmented frame, in this embodiment, an optional implementation method is provided for determining at least one second segmented frame for segmenting the digital human video according to the phoneme interval of the audio data in the digital human video. Specifically, the audio data is extracted from the digital human video, and the extracted audio data is used to generate an audio file. The phonemes in the audio file are identified frame by frame by the audio segmenter, and the timestamp corresponding to each phoneme is determined. Optionally, the audio segmenter determines whether the sound interval between the two adjacent phonemes identified exceeds the interval threshold; if it exceeds, any image frame corresponding to the two adjacent phonemes is used as the second segmented frame; otherwise, there is no need to use the image frame corresponding to the two adjacent phonemes as the second segmented frame, wherein the interval threshold is proportional to the sound playback speed corresponding to the audio.

[0114] In this embodiment, the first segmentation frame can be determined according to the digital human lip features of each image frame in the digital human video, and the second segmentation frame for segmenting the digital human video can be determined according to the phoneme interval of the audio data in the digital human video, and the target segmentation frame for segmenting the digital human video can be determined based on the first segmentation frame and the second segmentation frame, and the digital human video can be segmented based on the target segmentation frame to obtain at least two audio segments with timestamps. In this embodiment, the digital human video can be segmented based on the two dimensions of the digital human lip features and the phoneme interval of the audio data, thereby improving the accuracy of segmenting the digital human video.

[0115] Based on the above embodiment, this embodiment provides a second optional implementation of the method for adding subtitles to a digital human video, such as Figure 5 As shown, the method comprises the following steps:

[0116] S501: Generate a digital human video based on the original text.

[0117] The original text includes at least two first text segments.

[0118] S502: Determine the lip features of the digital human in each image frame according to the lip closure status of the digital human in each image frame in the digital human video.

[0119] Among them, the digital human lip features include lip closing features and lip opening features.

[0120] S503: Divide the image frames corresponding to the lip closure features into at least one candidate segment set according to the continuity of the image frames corresponding to the lip closure features.

[0121] S504: Determine at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set.

[0122] S505: Select a segment frame from the image frames included in each target segment set.

[0123] S506: Segment the digital human video according to the segmented frames to obtain at least two audio segments with time stamps.

[0124] S507: Perform text recognition on at least two audio segments with time stamps respectively by at least two audio recognizers to obtain candidate text segments with time stamps recognized by each audio recognizer for each audio segment.

[0125] S508: Determine a second text segment with a timestamp corresponding to each audio segment according to similarities of candidate text segments recognized by different audio recognizers for the same audio segment.

[0126] S509: Determine the timestamps corresponding to the first text segments in the original text according to the second text segments with timestamps corresponding to the audio segments.

[0127] S510: Add subtitles to the digital human video according to each first text segment and the time stamp of each first text segment.

[0128] In an optional implementation, the digital human video may be segmented according to the digital human lip features of each image frame in the digital human video and the phoneme interval of the audio data in the digital human video to obtain at least two audio segments with time stamps.

[0129] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0130] Based on the same inventive concept, the embodiment of the present application also provides a device for adding subtitles to a digital human video for implementing the method for adding subtitles to a digital human video involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme and beneficial effects recorded in the above method, so the specific limitations in the embodiments of one or more devices for adding subtitles to digital human videos provided below can refer to the limitations of the method for adding subtitles to a digital human video above, and will not be repeated here.

[0131] In one embodiment, Figure 6 As shown, a device 1 for adding subtitles to a digital human video is provided, comprising:

[0132] A generating module 601 is used to generate a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0133] Segmentation module 602, used to segment the digital human video according to the digital human lip features of each image frame in the digital human video, to obtain at least two audio segments with time stamps;

[0134] The recognition module 603 is used to perform text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0135] The adding module 604 is used to add subtitles to the digital human video according to the original text and the second text segments with time stamps corresponding to each audio segment.

[0136] In one embodiment, the segmentation module 602 includes:

[0137] A first determining unit is used to determine the digital human lip features of each image frame according to the lip closure of the digital human in each image frame in the digital human video; wherein the digital human lip features include lip closure features and lip opening features;

[0138] A second determining unit is used to determine a segmented frame from the image frame corresponding to the lip closure feature;

[0139] The segmentation unit is used to segment the digital human video according to the segmentation frames.

[0140] In one embodiment, the second determining unit is specifically configured to:

[0141] According to the continuity of the image frames corresponding to the lip closure features, the image frames corresponding to the lip closure features are divided into at least one candidate segment set;

[0142] Determine at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set;

[0143] From the image frames included in each target segmentation set, a segmentation frame is selected.

[0144] In one embodiment, the segmentation module 602 is specifically configured to:

[0145] According to the lip features of the digital human in each image frame in the digital human video and the phoneme interval of the audio data in the digital human video, the digital human video is segmented to obtain at least two audio segments with time stamps.

[0146] In one embodiment, the identification module 603 is specifically configured to:

[0147] Performing text recognition on at least two audio segments with time stamps respectively by at least two audio recognizers to obtain candidate text segments with time stamps recognized by each audio recognizer for each audio segment;

[0148] According to the similarity of the candidate text segments recognized by different audio recognizers for the same audio segment, a second text segment with a timestamp corresponding to each audio segment is determined.

[0149] In one embodiment, the adding module 604 is specifically configured to:

[0150] Determine the timestamps corresponding to the first text segments in the original text according to the second text segments with timestamps corresponding to the audio segments;

[0151] Subtitles are added to the digital human video according to each first text segment and the time stamp of each first text segment.

[0152] Each module in the above-mentioned device for adding subtitles to digital human videos can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0153] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, subtitles are added to the digital human video. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0154] Those skilled in the art will understand that Figure 7The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: generating a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0156] Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio clips with time stamps;

[0157] Performing text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0158] Subtitles are added to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0159] In one embodiment, segmenting the digital human video according to the digital human lip features of each image frame in the digital human video includes:

[0160] According to the lip closure of the digital human in each image frame in the digital human video, the lip features of the digital human in each image frame are determined; wherein the lip features of the digital human include lip closure features and lip opening features;

[0161] Determine a segmented frame from the image frames corresponding to the lip closure features;

[0162] The digital human video is segmented according to the segmented frames.

[0163] In one embodiment, determining a segmented frame from an image frame corresponding to a lip closure feature comprises:

[0164] According to the continuity of the image frames corresponding to the lip closure features, the image frames corresponding to the lip closure features are divided into at least one candidate segment set;

[0165] Determine at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set;

[0166] From the image frames included in each target segmentation set, a segmentation frame is selected.

[0167] In one embodiment, the digital human video is segmented according to the digital human lip features of each image frame in the digital human video to obtain at least two audio segments with time stamps, including:

[0168] According to the lip features of the digital human in each image frame in the digital human video and the phoneme interval of the audio data in the digital human video, the digital human video is segmented to obtain at least two audio segments with time stamps.

[0169] In one embodiment, performing text recognition on at least two audio segments with timestamps to obtain second text segments with timestamps corresponding to the audio segments includes:

[0170] Performing text recognition on at least two audio segments with time stamps respectively by at least two audio recognizers to obtain candidate text segments with time stamps recognized by each audio recognizer for each audio segment;

[0171] According to the similarity of the candidate text segments recognized by different audio recognizers for the same audio segment, a second text segment with a timestamp corresponding to each audio segment is determined.

[0172] In one embodiment, adding subtitles to a digital human video according to the original text and the second text segment with a time stamp corresponding to each audio segment includes:

[0173] Determine the timestamps corresponding to the first text segments in the original text according to the second text segments with timestamps corresponding to the audio segments;

[0174] Subtitles are added to the digital human video according to each first text segment and the time stamp of each first text segment.

[0175] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: generating a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0176] Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio clips with time stamps;

[0177] Performing text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0178] Subtitles are added to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0179] In one embodiment, segmenting the digital human video according to the digital human lip features of each image frame in the digital human video includes:

[0180] According to the lip closure of the digital human in each image frame in the digital human video, the lip features of the digital human in each image frame are determined; wherein the lip features of the digital human include lip closure features and lip opening features;

[0181] Determine a segmented frame from the image frames corresponding to the lip closure features;

[0182] The digital human video is segmented according to the segmented frames.

[0183] In one embodiment, determining a segmented frame from an image frame corresponding to a lip closure feature comprises:

[0184] According to the continuity of the image frames corresponding to the lip closure features, the image frames corresponding to the lip closure features are divided into at least one candidate segment set;

[0185] Determine at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set;

[0186] From the image frames included in each target segmentation set, a segmentation frame is selected.

[0187] In one embodiment, the digital human video is segmented according to the digital human lip features of each image frame in the digital human video to obtain at least two audio segments with time stamps, including:

[0188] According to the lip features of the digital human in each image frame in the digital human video and the phoneme interval of the audio data in the digital human video, the digital human video is segmented to obtain at least two audio segments with time stamps.

[0189] In one embodiment, performing text recognition on at least two audio segments with timestamps to obtain second text segments with timestamps corresponding to the audio segments includes:

[0190] Performing text recognition on at least two audio segments with time stamps respectively by at least two audio recognizers to obtain candidate text segments with time stamps recognized by each audio recognizer for each audio segment;

[0191] According to the similarity of the candidate text segments recognized by different audio recognizers for the same audio segment, a second text segment with a timestamp corresponding to each audio segment is determined.

[0192] In one embodiment, adding subtitles to a digital human video according to the original text and the second text segment with a time stamp corresponding to each audio segment includes:

[0193] Determine the timestamps corresponding to the first text segments in the original text according to the second text segments with timestamps corresponding to the audio segments;

[0194] Subtitles are added to the digital human video according to each first text segment and the time stamp of each first text segment.

[0195] The present application also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented: generating a digital human video according to an original text, wherein the original text includes at least two first text segments;

[0196] Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio clips with time stamps;

[0197] Performing text recognition on at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to each audio segment;

[0198] Subtitles are added to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments.

[0199] In one embodiment, segmenting the digital human video according to the digital human lip features of each image frame in the digital human video includes:

[0200] According to the lip closure of the digital human in each image frame in the digital human video, the lip features of the digital human in each image frame are determined; wherein the lip features of the digital human include lip closure features and lip opening features;

[0201] Determine a segmented frame from the image frames corresponding to the lip closure features;

[0202] The digital human video is segmented according to the segmented frames.

[0203] In one embodiment, determining a segmented frame from an image frame corresponding to a lip closure feature comprises:

[0204] According to the continuity of the image frames corresponding to the lip closure features, the image frames corresponding to the lip closure features are divided into at least one candidate segment set;

[0205] Determine at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set;

[0206] From the image frames included in each target segmentation set, a segmentation frame is selected.

[0207] In one embodiment, the digital human video is segmented according to the digital human lip features of each image frame in the digital human video to obtain at least two audio segments with time stamps, including:

[0208] According to the lip features of the digital human in each image frame in the digital human video and the phoneme interval of the audio data in the digital human video, the digital human video is segmented to obtain at least two audio segments with time stamps.

[0209] In one embodiment, performing text recognition on at least two audio segments with timestamps to obtain second text segments with timestamps corresponding to the audio segments includes:

[0210] Performing text recognition on at least two audio segments with time stamps respectively by at least two audio recognizers to obtain candidate text segments with time stamps recognized by each audio recognizer for each audio segment;

[0211] According to the similarity of the candidate text segments recognized by different audio recognizers for the same audio segment, a second text segment with a timestamp corresponding to each audio segment is determined.

[0212] In one embodiment, adding subtitles to a digital human video according to the original text and the second text segment with a time stamp corresponding to each audio segment includes:

[0213] Determine the timestamps corresponding to the first text segments in the original text according to the second text segments with timestamps corresponding to the audio segments;

[0214] Subtitles are added to the digital human video according to each first text segment and the time stamp of each first text segment.

[0215] It should be noted that the data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all data authorized by the user or fully authorized by all parties.

[0216] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0217] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0218] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for adding subtitles to a digital human video, characterized in that: The method comprises: Generate a digital human video according to an original text, wherein the original text includes at least two first text segments; Segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio segments with time stamps; Performing text recognition on the at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to the respective audio segments; Adding subtitles to the digital human video according to the original text and the second text segments with timestamps corresponding to the audio segments; The digital human lip features include lip closing features and lip opening features, and segmenting the digital human video according to the digital human lip features of each image frame in the digital human video includes: According to the continuity of the image frames corresponding to the lip closure features, the image frames corresponding to the lip closure features are divided into at least one candidate segment set; Determining at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set; Selecting a segmented frame from the image frames included in each target segmentation set; The digital human video is segmented according to the segmented frames.

2. The method according to claim 1, characterized in that The method further comprises: According to the lip closure status of the digital human in each image frame in the digital human video, the lip features of the digital human in each image frame are determined.

3. The method according to any one of claims 1 to 2, characterized in that: The step of segmenting the digital human video according to the digital human lip features of each image frame in the digital human video to obtain at least two audio segments with time stamps includes: According to the digital human lip features of each image frame in the digital human video and the phoneme interval of the audio data in the digital human video, the digital human video is segmented to obtain at least two audio segments with time stamps.

4. The method according to any one of claims 1 to 2, characterized in that: The performing text recognition on the at least two audio segments with time stamps to obtain second text segments with time stamps corresponding to the audio segments includes: Performing text recognition on the at least two audio segments with timestamps respectively by at least two audio recognizers to obtain candidate text segments with timestamps recognized by each audio recognizer for each audio segment; According to the similarity of the candidate text segments recognized by different audio recognizers for the same audio segment, a second text segment with a timestamp corresponding to each audio segment is determined.

5. The method according to any one of claims 1 to 2, characterized in that: The adding subtitles to the digital human video according to the original text and the second text segments with time stamps corresponding to the audio segments includes: Determining the timestamps corresponding to the first text segments in the original text according to the second text segments with timestamps corresponding to the audio segments; Subtitles are added to the digital human video according to the first text segments and the timestamps of the first text segments.

6. A device for adding subtitles to a digital human video, characterized in that: The device comprises: A generating module, configured to generate a digital human video according to an original text, wherein the original text includes at least two first text segments; A segmentation module, configured to segment the digital human video according to the digital human lip features of each image frame in the digital human video, to obtain at least two audio segments with time stamps; A recognition module, configured to perform text recognition on the at least two audio segments with time stamps to obtain a second text segment with time stamps corresponding to each audio segment; An adding module, used for adding subtitles to the digital human video according to the original text and the second text segments with time stamps corresponding to each audio segment; The digital human lip features include lip closing features and lip opening features, and segmenting the digital human video according to the digital human lip features of each image frame in the digital human video includes: According to the continuity of the image frames corresponding to the lip closure features, the image frames corresponding to the lip closure features are divided into at least one candidate segment set; Determining at least one target segment set from each candidate segment set according to the number of image frames included in each candidate segment set; Selecting a segmented frame from the image frames included in each target segmentation set; The digital human video is segmented according to the segmented frames.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images