Data processing method and system for digital human model training

Abstract

The invention discloses a data processing method and system for digital human model training, and the method comprises the steps: screening a broadcast type high-definition real human video in a digital human material library, and storing the video as a video file; recording the starting time and the ending time of the segment where only one face is located in the video file; independently extracting audio content in the video file; recording the starting time and the ending time of the segment where the voice is located in the audio file; and based on the starting time and the ending time of the segment where the human face is located in the video file and the starting time and the ending time of an intersection of the starting time and the ending time of the segment where the human voice is located in the audio file, cutting the video file to obtain an effective video segment. The method has the advantages that high-quality data is provided for model training, synchronization and matching of human faces and human voices in the data are ensured, simulation of the digital human in the aspects of facial expressions, language expressions and the like is more accurate and harmonious, and the continuity of overall expression is improved.

Description

Technical Field

[0001] The present invention relates to the cross-technical field of computer vision, speech processing and artificial intelligence, and in particular to a data processing method and system for training digital human models. Background Art

[0002] When processing digital humans, existing technologies may have problems such as a lack of synchronization or matching between the face and the audio, and the inability to focus on the face and voice of a single individual, causing confusion and interference among multiple faces, making it impossible for the model to accurately learn and simulate the characteristics and expressions of a specific individual. In addition, there is also the problem of data redundancy, where no effective video clips are cut out, and there is a large amount of useless or duplicate data, which increases the burden of data storage and processing. Summary of the invention

[0003] The purpose of the present invention is to provide a data processing method and system for digital human model training, so as to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A data processing method for digital human model training comprises the following steps:

[0006] S1. Video data collection: select high-definition live-action videos of broadcasting in the digital human material library and save these videos as video files;

[0007] S2. Face detection: Perform face detection on each frame of the video file. For an image with only one face, record the start time and end time of the segment where the face is located in the video file.

[0008] S3, audio track separation: extract the audio content from the video file separately and save the audio content as a wav format audio file;

[0009] S4, audio detection: feature extraction and pattern matching are performed on the audio signal of each frame or each time period in the audio file to determine whether the audio signal of the frame or the time period is a sound segment or a silent segment, and to distinguish between human voice and non-human voice in the sound segment. For the human voice segment, the start time and end time of the human voice segment in the audio file are recorded;

[0010] S5. Cropping of effective video segments: based on the start and end time of the intersection of the start and end time of the segment containing the face in the video file and the start and end time of the segment containing the voice in the audio file, the video file is cropped to obtain effective video segments.

[0011] Preferably, step S2 specifically includes the following contents:

[0012] S21, reading the saved video file, and analyzing and processing each frame of the video file;

[0013] S22. Use a deep learning-based face detection model to determine whether there is one and only one face in the current frame image. If so, record the start time and end time of the segment containing the face in the video file; otherwise, skip the current frame image.

[0014] Preferably, step S4 specifically includes the following contents:

[0015] S41, reading the audio file to be analyzed;

[0016] S42, using audio activity detection technology to extract features from the audio signal of each frame or each time period of the audio file, and determining whether the audio signal of the frame or the time period is a sound segment or a silent segment based on a corresponding threshold;

[0017] S43, using a speech recognition algorithm-related model to perform feature extraction and pattern matching on the extracted voiced segment to distinguish between human voice and non-human voice in the voiced segment;

[0018] S44. For the segment determined to be a human voice, accurately record its start time and end time in the audio file.

[0019] Preferably, step S42 is specifically as follows: for the energy characteristics of the sound, the sum of the squares of the amplitudes of the audio signals of each frame or each time period is calculated to represent the energy thereof; for the frequency characteristics of the sound, the distribution of the audio signals of each frame or each time period in different frequency bands is analyzed to characterize the frequency thereof; and the energy threshold and the frequency threshold are set. When the energy and the frequency of the audio signals of a certain frame or each time period exceed the corresponding thresholds, the audio signals of the frame or the time period are considered to have sound activity, otherwise the audio signals of the frame or the time period are considered to be silent segments.

[0020] Preferably, step S42 specifically includes analyzing the spectrum, rhythm, and timbre characteristics of the sound in the audio segment, comparing and analyzing them with the pre-learned human voice and non-human voice patterns, and then distinguishing the human voice and non-human voice in the audio segment.

[0021] Preferably, step S5 specifically includes the following contents:

[0022] S51, comparing and calculating the start time and end time of the segment containing the face extracted in step S2 and the start time and end time of the segment containing the voice extracted in step S4, and determining the intersection of the time corresponding to the segment containing the face and the segment containing the voice;

[0023] S52, using a video editing tool or a related programming interface to perform a cropping operation on the video file according to the start time and the end time of the intersection;

[0024] S53: Repeat the above process to cut the video file into multiple valid video clips.

[0025] Preferably, the resolution of the high-definition live video is above 1080*1920.

[0026] Preferably, after step S5, the method further includes:

[0027] S6. Video transcoding: Transcode the valid video clips into mp4 format to obtain the final video.

[0028] The present invention also aims to provide a data processing system for digital human model training, the system can implement any of the above methods, the system comprises:

[0029] Video data acquisition module: selects high-definition live broadcast videos in the digital human material library and saves these videos as video files;

[0030] Face detection module: performs face detection on each frame of the video file. For an image with only one face, it records the start and end time of the segment where the face is located in the video file.

[0031] Audio track separation module: extract the audio content from the video file separately and save the audio content as a wav format audio file;

[0032] Audio detection module: extract features and perform pattern matching on the audio signal of each frame or each time period in the audio file, determine whether the audio signal of the frame or the time period is a sound segment or a silent segment, and distinguish between human voice and non-human voice in the sound segment. For the human voice segment, record the start time and end time of the human voice segment in the audio file;

[0033] Valid video segment cropping module: based on the start and end time of the intersection between the start and end time of the segment where the face is located in the video file and the start and end time of the segment where the voice is located in the audio file, the video file is cropped to obtain valid video segments;

[0034] Video transcoding module: uniformly transcodes valid video clips into mp4 format to obtain the final video.

[0035] The beneficial effects of the present invention are as follows: 1. It can optimize the quality of digital human training data, improve the fidelity of digital human performance, and enhance the adaptability and versatility of digital human. 2. By accurately screening and cutting out effective segments containing clear human faces and accurate human voices, high-quality data is provided for model training, ensuring the synchronization and matching of human faces and human voices in the data, making the simulation of digital humans in facial expressions, language expressions, etc. more accurate and coordinated, and improving the coherence of overall performance. 3. It removes irrelevant and redundant data, reduces the computational workload and time cost of model training, and speeds up training. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a flow chart of a data processing method in an embodiment of the present invention;

[0037] Figure 2 is a flow chart of face detection in an embodiment of the present invention;

[0038] Figure 3 is a flow chart of audio detection in an embodiment of the present invention;

[0039] Figure 4 4 is a flowchart of effective video clip cutting in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] like Figure 1 As shown, in this embodiment, a data processing method for digital human model training is provided, which mainly includes the following parts:

[0042] 1. Video Data Collection

[0043] Select high-definition live-action broadcast videos from the company's digital human material library and save them as video files.

[0044] Broadcast videos were chosen because they usually have clearer and more standardized language expressions and facial expressions, which are conducive to the learning and training of subsequent models.

[0045] The resolution of high-definition live-action videos is required to be above 1080*1920, which can ensure that the video provides richer details and clearer image information, helping the model to more accurately capture facial features and expression changes.

[0046] 2. Face Detection

[0047] Perform face detection on each frame of the video file. For images with only one face, record the start and end time of the segment where the face is located in the video file.

[0048] like Figure 2 As shown, the specific steps include the following:

[0049] 2.1. Read the saved video file and analyze and process each frame of the video file.

[0050] 2.2. Use a deep learning-based face detection model to determine whether there is one and only one face in the current frame image. If so, record the start and end time of the segment containing the face in the video file; otherwise, skip the current frame image.

[0051] In this embodiment, only images with one and only one face are recorded in time, which can avoid interference and confusion caused by multiple faces and enable the model to focus on learning and analyzing the facial features and expressions of a single individual. For frames or videos that do not meet the conditions (no face or the number of faces is not 1), they are directly skipped and the next image or video is processed.

[0052] 3. Track Separation

[0053] Extract the audio content from the video file separately for subsequent speech analysis and processing. Save the audio content as a WAV format audio file, because WAV is a common lossless audio format that can retain high-quality audio information, which is conducive to subsequent speech processing and model training.

[0054] 4. Audio Detection

[0055] Perform feature extraction and pattern matching on the audio signal of each frame or each time period in the audio file to determine whether the audio signal of the frame or the time period is a sound segment or a silent segment, and distinguish between human voice and non-human voice in the sound segment. For the human voice segment, record the start time and end time of the human voice segment in the audio file.

[0056] In this embodiment, the audio file is analyzed to determine which segments in the audio stream are human voice segments by means of audio activity detection, and the start and end time of each human voice segment is recorded.

[0057] like Figure 3 As shown, the specific steps include the following:

[0058] 4.1. Read the audio file to be analyzed.

[0059] 4.2. Use audio activity detection technology to extract features from the audio signal of each frame or each time period of the audio file, and determine whether the audio signal of the frame or the time period is a sound segment or a silent segment based on the corresponding threshold.

[0060] For the energy characteristics of the sound, the sum of the squares of the amplitudes of the audio signal in each frame or each time period is calculated to represent its energy. For the frequency characteristics of the sound, the distribution of the audio signal in each frame or each time period in different frequency bands is analyzed to characterize its frequency. The energy threshold and frequency threshold are set. When the energy and frequency of the audio signal in a frame or each time period exceed the corresponding thresholds, the audio signal in this frame or this time period is considered to have sound activity, otherwise the audio signal in this frame or this time period is considered to be a silent segment.

[0061] In this way, the parts with sound activity can be screened out, thereby reducing the subsequent processing of silent segments and improving the efficiency of the entire analysis. The activity of the audio is judged based on the energy, frequency and other characteristics of the sound, and the parts with sound activity are screened out, reducing the processing of silent segments and improving efficiency.

[0062] 4.3. Using the speech recognition algorithm-related model, perform feature extraction and pattern matching on the extracted voiced segments to distinguish between human voice and non-human voice in the voiced segments.

[0063] The model analyzes the more complex features of the sound in the audio clip, such as the spectrum, rhythm, and color, and compares and analyzes them with the pre-learned human and non-human voice patterns, thereby accurately distinguishing the human and non-human voice parts in the audio clip. For example, it distinguishes non-human voice parts such as background music and environmental noise from the actual speaking voice.

[0064] 4.4. For the segments judged as human voices, accurately record their start and end times in the audio file.

[0065] 5. Effective video clip cropping

[0066] The video file is cropped based on the start and end time of the intersection of the start and end time of the segment where the face is located in the video file and the start and end time of the segment where the voice is located in the audio file to obtain a valid video segment.

[0067] In this embodiment, the start and end time of the human voice and the effective start and end time of face detection are intersected, and the intersection of each segment is taken to obtain the start and end time of the intersection, and the video is trimmed according to the start and end time to form valid video clips.

[0068] like Figure 4 As shown, the specific steps include the following:

[0069] 5.1. For the start time and end time of the segment containing the face extracted in the second part and the start time and end time of the segment containing the voice extracted in the fourth part, compare and calculate each group of corresponding times to determine the intersection of the times corresponding to the segment containing the face and the segment containing the voice.

[0070] 5.2. According to the start time and end time of the intersection, use video editing tools or related programming interfaces to crop the video file.

[0071] 5.3. Repeat the above process to cut the video file into multiple valid video clips.

[0072] 6. Video transcoding

[0073] Transcode the valid video clips into mp4 format to obtain the final video.

[0074] In this embodiment, a data processing system for digital human model training is provided, and the system can implement the above-mentioned method. The system includes:

[0075] (1) Video data acquisition module: select high-definition live-action videos of broadcasting in the digital human material library and save these videos as video files;

[0076] (2) Face detection module: performs face detection on each frame of the video file. For an image with only one face, the start and end time of the segment containing the face in the video file are recorded.

[0077] (3) Audio track separation module: extract the audio content from the video file separately and save the audio content as a wav format audio file;

[0078] (4) Audio detection module: performs feature extraction and pattern matching on the audio signal of each frame or each time period in the audio file, determines whether the audio signal of the frame or the time period is a sound segment or a silent segment, and distinguishes between human voice and non-human voice in the sound segment. For the human voice segment, records the start time and end time of the human voice segment in the audio file;

[0079] (5) Valid video segment cropping module: based on the start and end time of the intersection between the start and end time of the segment where the face is located in the video file and the start and end time of the segment where the voice is located in the audio file, the video file is cropped to obtain valid video segments;

[0080] (6) Video transcoding module: transcode the valid video clips into mp4 format to obtain the final video.

[0081] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0082] The present invention provides a data processing method and system for digital human model training, which can optimize the quality of digital human training data, improve the fidelity of digital human performance, and enhance the adaptability and versatility of digital human. By accurately screening and cutting out effective segments containing clear human faces and accurate human voices, high-quality data is provided for model training, ensuring the synchronization and matching of human faces and human voices in the data, making the simulation of digital humans in facial expressions, language expressions, etc. more accurate and coordinated, and improving the coherence of overall performance. Irrelevant and redundant data are removed, the computational workload and time cost of model training are reduced, and the training speed is accelerated.

[0083] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A data processing method for digital human model training, characterized by: The following steps are included: S1. Video data collection: select high-definition live-action videos of broadcasting in the digital human material library and save these videos as video files; S2. Face detection: Perform face detection on each frame of the video file. For an image with only one face, record the start time and end time of the segment where the face is located in the video file. S3, audio track separation: extract the audio content from the video file separately and save the audio content as a wav format audio file; S4, audio detection: feature extraction and pattern matching are performed on the audio signal of each frame or each time period in the audio file to determine whether the audio signal of the frame or the time period is a sound segment or a silent segment, and to distinguish between human voice and non-human voice in the sound segment. For the human voice segment, the start time and end time of the human voice segment in the audio file are recorded; S5. Cropping of effective video segments: based on the start and end time of the intersection of the start and end time of the segment containing the face in the video file and the start and end time of the segment containing the voice in the audio file, the video file is cropped to obtain effective video segments.

2. The data processing method for digital human model training according to claim 1, characterized in that: Step S2 specifically includes the following contents: S21, reading the saved video file, and analyzing and processing each frame of the video file; S22. Use a deep learning-based face detection model to determine whether there is one and only one face in the current frame image. If so, record the start time and end time of the segment containing the face in the video file; otherwise, skip the current frame image.

3. The data processing method for digital human model training according to claim 1, characterized in that: Step S4 specifically includes the following contents: S41, reading the audio file to be analyzed; S42, using audio activity detection technology to extract features from the audio signal of each frame or each time period of the audio file, and determining whether the audio signal of the frame or the time period is a sound segment or a silent segment based on a corresponding threshold; S43, using a speech recognition algorithm-related model to perform feature extraction and pattern matching on the extracted voiced segment to distinguish between human voice and non-human voice in the voiced segment; S44. For the segment determined to be a human voice, accurately record its start time and end time in the audio file.

4. The data processing method for digital human model training according to claim 3 is characterized in that: Step S42 specifically includes, for the energy characteristics of the sound, calculating the sum of the squares of the amplitudes of the audio signal of each frame or each time period to indicate its energy; for the frequency characteristics of the sound, analyzing the distribution of the audio signal of each frame or each time period in different frequency bands to indicate its frequency; setting the energy threshold and the frequency threshold. When the energy and frequency of the audio signal of a frame or each time period exceed the corresponding thresholds, the audio signal of the frame or the time period is considered to have sound activity, otherwise the audio signal of the frame or the time period is considered to be a silent segment.

5. The data processing method for digital human model training according to claim 3, characterized in that: Step S42 specifically analyzes the spectrum, rhythm, and timbre features of the sound in the audio segment, compares and analyzes them with the pre-learned human voice and non-human voice patterns, and then distinguishes the human voice and non-human voice in the audio segment.

6. The data processing method for digital human model training according to claim 1, characterized in that: Step S5 specifically includes the following contents: S51, comparing and calculating the start time and end time of the segment containing the face extracted in step S2 and the start time and end time of the segment containing the voice extracted in step S4, and determining the intersection of the time corresponding to the segment containing the face and the segment containing the voice; S52, using a video editing tool or a related programming interface to perform a cropping operation on the video file according to the start time and the end time of the intersection; S53: Repeat the above process to cut the video file into multiple valid video clips.

7. The data processing method for digital human model training according to claim 1, characterized in that: The resolution of the high-definition live video is above 1080*1920.

8. The data processing method for digital human model training according to claim 1, characterized in that: Step S5 also includes: S6. Video transcoding: transcode the valid video clips into mp4 format to obtain the final video.

9. A data processing system for digital human model training, characterized by: The system can implement the method described in any one of claims 1 to 8, and the system includes: Video data acquisition module: selects high-definition live broadcast videos in the digital human material library and saves these videos as video files; Face detection module: performs face detection on each frame of the video file. For images with only one face, the start and end time of the segment containing the face in the video file are recorded. Audio track separation module: extract the audio content from the video file separately and save it as a wav format audio file; Audio detection module: extract features and perform pattern matching on the audio signal of each frame or each time period in the audio file, determine whether the audio signal of the frame or the time period is a sound segment or a silent segment, and distinguish between human voice and non-human voice in the sound segment. For the human voice segment, record the start time and end time of the human voice segment in the audio file; Valid video segment cropping module: based on the start and end time of the intersection between the start and end time of the segment where the face is located in the video file and the start and end time of the segment where the voice is located in the audio file, the video file is cropped to obtain valid video segments; Video transcoding module: uniformly transcodes valid video clips into mp4 format to obtain the final video.

Images