Data preparation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data preparation method and device, electronic equipment and a storage medium. The method comprises the steps that an input video is cut into at least one short video only containing a single sentence; performing face detection on the short video, and identifying character information in the short video; matching the voice information corresponding to the short video with the character information, and determining a pronunciation character corresponding to the short video; intercepting a face area of a pronunciation character from the short video and storing the face area into a first data set; the target training data set is determined according to the first data set, the problem that data preparation of a current mainstream scheme completely depends on an output result of an audio and video synchronization network, and consequently the reliability and robustness of a data preparation method are low is solved, and then the data preparation efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of digital human photography, and more particularly to a data preparation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the advancement of information technology and the development of mobile internet technology, digital humans based on photographs are gradually becoming a hot topic. Although digital humans have been widely used in various fields, training digital human models has always been a challenge, especially the preparation of data used for training these models.

[0003] The current mainstream data preparation method is based on deep learning-trained audio-video synchronization neural networks to directly determine the synchronization of video and audio data, and saves the data that meets the synchronization requirements as a training dataset. This approach has advantages such as high processing efficiency, simplicity, and no need for additional development. However, current solutions do not consider the audio-video synchronization problem in scenarios with multiple faces. Furthermore, most videos typically contain multiple sentences, and it is difficult to guarantee that the lip movements of the characters are naturally closed during pauses between sentences. This inevitably affects the synchronization network's judgment, resulting in extremely low data utilization and increasing the difficulty of data preparation. In addition, the current mainstream approach relies entirely on the output of the audio-video synchronization network, leading to low reliability and robustness of the data preparation method, thus reducing the efficiency of data preparation. Summary of the Invention

[0004] This invention provides a data preparation method, apparatus, electronic device, and storage medium, which can solve the problem in related technologies that rely entirely on the output of audio and video synchronization networks, resulting in low reliability and robustness of data preparation methods and reduced efficiency.

[0005] On one hand, embodiments of the present invention disclose a data preparation method, the method comprising:

[0006] Cut the input video into at least one short video containing only a single sentence;

[0007] Perform face detection on the short video to identify the people in the short video;

[0008] The audio information corresponding to the short video is matched with the person information to determine the speaker of the short video;

[0009] Extract the facial region of the person making the speech from the short video and store it in the first dataset;

[0010] The target training dataset is determined based on the first dataset.

[0011] Optionally, the step of performing face detection on the short video and identifying the people in the short video includes:

[0012] Using the facial information of the target object in the first frame of the short video as the reference face, the facial information of the target object in the remaining frames of the short video is identified;

[0013] If the facial information of the target object is not found in any frame of the short video, the detection of the short video is stopped, and the next short video is traversed.

[0014] Optionally, the step of identifying the facial information of the target object in the remaining frames of the short video, based on the facial information of the target object in the first frame of the short video, includes:

[0015] The rectangles corresponding to each detected face in the remaining frames of the short video are compared with the reference face in the first frame to calculate the intersection and union ratio, and the calculation result is obtained.

[0016] If the calculation result is greater than a preset threshold, the detected face is identified as the face information of the target object;

[0017] If the calculation results for each detected face in the second frame are all less than or equal to the preset threshold, it is determined that the matching of the person information corresponding to the second frame and the first frame has failed, and the matching of the short video is stopped; the second frame is any frame in the short video other than the first frame.

[0018] Optionally, the person information includes at least one candidate face; the step of matching the audio information corresponding to the short video with the person information to determine the speaker corresponding to the short video includes:

[0019] The facial mouth feature points of each candidate face in the aforementioned person information are statistically analyzed to obtain statistical results;

[0020] The statistical results are normalized, and the variance corresponding to each candidate face is determined based on the normalized statistical results.

[0021] Based on a first threshold and the variance corresponding to each candidate face, the speaker corresponding to the short video is determined from the at least one candidate face.

[0022] Optionally, determining the speaker corresponding to the short video from the at least one candidate face based on the first threshold and the variance corresponding to each candidate face includes:

[0023] If there exists a candidate face whose variance is greater than or equal to the first threshold, then the candidate face is identified as the voice-over character corresponding to the short video.

[0024] If there are at least two candidate faces whose variance is greater than or equal to the first threshold, then the candidate face with the largest variance is determined as the voice of the person in the short video.

[0025] If there exists a candidate face whose variance is less than the first threshold, stop the first threshold determination of the short video and start iterating through the next short video.

[0026] Optionally, the method further includes:

[0027] Establish the correspondence between lip-shape features and speech features through an audio-video synchronization network;

[0028] Having obtained the speech features of the speaker, the lip-shape features are traversed based on the correspondence and the speech features;

[0029] When the lip-shape features match the speech features, it is determined that the speech and lip-shape of the speaker are synchronized.

[0030] Optionally, the step of cutting the input video into at least one short video containing only a single sentence includes:

[0031] Standardize the parameters of the input video;

[0032] The input video with unified parameters is segmented into sentences to obtain at least one short video containing only a single sentence.

[0033] Optionally, determining the target training dataset based on the first dataset includes:

[0034] Obtain the confidence level corresponding to each data point in the first dataset;

[0035] If all confidence levels are greater than or equal to the second threshold, the first dataset is determined to be the target training dataset.

[0036] On the other hand, embodiments of the present invention disclose a data preparation apparatus, the apparatus comprising:

[0037] The segmentation module is used to segment the input video into at least one short video containing only a single sentence;

[0038] The recognition module is used to perform face detection on the short video and identify the people in the short video;

[0039] The determination module is used to match the audio information corresponding to the short video with the character information to determine the speaker corresponding to the short video;

[0040] A storage module is used to extract the facial region of the person making the speech from the short video and store it in a first dataset;

[0041] The determining module is further configured to determine the target training dataset based on the first dataset.

[0042] Optionally, the identification module includes:

[0043] The recognition submodule is used to recognize the facial information of the target object in the remaining frames of the short video, based on the facial information of the target object in the first frame of the short video.

[0044] The traversal module is used to stop the detection of the short video and traverse the next short video if the facial information of the target object is not found in any frame of the short video.

[0045] Optionally, the identification submodule includes:

[0046] The module is used to perform intersection-union calculation on the bounding boxes corresponding to each detected face in the remaining frames of the short video and the reference face in the first frame to obtain the calculation result;

[0047] The first determining submodule is used to determine the detected face as the face information of the target object when the calculation result is greater than a preset threshold; and to stop the matching of the short video when the calculation results corresponding to each detected face in the second frame are all less than or equal to the preset threshold; the second frame is any frame in the short video other than the first frame.

[0048] Optionally, the determining module includes:

[0049] The acquisition module is used to statistically analyze the facial mouth feature points of each candidate face in the person information and obtain the statistical results.

[0050] The second determining submodule is used to normalize the statistical results and determine the variance corresponding to each candidate face based on the normalized statistical results; and to determine the voice of the person speaking in the short video from the at least one candidate face based on the first threshold and the variance corresponding to each candidate face.

[0051] Optionally, the second determining submodule includes:

[0052] The third determining submodule is used to determine the candidate face as the voice of the short video if the variance of a candidate face is greater than or equal to the first threshold; and to determine the candidate face with the largest variance as the voice of the short video if the variance of at least two candidate faces is greater than or equal to the first threshold.

[0053] The traversal submodule is used to stop the first threshold judgment of the short video and traverse the next short video if there is a candidate face whose variance is less than the first threshold.

[0054] Optionally, the device further includes:

[0055] A module is established to create a correspondence between lip-shape features and speech features through an audio-video synchronization network.

[0056] The first traversal submodule is used to traverse lip shape features based on the correspondence and the voice features when the voice features of the speaker are obtained;

[0057] The fourth determining submodule is used to determine the synchronization of the speaker's speech and lip movements when the lip-shape features match the speech features.

[0058] Optionally, the cutting module includes:

[0059] The unified module is used to unify the parameters of the input video.

[0060] The submodule is used to segment the input video after parameter unification into sentences, and obtain at least one short video containing only a single sentence.

[0061] Optionally, the determining module further includes:

[0062] The acquisition submodule is used to acquire the confidence level corresponding to each data point in the first dataset.

[0063] The fifth determining submodule is used to determine the first dataset as the target training dataset when the confidence levels are all greater than or equal to the second threshold.

[0064] In another aspect, embodiments of the present invention also disclose an electronic device, which includes a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store executable instructions, which cause the processor to execute the aforementioned data preparation method.

[0065] This invention also discloses a readable storage medium, which, when the instructions in the readable storage medium are executed by the processor of an electronic device, enables the electronic device to perform the aforementioned data preparation method.

[0066] The embodiments of the present invention have the following advantages:

[0067] This invention provides a data preparation method that involves segmenting an input video into at least one short video containing only a single sentence; performing face detection on the short video to identify the person in the video; matching the audio information corresponding to the short video with the person information to determine the speaker; extracting the facial region of the speaker from the short video and storing it in a first dataset; and determining a target training dataset based on the first dataset. This invention first segments the input video into short videos containing only a single sentence, thus solving the problem of long videos. Then, it performs face detection on the short videos to identify the person in the video; furthermore, it matches the audio information corresponding to the short video with the person information to determine the speaker; finally, it extracts the facial region of the speaker from the short video and stores it in the first dataset. This method aligns facial regions and removes irrelevant image regions, ensuring the size consistency of all facial images. The target training dataset determined based on the first dataset improves data output, increases the robustness and accuracy of data preparation, and enhances the efficiency of data preparation. Attached Figure Description

[0068] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0069] Figure 1 This is a step flow of an embodiment of the data preparation method of the present invention. Figure 1 ;

[0070] Figure 2 This is a step flow of an embodiment of the data preparation method of the present invention. Figure 2 ;

[0071] Figure 3 This is a step flow of an embodiment of the data preparation method of the present invention. Figure 3 ;

[0072] Figure 4 This is a structural block diagram of an embodiment of the data preparation apparatus of the present invention;

[0073] Figure 5 This is a structural block diagram of an electronic device provided by an example of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, the first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In embodiments of this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0076] Method Implementation Examples

[0077] Reference Figure 1 The flowchart illustrates the steps of an embodiment of the data preparation method of the present invention. Figure 1 The method may specifically include the following steps:

[0078] Step S101: Cut the input video into at least one short video containing only a single sentence;

[0079] Step S102: Perform face detection on the short video and identify the people in the short video;

[0080] Step S103: Match the audio information corresponding to the short video with the person information to determine the person speaking in the short video;

[0081] Step S104: Extract the facial region of the person making the sound from the short video and store it in the first dataset;

[0082] Step S105: Determine the target training dataset based on the first dataset.

[0083] The input video is cut into at least one short video containing only a single sentence.

[0084] In the automated preparation of training data for digital humans from photos, specific requirements exist for the input video format. Common video formats such as MP4 are frequently chosen due to their efficient encoding methods and broad compatibility. They can play smoothly on various devices and platforms; both computer-based video editing software and mobile devices such as phones and tablets can easily recognize and process MP4 videos. MP4 also offers advantages in data transmission and storage. For example, in some cloud storage services, MP4 videos can be uploaded and downloaded more efficiently, facilitating the dataset preparation process.

[0085] In addition, the MOV format is also frequently used. It performs excellently in color reproduction and image quality, making it particularly suitable for scenarios requiring high-quality image data for digital human datasets. During video shooting and post-processing, the MOV format better integrates with the entire workflow, ensuring high-quality video input and laying the foundation for accurate facial information extraction and the generation of high-quality digital human data.

[0086] The input video is split into multiple short videos, and each short video contains only a single sentence.

[0087] For example, by segmenting the input video "A: Spring sleep is unaware of dawn, B: Birds are singing everywhere, C: The sound of wind and rain at night, D: How many flowers have fallen?", we can get "A: Spring sleep is unaware of dawn", "B: Birds are singing everywhere", "C: The sound of wind and rain at night", and "D: How many flowers have fallen?". That is, each segmented short video contains only a single sentence.

[0088] Perform face detection on short videos to identify the people in the videos.

[0089] In the preparation of digital human datasets based on photographs, face detection refers to the process of automatically identifying and locating facial regions in input image or video data using computer vision technology. It can accurately pinpoint the location of faces from massive amounts of image information, separating them from the background and other irrelevant objects, providing a clear target for subsequent analysis and processing of digital human data. Whether it's a single still photograph or consecutive video frames, face detection can quickly and accurately complete the task of face recognition and localization.

[0090] For example, face detection typically relies on specific algorithms and models. These algorithms and models are trained on large amounts of face data to learn various facial features, such as contours, facial feature distribution, and texture. When input image or video data, the algorithms and models analyze the data pixel-by-pixel or region-by-region, using methods such as pattern matching and feature extraction to determine whether each region contains a face. Once a face is detected, the algorithm outputs the face's location information, such as the coordinates and size of the bounding box, thus completely labeling the face region and providing fundamental and crucial data information for the dataset.

[0091] Furthermore, face detection plays a crucial role in the preparation of photo-based digital human datasets. It forms the foundation for a series of subsequent data processing steps. Only by accurately detecting faces can we further analyze and extract facial expressions, poses, and feature points, thereby constructing a high-quality digital human dataset. If face detection errors or omissions occur, subsequent training and generation of digital humans based on this data will be severely affected, leading to inaccurate digital human images and unnatural expressions. Therefore, accurate face detection plays a decisive role in ensuring the quality and usability of photo-based digital human datasets.

[0092] Information about people in short videos can include the specific location of a face within an image or video frame, precisely identified by the coordinates of a bounding box (such as the coordinates of the top-left corner). The size of the bounding box can be obtained through face detection, determining the range of face size. This bounding box provides the basis for operations such as face cropping and scaling, enabling faces to be accurately separated from complex backgrounds for individual processing and analysis.

[0093] Given visual data, face detection can definitively determine the presence of a face. This result is crucial in many scenarios, such as surveillance systems determining whether someone is present in the frame, and smart devices deciding whether to activate face-related functional modules, thus avoiding unnecessary waste of computing resources.

[0094] During the detection process, the algorithm identifies some basic facial features, such as the general outline of facial features and the orientation of the face. Although these features are relatively simple, they form the basis for more in-depth analysis. Detailed facial features, such as the precise location and shape of organs like the eyes, nose, and mouth, can then be extracted for tasks like facial recognition and expression analysis.

[0095] The audio information corresponding to the short video is matched with the person's information to determine the person who speaks in the short video.

[0096] The audio information associated with the short video can include the speech of any person in the video. For example, a person in the short video might introduce themselves, explain knowledge, or tell a story. This audio content is set according to the needs of data preparation and the application scenario.

[0097] For example, the voice information can also include the emotional tone and stylistic characteristics of the characters' voices in the short video. Emotionally, it may express different feelings such as happiness, sadness, calmness, and excitement to suit different scenarios and content needs. Stylistically, it can be formal, friendly, humorous, serious, etc. Alternatively, the voice information can also include pitch, timbre, speech rate, and intonation.

[0098] For example, the voice actor in a short video refers to the individual who provides voice expression in the video. For instance, it can be any person who, through an infectious voice and unique tone, incorporates their own emotions and opinions into the lines, giving the video a vivid personality.

[0099] Extract the facial region of the speaker from the short video and store it in the first dataset.

[0100] After identifying the speaker, it is necessary to extract the speaker's facial region. Based on the face detection results, the facial region needs to be aligned. This serves two purposes: firstly, it can eliminate irrelevant image areas and ensure the size consistency of all facial images; secondly, it can prepare data for subsequent audio-visual synchronization networks.

[0101] It's important to note that the purpose of cropping the facial region of the speaker is to obtain a valid lip-sync (only the head region containing the lip-sync needs to be included) and a matching segment with the speech. Once a match is successful, the matched head region needs to be cropped again and saved. This operation reduces memory requirements in subsequent processing (the cropped facial image is smaller) and also clearly identifies the face that matches the speech (especially when there are multiple faces in the video segment; by cropping the successfully matched face and deleting irrelevant faces, subsequent processing is simplified).

[0102] The first dataset may include facial regions of the speaker.

[0103] For example, the facial area of ​​the speaker may include the part in front of the speaker's head, from the hairline to the chin, including the area where various facial organs such as eyes, eyebrows, nose, mouth, cheeks, and ears are located.

[0104] For example, in studies on the correlation between speech and facial expressions, muscle movements and facial expression changes in the facial region can help understand the emotional state and prosodic features of speech. In some vision-based speech recognition systems, visual information from the facial region, such as lip movements and mouth shape changes, can also serve as supplementary cues to improve the accuracy of speech recognition. Furthermore, in fields such as animation production and digital human development, accurately simulating the movements and expressions of the facial region of a speaking character can make the speech of virtual characters more natural and vivid, enhancing the user's immersion and sense of realism.

[0105] The target training dataset is determined based on the first dataset.

[0106] The target training dataset can include image data, audio data, text data, and labeled data. For example, image data is the core of the dataset, containing a large number of photos of people from different angles, with different expressions, and under different lighting conditions. These photos are used to allow the model to learn the physical features, facial structure, and expression changes of people, so that the digital human can accurately present a realistic human image.

[0107] Speech data corresponds to image data and includes various speech content, such as different sentences, intonations, and speaking speeds. Speech data is used to train digital humans to generate natural and fluent speech based on input text, and to match speech with corresponding lip movements and facial expressions.

[0108] Text data can include descriptive information about people, dialogue content, and transcripts of speech. Text data provides the foundation for semantic understanding and language generation in models, helping digital humans understand user input and generate appropriate responses.

[0109] Annotated data allows models to better learn and understand data, and datasets contain various annotations. For example, facial feature points in images are labeled, indicating the location of parts such as eyes, nose, and mouth; speech data is labeled with attributes such as emotion category and speech rate; and text data is labeled with its grammatical structure and semantic category. This annotation information helps models learn and capture patterns in the data more accurately.

[0110] In an optional embodiment of the present invention, the step of performing face detection on short videos and identifying the information of people in the short videos may specifically include the following steps:

[0111] Step S1021: Using the facial information of the target object in the first frame of the short video as the reference face, identify the facial information of the target object in the remaining frames of the short video;

[0112] Step S1022: If the target object's facial information is not found in any frame of the short video, stop the detection of the short video and traverse the next short video.

[0113] Using the facial information of the target object in the first frame of a short video as the baseline, the facial information of the target object in the remaining frames of the short video is identified.

[0114] The facial information of the target object in the first frame may include facial feature point coordinates, face position and size, facial attribute information, skin color information, etc.

[0115] For example, facial feature point coordinates are a crucial part of facial information. By detecting the coordinates of key facial features, such as the eyes, eyebrows, nose, mouth, and chin, the shape and structure of the face can be accurately located and described. Generally, the number of commonly used feature points may range from dozens to hundreds. These feature points can outline the contours of the face and the relative positions of facial organs in detail, providing basic data for subsequent tasks such as face recognition and expression analysis.

[0116] For example, the position and size of a face include its specific location within an image or video frame, typically represented by a rectangular or other bounding box. The coordinates of the top-left and bottom-right corners (or the coordinates of the center point and its width and height) determine the face's position and the size of the area it occupies in the image. This is crucial for locating faces in complex scenes and for subsequent processing such as scale normalization based on face size.

[0117] For example, facial attribute information includes gender, age, expression, and posture. For instance, by analyzing the overall features, facial contours, and proportions of the face, the gender of the face can be determined, typically identified as male or female. Alternatively, the age range of the face can be estimated based on features such as facial texture, skin condition, and the degree of development of facial features. This could be a specific age value or an age group, such as child, teenager, adult, or elderly. Or, facial expressions can be analyzed, such as smiling, anger, sadness, or surprise. This is achieved by detecting the movements and muscle changes of the facial features; for example, the upward or downward tilt of the corners of the mouth, the degree of eye opening, and the furrowing or relaxing of the eyebrows can all serve as criteria for judging expressions. Alternatively, the orientation and angle of the head, including left and right rotation, up and down tilting, and lateral tilting, can be used to describe the posture changes of the face in space, which is crucial for subsequent posture normalization and multi-angle face recognition.

[0118] For example, skin color can be represented by numerical values ​​in a color space. Typically, the image is converted from the common RGB color space to other color spaces that are more suitable for skin color analysis, such as the YCbCr space. Then, statistical information such as the mean and variance of skin color is extracted to assist in face detection and recognition. In some complex backgrounds, skin color information can help to quickly locate the face region.

[0119] A reference face refers to a representative face used to provide a standard for comparison. For example, in a face recognition system, a reference face can be used as a template to compare with the face to be identified, and the similarity between the two is calculated to determine whether they are the same person. In tasks such as facial expression analysis and pose estimation, reference faces can also be used to define standard facial expression states or neutral poses in order to more accurately analyze and describe the expression changes and pose differences of other faces.

[0120] The facial information of the target object in the remaining frames of the short video can include the outline shape of the face, the precise position of the facial features and their proportional relationship, such as the distance between the eyes, the width of the nose, and the outline of the lips. This information can be obtained by detecting the coordinates of key facial feature points and can intuitively reflect the basic structural features of the face.

[0121] Alternatively, facial information can also include visible elements such as skin tone, skin texture, hair color and style, and facial adornments (such as glasses and jewelry). Skin tone can be extracted using color space transformation to obtain relevant numerical descriptions, and details such as the smoothness of skin texture and the presence or absence of wrinkles or blemishes can also serve as important information. These appearance features help distinguish different individuals and play a crucial role in scenarios such as facial recognition and image retrieval.

[0122] In addition, facial information also involves dynamic behavior and state characteristics. Facial expressions are an important part of this, such as facial muscle movements and changes in facial features caused by expressions like smiling, frowning, and surprise; head posture, including angle information such as left and right turning, tilting, and lateral tilting; and eye state, such as open eyes, closed eyes, and gaze direction. This dynamic information provides rich evidence for analyzing the emotional state and behavioral intentions of the target.

[0123] If the target object's facial information is not found in any frame of the short video, stop detecting the short video and start traversing the next short video.

[0124] If no facial information is detected, the short video being detected is considered to be non-compliant, and the detection of the short video is stopped.

[0125] If no facial information is detected in the previous short video, the detection of the previous short video can be stopped, and the next short video can be traversed directly.

[0126] In an optional embodiment of the present invention, the step of identifying the facial information of the target object in the remaining frames of the short video based on the facial information of the target object in the first frame of the short video may specifically include the following steps:

[0127] Step S10211: Perform intersection and union calculation on the bounding boxes corresponding to each detected face in the remaining frames of the short video and the reference face in the first frame to obtain the calculation result;

[0128] Step S10212: If the calculation result is greater than the preset threshold, the detected face is identified as the face information of the target object;

[0129] Step S10213: If the calculation results corresponding to each detected face in the second frame are all less than or equal to the preset threshold, it is determined that the matching of the person information corresponding to the second frame and the first frame has failed, and the matching of the short video is stopped; the second frame is any frame in the short video other than the first frame.

[0130] The cross-intersection over union (CUI) of the bounding boxes corresponding to each detected face in the remaining frames of the short video with the reference face in the first frame is calculated to obtain the result.

[0131] The bounding box corresponding to the detected face is a geometric shape consisting of four sides that completely encloses the face region. Its main function is to clearly indicate the specific location of the face in the image, enabling the computer to accurately identify and locate the face. It also provides a clear area for subsequent face analysis tasks, such as face recognition and face feature extraction, avoiding unnecessary processing of non-face areas.

[0132] For example, a rectangle is typically represented by its top-left corner coordinates (x, y) along with its width and height. For instance, a rectangle with its top-left corner at (100, 100), a width of 200 pixels, and a height of 300 pixels represents the area in the image where the face is located, defined by the x-coordinate from 100 to 300 (100+200) and the y-coordinate from 100 to 400 (100+300). In some applications, other representations may be used, such as the center point coordinates of the rectangle plus its width and height, but essentially, all these methods aim to accurately describe the rectangle's position and size.

[0133] It's worth noting that the bounding boxes used to detect faces have wide applications in various face detection-related fields. In security monitoring, by detecting faces and drawing bounding boxes, it's possible to quickly locate people's faces in the monitored area, enabling functions such as face recognition access control and personnel tracking. In photography and image processing software, face detection bounding boxes can be used to assist in focusing and automatic composition, helping users better capture portraits or perform specific image processing operations on faces. In human-computer interaction systems, the position and changes of the face bounding boxes can be used to implement face-based interactive functions, such as adjusting the position of interface elements based on face movement.

[0134] The bounding box corresponding to the reference face can determine the region where the reference face is located, providing a foundation for a series of operations related to the reference face. In tasks such as face detection and recognition, the bounding box of the face to be detected or recognized can be compared with the bounding box of the reference face. By analyzing the relationship between their positions, sizes, angles, etc., the face can be standardized and compared. For example, by calculating the positional offset between the bounding box of the face to be detected and the bounding box of the reference face, the face can be calibrated to make its position in the image more consistent with the standard, facilitating subsequent feature extraction and analysis.

[0135] In terms of representation, similar to general face detection bounding boxes, the coordinates (x, y) of the top-left corner of the rectangle, as well as its width and height, are often used to determine its specific position and size. For example, if the top-left corner of the baseline face bounding box is (50, 50), the width is 150 pixels, and the height is 200 pixels, then the baseline face lies within the region of the image with an x-coordinate from 50 to 200 (50+150) and a y-coordinate from 50 to 250 (50+200).

[0136] For example, in face modeling, a standard face model is constructed based on the face image data within the reference face rectangle; when evaluating expression analysis algorithms, the expression within the reference face rectangle is used as a neutral or standard expression reference to compare the expression changes of other test faces; in the face recognition system, the position and size of the face in the input image are standardized by using the reference face rectangle to improve the accuracy and stability of recognition.

[0137] Intersection over Union (IoU) calculation can be used to evaluate the degree of overlap between detected face bounding boxes and real face bounding boxes.

[0138] For example, the intersection-union ratio (IU) is the ratio of the area of ​​the intersection of the detection bounding boxes (the rectangles corresponding to each detected face in the remaining frames) and the ground truth bounding boxes (the reference face in the first frame) to the area of ​​their union. It quantitatively reflects the degree of matching between two frames. The higher the value, the more accurate the detection result, and the more likely the detected face is the face information of the target object.

[0139] For example, when training a face detection model, the intersection-union ratio (IoU) is one of the key metrics for evaluating model performance. By calculating the IoU of the detected bounding boxes and ground truth boxes in a large number of test samples and obtaining statistics such as the mean (mIoU), one can intuitively understand the model's accuracy in detecting face locations, helping developers determine whether the model is overfitting or underfitting, and thus adjust the model's parameters and structure.

[0140] For example, when different face detection algorithms are evaluated on the same dataset, the intersection-over-union ratio (IoU) can serve as an objective metric for comparing the performance of different algorithms. Generally, algorithms with higher IoU perform better and have greater advantages in practical applications.

[0141] For example, the IoU calculation process includes the face detection bounding box results of the previous frame (which may be multiple face boxes) and the face detection results of the current frame (which may also be multiple face boxes). The purpose of performing face IoU determination on the two frames is to ensure the continuity and stability of the person corresponding to each face box in the current video, and to ensure as much as possible that each face box is the same person's face from beginning to end (for example, if in a video clip, the first part corresponds to the speaker's face, but the second part corresponds to the face of another silent person, and each frame of the video contains a face, without IoU determination, the current video may be judged as a valid video clip, because each frame contains a face, and because the first part is the speaker's face, whose mouth is in motion, although the second part introduces a silent face, the mean and variance of the mouth keypoints calculated for the entire video clip may meet the confidence requirements, causing an incorrect video to be adopted).

[0142] Furthermore, in practical face detection applications, multiple possible face detection boxes may be obtained. By setting an intersection-union ratio (IU) threshold, detection boxes with an IU greater than the threshold are retained, while those with an IU that is too low are removed, thereby obtaining more accurate and reliable face detection results and reducing false positives and false negatives.

[0143] The calculation result refers to the ratio of the intersection area to the union area of ​​the detection box (the rectangle corresponding to each detected face in the remaining frames) and the ground truth box (the reference face in the first frame).

[0144] If the calculated result is greater than the preset threshold, the detected face will be identified as the face information of the target object.

[0145] During face detection, the preset threshold can be a pre-set standard value. If the value exceeds the preset threshold, the detected face will be considered as the face information of the target object.

[0146] If the calculation results for each detected face in the second frame are all less than or equal to the preset threshold, it is determined that the matching of the person information in the second frame with the first frame has failed, and the matching of the short video is stopped; the second frame is any frame in the short video other than the first frame.

[0147] For example, if the calculation result does not exceed the preset threshold, it will be considered that the matching of the person information corresponding to the second frame and the first frame has failed, that is, the person between the two frames has changed and is not the same person, and the matching of the short video can be stopped; the second frame is any frame in the short video other than the first frame.

[0148] In an optional embodiment of the present invention, the person information includes at least one candidate face; the step of matching the audio information corresponding to the short video with the person information to determine the speaker corresponding to the short video may specifically include the following steps:

[0149] Step S1031: Statistical analysis of facial mouth feature points for each candidate face in the person information to obtain statistical results;

[0150] Step S1032: Normalize the statistical results and determine the variance corresponding to each candidate face based on the normalized statistical results.

[0151] Step S1033: Based on the first threshold and the variance corresponding to each candidate face, determine the voice of the person in the short video from at least one candidate face.

[0152] The facial features of each candidate in the information were statistically analyzed, and the statistical results were obtained.

[0153] The statistical process of facial mouth feature points refers to the process of locating, extracting, and analyzing key facial mouth feature points using specific technologies and methods. It mainly includes steps such as data collection, feature point location, feature extraction, and statistical analysis.

[0154] For example, the first step is to collect a large amount of facial image data. These images should have different expressions, poses, lighting conditions, etc., to ensure that various variations in mouth features are covered. The data source can be a publicly available facial database or images obtained through self-taken photos. For example, images containing various facial expressions can be downloaded from the internet, or photos of different people's faces from various angles, such as front and side views, can be taken using a camera.

[0155] Secondly, face detection algorithms are used to determine the location and approximate extent of faces in the image. Then, based on this, specialized mouth feature point localization algorithms are used to precisely locate the various feature points of the mouth. These feature points typically include the corners of the mouth, key points on the lip contour, and the center of the lips. Common localization methods include machine learning-based algorithms, such as convolutional neural networks (CNNs) in deep learning. These CNNs can learn from a large number of face images with labeled feature points and automatically extract features that accurately describe the location of mouth feature points, thus achieving precise localization. Other methods are based on traditional image processing techniques, such as edge detection and template matching, to find mouth feature points.

[0156] After locating the mouth feature points, it is necessary to extract the feature information associated with these points. These features can be geometric features, such as the angle of the corners of the mouth, the width and height of the lips, and the curvature of the lips; they can also be grayscale features, i.e., the grayscale value distribution of pixels around the feature point; and they can also be texture features, such as the texture information of the lip surface. By extracting these features, the characteristics of the mouth can be described more comprehensively. For example, the distance from the corner of the mouth to the center of the lips can be calculated as a geometric feature, or the Local Binary Pattern (LBP) features of the lip region can be extracted to represent texture information.

[0157] Finally, statistical analysis is performed on the extracted mouth feature points. This may include calculating statistical measures such as the mean, standard deviation, and distribution of the features to understand the variation patterns of mouth features across different individuals or expressions. For example, the mean and standard deviation of lip widths in different people can be calculated to observe whether their distribution conforms to a normal distribution; or the range and frequency of changes in mouth corner angles under different expressions can be analyzed. Through statistical analysis, some commonalities and characteristics of mouth feature points can be discovered, providing data support for subsequent applications, such as face recognition, expression recognition, and mouth movement analysis.

[0158] The statistical results are normalized, and the variance corresponding to each candidate face is determined based on the normalized statistical results.

[0159] The statistical results may include the mean and variance. For example, when smiling, the mean of the corners of the mouth moves upward and when angry, it moves downward, reflecting the trend of the expression, that is, the mean changes. Or when talking or laughing, the mouth opens, the distribution of feature points is more dispersed, and the variance increases significantly, that is, the variance increases. A lower variance value indicates that the mouth is closed or the range of motion is small.

[0160] The statistical results need to be normalized, that is, the coordinates are converted into proportional values ​​relative to the face bounding box (such as 0 to 1) to eliminate the influence of individual size differences.

[0161] For example, variance can be used to measure the dispersion of facial mouth feature point coordinates relative to the mean. A larger variance indicates more dispersed feature points, meaning these feature points fluctuate significantly relative to the mean; a smaller variance indicates that feature points are more concentrated around the mean, suggesting a relatively stable distribution. Taking feature points along the mouth width direction as an example, a large variance in the x-coordinates of these feature points means that the mouth width varies considerably across different face images, with some mouths being wider and others narrower; conversely, a smaller variance indicates that the mouth width is relatively consistent. Variance is calculated by first squared the difference between the coordinates of each feature point and the mean, then summing these squared values ​​and dividing by the number of feature points. It helps analyze the stability and range of variation of mouth feature points, and is of great significance for judging the diversity of mouth states in tasks such as face recognition and expression analysis.

[0162] For example, the mean refers to the average of the coordinate values ​​(usually x and y coordinates on a two-dimensional plane) of the feature points of the mouth. During calculation, the x-coordinates of all mouth feature points are summed and then divided by the number of feature points to obtain the mean x-coordinate; similarly, the y-coordinates of all mouth feature points are summed and divided by the number to obtain the mean y-coordinate. The mean reflects the average position of the mouth feature points in the image, representing the "centering" tendency of the mouth. For instance, by calculating the mean of mouth feature points in a large number of face images, the approximate position of the mouth within the overall facial structure can be determined, providing a baseline positional information for subsequent analysis and processing.

[0163] For example, by calculating and analyzing the mean and variance of facial mouth feature points, statistical features such as mouth position and shape changes can be obtained, providing valuable reference for further image processing, pattern recognition and other tasks.

[0164] Based on a first threshold and the variance corresponding to each candidate face, the speaker corresponding to the short video is determined from at least one candidate face.

[0165] The first threshold measures the dispersion of facial mouth feature points and compares them with the variance of each candidate face to identify the speaker in the short video from at least one candidate face.

[0166] In an optional embodiment of the present invention, determining the speaker corresponding to the short video from at least one candidate face based on a first threshold and the variance corresponding to each candidate face may specifically include the following steps:

[0167] Step S10331: If there exists a candidate face whose variance is greater than or equal to the first threshold, then the candidate face is identified as the voice of the person in the short video.

[0168] Step S10332: If there are at least two candidate faces whose variances are greater than or equal to the first threshold, then the candidate face with the largest variance is determined as the voice of the person in the short video.

[0169] Step S10333: If there is a candidate face whose variance is less than the first threshold, stop the first threshold judgment of the short video and traverse the next short video.

[0170] If there exists a candidate face whose variance is greater than or equal to the first threshold, then the candidate face is identified as the voice of the person in the short video.

[0171] For example, taking the feature points in the width direction of the mouth as an example, if the variance of the x-coordinate of these feature points is large, it means that the width of the mouth varies greatly in different face images, which means that the candidate face can be identified as the voice of the person in the short video.

[0172] If there are at least two candidate faces whose variance is greater than or equal to the first threshold, then the candidate face with the largest variance is identified as the voice of the person in the short video.

[0173] For example, if there are at least two candidate faces whose variance is greater than or equal to the first threshold, that is, if there are multiple candidates that meet the threshold condition, then the candidate face with the largest variance is identified as the voice of the person in the short video. In other words, taking the feature points in the width direction of the mouth as an example, the candidate face with the largest variance has a larger degree of variation in the mouth.

[0174] If there is a candidate face whose variance is less than the first threshold, stop judging the first threshold of the short video and start iterating through the next short video.

[0175] For example, if there exists a candidate face whose variance is less than the first threshold, it can be considered that the candidate face has few facial feature changes and does not meet the first threshold judgment, and then the next short video can be traversed.

[0176] In an optional embodiment of the present invention, the method may further include the following steps:

[0177] Step S106: Establish the correspondence between lip shape features and speech features through an audio-video synchronization network;

[0178] Step S107: After obtaining the speech features of the speaker, traverse the lip shape features based on the correspondence and speech features;

[0179] Step S108: When the lip-shape features match the speech features, determine that the speech and lip-shape of the speaker are synchronized.

[0180] The correspondence between lip-shape features and speech features is established through an audio-video synchronization network.

[0181] The audio-video synchronization network processes and transmits audio and video data, ensuring correct timing at the receiving end. Its core function is managing and synchronizing the timestamps of audio and video signals. At the sending end, a timestamp is added to each audio and video frame to identify its creation time. At the receiving end, these timestamps are used to adjust the playback order and timing of the audio and video to guarantee synchronization.

[0182] For example, synchronization can be determined by analyzing the features of audio and video. For video, lip-sync features are extracted, such as lip shape and degree of opening; for audio, acoustic features of speech, such as frequency and amplitude, are extracted. Then, machine learning or signal processing algorithms are used to establish the correspondence between speech features and lip-sync features. In a simple example, when a specific pronunciation occurs in the audio, a corresponding specific lip shape should appear in the video. By continuously matching and comparing these features, it can be determined whether the speech and lip-sync are synchronized. If the features of audio and video match consistently and stably over a period of time, then the speech and lip-sync can be considered synchronized.

[0183] Alternatively, audio and video buffers can be set up on the playback end. Due to network transmission and other factors, the arrival time of audio and video data may fluctuate. The purpose of the buffer is to temporarily store this data for unified scheduling and playback. The playback end will dynamically adjust the size of the buffer and the data reading speed according to the playback progress of the audio and video. For example, if it is found that the video playback progress is faster than the audio, the video playback speed will be slowed down appropriately, or the audio playback speed will be increased (within the allowable range). In this way, the synchronization of audio and video is fine-tuned to ensure that the speech and lip movements are consistent during playback.

[0184] Alternatively, during the audio and video encoding process, a timestamp is added to each frame of video and each segment of audio. The timestamp records the temporal position of that frame or segment of data within the original content. When the playback device receives the audio and video data, it aligns the audio and video according to the timestamps. Specifically, it plays the corresponding audio and video frames at the same time as indicated by the timestamps, thus ensuring temporal synchronization between speech and lip movements. For example, when a video frame shows a person opening their mouth to speak, the corresponding audio with the same timestamp will also be played simultaneously, ensuring that speech and lip movements remain consistent in time.

[0185] Once the speech features of the speaker are obtained, the lip shape features are iterated based on the correspondence and speech features.

[0186] When lip-shape features match speech features, the synchronization between the speaker's speech and lip-shape is determined.

[0187] For example, when speech features are obtained, the lip-shape features can be traversed according to the correspondence between speech features and lip-shape features. If the lip-shape and speech match, it can be determined that the speaker's speech and lip-shape are synchronized.

[0188] In an optional embodiment of the present invention, the step of cutting the input video into at least one short video containing only a single sentence may specifically include the following steps:

[0189] Step S1011: Unify the parameters of the input video;

[0190] Step S1012: Segment the input video after parameter unification to obtain at least one short video containing only a single sentence.

[0191] Standardize the parameters of the input video.

[0192] Among them, parameter unification can include unifying the video parameter of 25FPS and the audio parameter of 16KHz sampling frequency.

[0193] The input video with unified parameters is segmented into sentences to obtain at least one short video containing only a single sentence.

[0194] One approach is to use an ASR (Automatic Speech Recognition) model to segment the input video into sentences, breaking down long videos into multiple short videos of single sentences.

[0195] Because the captured videos are often long, a single video may contain multiple sentences. Furthermore, due to differences in pronunciation habits among individuals, the distribution of mouth positions between sentences is relatively random, making it difficult to ensure that the mouths are naturally closed. While this distribution may appear visually plausible, for audio-visual synchronization networks, the mouths of individuals in silent intervals should be strictly closed. This random distribution of mouth positions leads to increased output errors in the audio-visual synchronization network, severely impacting its judgment of the synchronization of the entire video and rendering the model's output unreliable.

[0196] The ASR algorithm is used to identify effective pronunciation intervals in speech. Therefore, the ASR algorithm can obtain the time coordinates of the effective pronunciation intervals in the current audio. With these time coordinates, silent intervals can be deleted (the purpose of ASR segmentation is to select only the segments containing speech pronunciation and delete the silent interval segments without speech pronunciation).

[0197] The ASR (Automatic Speech Recognition) algorithm is used to segment the video into sentences. ASR can mark the start and end times of each sentence in the video, and there are no silent intervals between the marked points. Based on these marked points and the video segmentation code, a single long video can be divided into multiple short videos of single sentences, ensuring that each short video does not contain silent intervals. This avoids introducing pauses or other silent intervals into the dataset, thus ensuring the reliability of the audio-video synchronization network output.

[0198] This sentence segmentation process can not only separate different sentences from the same person, but also separate sentences from different people in a multi-person conversation, ensuring that each short video segment belongs to the voice data of one person.

[0199] In an optional embodiment of the present invention, determining the target training dataset based on the first dataset may specifically include the following steps:

[0200] Step S1051: Obtain the confidence level of each data point in the first dataset.

[0201] Step S1052: If the confidence level is greater than or equal to the second threshold, determine the first dataset as the target training dataset.

[0202] Obtain the confidence score for each data point in the first dataset.

[0203] For example, an audio / video synchronization network can be used to obtain the confidence level of each data point in the first dataset.

[0204] If the confidence level is greater than or equal to the second threshold, the first dataset is determined as the target training dataset.

[0205] The second threshold can be a predefined probability threshold that filters out face detection results deemed "uncertain" or "low-quality" by the model, retaining only those with a confidence score (the model's certainty score) higher than the threshold. This is a common post-processing step in object detection tasks, used to balance precision and recall.

[0206] For example, the higher the confidence threshold, the more deterministic the model outputs detection results, the lower the probability of false detections (such as mistaking background objects for faces), but the more likely it is to miss detections (false negatives).

[0207] For example, the lower the threshold, the more faces are detected, but the risk of false detection increases.

[0208] For example, in real-time applications (such as camera surveillance), the threshold needs to be adjusted according to the scenario requirements: a high threshold (e.g., 0.9) ensures highly reliable detection results and is suitable for scenarios such as security verification. A low threshold (e.g., 0.5) improves the face detection rate and is suitable for scenarios requiring quick previews or low false negatives.

[0209] This invention proposes an offline automated method for creating training datasets for digital human models based on photographs. This method not only retains the advantages of mainstream methods, such as high processing efficiency and simplicity, but also effectively solves the data filtering problem in cases involving multiple faces and long videos. Furthermore, through cooperation with other algorithms, the accuracy and robustness of the entire preparation process are greatly improved.

[0210] This invention proposes an automated data preparation method for training digital human models from photos. This method is applicable to dataset preparation in any language and can effectively solve the data filtering problem in cases involving multiple faces and long videos. The method mainly includes: first, converting the video file format to unify the parameters of the video files and ensure consistency in subsequent processing; then, using an ASR speech recognition model to segment the input video into sentences, cutting long videos into multiple short videos of single sentences; next, using a face detection model to perform face detection and identity verification on each short video, ensuring that the video speech corresponds to the person in the video, and extracting the facial region of the speaker and saving it as a new video segment; finally, using a pre-trained audio-video synchronization network model to determine the synchronization between speech and lip movements, retaining video segments that meet the confidence threshold as the subsequent training dataset. A flowchart of the overall algorithm is shown below. Figure 2 As shown.

[0211] Reference Figure 2 The flowchart illustrates the steps of an embodiment of the data preparation method of the present invention. Figure 2 The method may specifically include the following steps:

[0212] Step S201: Input the original video.

[0213] The original video is input into the model.

[0214] Step S202: Convert the video to a unified video format.

[0215] Perform video file format conversion to unify the parameters of the video files, including image and audio parameters. For example, the video parameter is 25 FPS and the audio parameter is a sampling frequency of 16 kHz to ensure consistency in subsequent processing.

[0216] Step S203: The speech recognition model segments the video.

[0217] Automatic Speech Recognition (ASR) algorithms are used to segment the video into sentences. ASR can mark the start and end times of each sentence in the video, and there are no silent intervals between the marked points. Based on these marked points and video segmentation code, a single long video can be divided into multiple short videos of single sentences, ensuring that each short video does not contain silent intervals. This avoids introducing pauses or other silent intervals into the dataset, thus ensuring the reliability of the audio-video synchronization network output.

[0218] Step S204: Loop through the segmented video set.

[0219] The speech recognition model can segment different sentences from the same person, and it can also segment sentences from different people in a multi-person conversation, ensuring that each short video segment belongs to the voice data of one person, and storing the segmented videos in a collection.

[0220] Step S205: Face detection model.

[0221] The segmented video is processed using a face detection model.

[0222] Step S206: Determine whether a human face exists in each frame.

[0223] If the conditions are met, proceed to step S207; otherwise, proceed to step S204.

[0224] Determine whether a human face exists in each frame of the segmented short video.

[0225] Step S207: Determine whether the speaker meets the confidence level based on the distance between the inter-frame face loU and the mouth key points.

[0226] If the conditions are met, proceed to step S208; otherwise, proceed to step S204.

[0227] When each frame of the entire video contains only one face detection result, it is assumed that this unique face corresponds to the audio data. The facial mouth feature points are then statistically analyzed, and their mean and variance are calculated. Only when the statistical mean and variance of the feature points exceed a preset threshold is the face considered to correspond to the audio data, and the short video is deemed a qualified candidate video. Otherwise, the short video is directly filtered out, and the process moves to the next short video.

[0228] When there are multiple faces in at least one frame of a video, it is necessary to make a judgment based on the IoU and the statistical results of the feature points of the mouth of each face.

[0229] If the confidence requirement is not met, discard the currently processed video segment and start processing the next video segment from the video segment set again.

[0230] It should be noted that this invention aims to define a video interval so that each face (lip movement) within the video interval corresponds one-to-one with the audio data. Therefore, in each video segment (after ASR algorithm to ensure each segment is a complete sentence, i.e., each frame corresponds to audio data), it is first necessary to ensure that each frame contains a face. If a frame lacks a face, it indicates that the speech in that frame cannot match the lip movement, thus failing to meet the requirement. In this case, the current video segment is abandoned, and the process moves to the next video segment. Assuming faces are present in all video intervals, it is also necessary to ensure that multiple faces are present in each video frame. The final obtained face data is the face that matches the speech (i.e., the speaking face among multiple faces). An IoU threshold between video frames is used to ensure that there are no significant cuts between frames, maintaining a relatively stable position and size of the speaking face, and excluding cases where faces suddenly appear in a frame (where the IoU is likely 0). If the IoU between video frames is insufficient, it may indicate cuts or other issues, preventing the acquisition of the person (lip movement) that corresponds one-to-one with the speech. If the current video segment cannot meet the preset requirements, then the processing of the current video segment ends and the processing of the next video segment begins. If the IoU judgment is satisfied, the average and variance of the distance between the key points of the mouths of the characters in the video segment are used to ensure that the acquired face is the face corresponding to the speech (for example, if multiple faces are stably present in each frame of a certain video segment and only one person is speaking, then the IoU of multiple faces satisfies the confidence judgment. This scenario is common in interview, news and other videos). Since the mouths of non-speaking faces are relatively stable, the mean and variance of the feature points are relatively small. If the statistical values ​​of the key points of all faces that currently satisfy the IoU judgment do not meet the confidence requirements of the feature point distribution, then the currently acquired faces may not be able to match the speech frame by frame (for example, in an interview video, there may be only one person in the video, but that person is a listener during the current video segment, and the person speaking is not in the video segment). In this case, the current video segment is discarded and the next loop begins.

[0231] Step S208: Extract the face region and save it separately.

[0232] After matching faces and voices, it is necessary to align face regions based on the face detection results. This operation can eliminate irrelevant image regions and ensure the size consistency of all face images. On the other hand, it can prepare data for subsequent audio and video synchronization networks.

[0233] Step S209: Synchronization network judgment.

[0234] If the conditions are met, proceed to step S210; otherwise, proceed to step S204.

[0235] Based on an audio-video synchronization network, the synchronization status of speech and lip movements in the current video is determined. Due to the natural pronunciation differences between different languages, when an audio-video synchronization network trained on one language is directly used to determine the synchronization of video data in another language, the results often show significant fluctuations.

[0236] Step S210: Candidate training dataset.

[0237] The basic matching of face and voice has been completed in the previous steps, which reduces the limitation of directly relying on step S210 as the judgment standard. Step S210 is only used as a posterior algorithm to further confirm and filter the results. Therefore, the filtering threshold in step S210 can be increased so that the fluctuation of prediction error caused by different languages ​​can be included, thereby reducing the possibility of qualified data being filtered out, improving the data output rate, and increasing the robustness and accuracy of the whole process.

[0238] For example, obtaining facial regions (lip movements) that correspond one-to-one with speech from a video can be achieved using ASR (Automatic Speech Recognition) algorithms. While these algorithms can segment long videos, they cannot guarantee that all segmented video segments will meet the requirements. Specifically, they cannot ensure that every frame of a video segment will contain a matching lip movement. ASR's role is to divide the video into smaller, effective segments (removing silent intervals between sentences), minimizing the processing difficulty and errors that silent intervals between sentences might introduce in subsequent processing. This also increases the probability of successful speech-lip movement matching and the likelihood of obtaining successful data (each sentence corresponds to one person's sentence, making lip movement matching easier, and each sub-video segment is relatively independent and stable). Therefore, the ASR algorithm is not limited to processing long videos; it remains effective even for short videos consisting of only two sentences.

[0239] Similarly, calculating IoU and counting keypoints on video cannot guarantee that every frame of the acquired video corresponds to valid speech data. The main purpose of IoU and keypoint statistics is to ensure that the final acquired face (lip movements) corresponds one-to-one with the speech. Solving the problem of multiple faces is only one part of it; it can also solve problems such as mismatch between lip movements and audio in the case of a single face. In addition, it is difficult to directly count face-related parameters on the original video. On the one hand, the original video is too long, which increases the number of face parameters to be counted, leading to an increase in computation and memory usage. On the other hand, it is difficult to make face-related judgments (including the continuity of face appearance and the matching degree with speech) in the complex environment of long videos, and it is not a perfect match with the final goal (the goal is to obtain lip movements that match the speech, regardless of the length of the video). Therefore, judging face-related parameters based on the processing results of ASR can ensure that each processed segment is a valid speech segment. At this time, we can only focus on whether there is a match between face and speech, which greatly reduces the difficulty of data processing.

[0240] Therefore, the entire algorithm is a system problem. It cannot be solved by taking out a single method and talking about it alone. Instead, multiple solutions work together to achieve the final goal.

[0241] Reference Figure 3 The flowchart illustrates the steps of an embodiment of the data preparation method of the present invention. Figure 3 The method may specifically include the following steps:

[0242] Step S301: Perform video file format conversion to unify the parameters of the video file.

[0243] Since video data may originate from various acquisition devices and platforms, it is difficult to ensure the uniformity of the acquired video formats and parameters. To guarantee the accuracy of subsequent processing, the videos need to be uniformly converted. In this solution, video format conversion code is used to convert the video data format, uniformly converting all videos into a combination of 25FPS image and 16KHz audio, ensuring consistency in subsequent processing.

[0244] Step S302: Use the ASR speech recognition model to segment the input video into sentences, cutting the long video into multiple short videos of single sentences.

[0245] Because the captured videos are often long, a single video may contain multiple sentences. Furthermore, due to differences in pronunciation habits among individuals, the distribution of mouth positions between sentences is relatively random, making it difficult to ensure that the mouths are naturally closed. While this distribution may appear visually plausible, for audio-visual synchronization networks, the mouths of individuals in silent intervals should be strictly closed. This random distribution of mouth positions leads to increased output errors in the audio-visual synchronization network, severely impacting its judgment of the synchronization of the entire video and rendering the model's output unreliable.

[0246] To address potential errors caused by pauses in long videos, this solution uses an Automatic Speech Recognition (ASR) algorithm to segment the video into sentences. ASR marks the start and end times of each sentence in the video, ensuring that there are no silent intervals between these marks. Based on these marks and the video segmentation code, a single long video can be divided into multiple short, single-sentence videos, ensuring that each short video is free of silent intervals. This avoids introducing pauses or other silent intervals into the dataset, thus guaranteeing the reliability of the audio-video synchronization network's output.

[0247] This operation can not only segment different sentences from the same person, but also segment sentences from different people in a multi-person conversation, ensuring that each short video segment belongs to the voice data of one person.

[0248] Step S303: Use a face detection model to perform face detection and identity judgment on each short video to ensure that the video audio corresponds to the video person, and extract the facial area of ​​the person speaking and save it as a new video clip.

[0249] It should be noted that the identity verification process refers to using IoU (Interchange of Identity) and facial landmarks to determine the identity of individuals. Each video segment is then filtered to select those where every frame contains lip movements that match the spoken words. In other words, this is a data filtering process.

[0250] While the single-sentence short videos generated in step S302 ensure that each video contains valid audio data, they cannot guarantee the legitimacy of the image data. Since multiple people may appear in a video frame, and camera movements and character switching may occur during video recording, this could potentially cause a mismatch between audio and lip movements. Therefore, it is necessary to perform legitimacy checks and filtering on the image data in the dataset.

[0251] It should be noted that the legitimacy check requires ensuring that each frame of audio corresponds to a video frame with a corresponding speaker's face. This legitimacy check utilizes Interchange of Units (IoU) and facial landmark statistics to filter and screen the data.

[0252] This solution uses face detection and statistical methods to perform the image legitimacy detection process:

[0253] First, a face detection algorithm is used to detect faces in each frame of the short video to ensure that each frame contains at least one face. If no face is found in any frame of the short video, it means that the audio and lip movements of at least one frame are not aligned. In this case, the short video is directly filtered out and the process moves to the next short video.

[0254] If each frame contains at least one face, the faces and speech need to be aligned to ensure the speech is from the current person. Since the long video was segmented using the ASR speech recognition algorithm in step two, removing silent frames, it can be assumed that the current short video has no silent intervals; that is, the lip movements of the person aligned with the speech should be continuously changing. Based on this assumption, processing is performed according to the number of faces in the video frame:

[0255] (1) When each frame of the entire video contains only one face detection result, it is assumed that the unique face corresponds to the voice data. The facial mouth feature points of the face are statistically analyzed, and the mean and variance of the mouth feature points are calculated. Only when the statistical mean and variance of the feature points are greater than the preset threshold is it determined that the face corresponds to the voice data and the short video is a qualified candidate video. Otherwise, the short video is directly filtered out and the next short video processing flow is entered.

[0256] (2) When there are multiple faces in at least one frame of the video, it is necessary to use the Intersection over Union (IoU) and the statistical results of the feature points of the mouth of each face for judgment: First, the faces appearing in the first frame are used as the reference for statistics (there may be more than one reference face). New faces appearing in other subsequent frames are considered invalid faces because they cannot match the speech that appeared before them, so they are not counted. Then, based on the reference faces, the IoU between all reference faces and all faces detected in the next frame is calculated. For each reference face, the detected face with the largest IoU is selected as the temporary matching object, and other detected faces are ignored. When the IoU is greater than the preset threshold, it is regarded as the update of the corresponding reference face in the frame, and the detected face is used to update the "matched" reference face. If the IoU is less than the preset threshold, it is determined that the reference face failed to match in the current frame, and the failed reference face is abandoned until the video frame matching ends or the reference face is empty. If the baseline face is empty after processing all video frames, the short video is directly filtered out and the process moves to the next short video. Otherwise, the mean and variance of the facial mouth feature points of all remaining baseline faces are statistically analyzed, and a confidence threshold is used for filtering and judgment. If multiple baseline faces meet the confidence threshold, the baseline face with the largest variance is selected as the face corresponding to the audio. If none of the remaining baseline faces meet the confidence threshold, the short video is directly filtered out and the process moves to the next short video.

[0257] After matching faces and voices, it is necessary to align face regions based on the face detection results. This operation can eliminate irrelevant image regions and ensure the size consistency of all face images. On the other hand, it can prepare data for subsequent audio and video synchronization networks.

[0258] Step S304: Use a pre-trained audio-video synchronization network model to determine the synchronization between speech and lip movements, and retain video segments that meet the confidence screening threshold as subsequent training datasets.

[0259] This study uses an audio-video synchronization network to determine the synchronization status of speech and lip movements in a current video. Due to the inherent pronunciation differences between languages, directly using an audio-video synchronization network trained on one language to judge the synchronization of video data in another language often results in significant fluctuations. When the selected threshold is too low, most qualified candidate videos may be filtered out, leading to a low dataset output rate. Conversely, when the selected threshold is too high, some data with mismatched faces and speech (e.g., transitions between faces in the video causing speech to only partially match lip movements, or silences in the speech where the mouth is not closed) may be considered qualified data, introducing a large amount of noisy data into the training dataset. Therefore, the current mainstream approach to data preparation relying solely on audio-video synchronization networks has significant limitations and low robustness.

[0260] In this scheme, the basic matching of face and voice is completed through the processing of steps S301 to S303, reducing the limitation of directly relying on step S304 as the judgment standard. Step S304 is only used as a posterior algorithm to further confirm and filter the results. Therefore, the filtering threshold in step S304 can be amplified so that the fluctuation of prediction error caused by different languages ​​can be included, thereby reducing the possibility of qualified data being filtered out, improving the data output rate, and increasing the robustness and accuracy of the whole process.

[0261] In summary, this invention provides a data preparation method that involves: segmenting an input video into at least one short video containing only a single sentence; performing face detection on the short video to identify the person in the video; matching the audio information corresponding to the short video with the person information to determine the speaker; extracting the facial region of the speaker from the short video and storing it in a first dataset; and determining a target training dataset based on the first dataset. This invention first segments the input video into short videos containing only a single sentence, thus solving the problem of long videos. Then, it performs face detection on the short videos to identify the person in the video; furthermore, it matches the audio information corresponding to the short video with the person information to determine the speaker; and finally, it extracts the facial region of the speaker from the short video and stores it in the first dataset. This method aligns facial regions and removes irrelevant image regions, ensuring the size consistency of all facial images. The target training dataset determined based on the first dataset improves data output, increases the robustness and accuracy of data preparation, and enhances the efficiency of data preparation.

[0262] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0263] Device Examples

[0264] Reference Figure 4 The diagram shows a structural block diagram of a data preparation apparatus according to the present invention, which may specifically include:

[0265] The cutting module 401 is used to cut the input video into at least one short video containing only a single sentence;

[0266] The recognition module 402 is used to perform face detection on the short video and identify the people information in the short video;

[0267] The determining module 403 is used to match the audio information corresponding to the short video with the character information to determine the speaker corresponding to the short video;

[0268] Storage module 404 is used to extract the facial region of the speaker from the short video and store it in the first dataset;

[0269] The determining module is further configured to determine the target training dataset based on the first dataset.

[0270] Optionally, the identification module includes:

[0271] The recognition submodule is used to recognize the facial information of the target object in the remaining frames of the short video, based on the facial information of the target object in the first frame of the short video.

[0272] The traversal module is used to stop the detection of the short video and traverse the next short video if the facial information of the target object is not found in any frame of the short video.

[0273] Optionally, the identification submodule includes:

[0274] The module is used to perform intersection-union calculation on the bounding boxes corresponding to each detected face in the remaining frames of the short video and the reference face in the first frame to obtain the calculation result;

[0275] The first determining submodule is used to determine the detected face as the face information of the target object when the calculation result is greater than a preset threshold; and to determine that the matching of the person information corresponding to the second frame and the first frame fails when the calculation result of each detected face in the second frame is less than or equal to the preset threshold, and to stop the matching of the short video; the second frame is any frame in the short video other than the first frame.

[0276] Optionally, the determining module includes:

[0277] The acquisition module is used to statistically analyze the facial mouth feature points of each candidate face in the person information and obtain the statistical results.

[0278] The second determining submodule is used to normalize the statistical results and determine the variance corresponding to each candidate face based on the normalized statistical results; and to determine the voice of the person speaking in the short video from the at least one candidate face based on the first threshold and the variance corresponding to each candidate face.

[0279] Optionally, the second determining submodule includes:

[0280] The third determining submodule is used to determine the candidate face as the voice of the short video if the variance of a candidate face is greater than or equal to the first threshold; and to determine the candidate face with the largest variance as the voice of the short video if the variance of at least two candidate faces is greater than or equal to the first threshold.

[0281] The traversal submodule is used to stop the first threshold judgment of the short video and traverse the next short video if there is a candidate face whose variance is less than the first threshold.

[0282] Optionally, the device further includes:

[0283] A module is established to create a correspondence between lip-shape features and speech features through an audio-video synchronization network.

[0284] The first traversal submodule is used to traverse lip shape features based on the correspondence and the voice features when the voice features of the speaker are obtained;

[0285] The fourth determining submodule is used to determine the synchronization of the speaker's speech and lip movements when the lip-shape features match the speech features.

[0286] Optionally, the cutting module includes:

[0287] The unified module is used to unify the parameters of the input video.

[0288] The submodule is used to segment the input video after parameter unification into sentences, and obtain at least one short video containing only a single sentence.

[0289] Optionally, the determining module further includes:

[0290] The acquisition submodule is used to acquire the confidence level corresponding to each data point in the first dataset.

[0291] The fifth determining submodule is used to determine the first dataset as the target training dataset when the confidence levels are all greater than or equal to the second threshold.

[0292] In summary, this invention provides a data preparation apparatus, comprising: a segmentation module for segmenting an input video into at least one short video containing only a single sentence; a recognition module for performing face detection on the short video and recognizing person information in the short video; a determination module for matching the speech information corresponding to the short video with the person information to determine the speaker corresponding to the short video; a storage module for extracting the facial region of the speaker from the short video and storing it in a first dataset; the determination module is further configured to determine a target training dataset based on the first dataset. This invention first segments the input video into short videos containing only a single sentence, thus solving the problem of long videos. Then, it performs face detection on the short videos to identify the people in them. Next, it matches the corresponding audio information with the person information to determine the speaker. Finally, it extracts the speaker's facial region from the short video and stores it in a first dataset. This approach aligns facial regions and removes irrelevant image areas, ensuring consistent size across all facial images. Based on the target training dataset determined from the first dataset, it improves data output, robustness, and accuracy of data preparation, thereby increasing the efficiency of data preparation.

[0293] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0294] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0295] Regarding the processor in the above embodiments, the specific manner in which each module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0296] Reference Figure 5 This is a structural block diagram of an electronic device for data preparation provided in an embodiment of the present invention. Figure 5 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store executable instructions, which cause the processor to execute the data preparation method of the aforementioned embodiment.

[0297] The processor can be a CPU, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable devices, transistor logic devices, hardware components, or any combination thereof. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0298] The communication bus may include a path for transmitting information between the memory and the communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0299] The memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0300] This invention also provides a non-transitory computer-readable storage medium that, when instructions in the storage medium are executed by a processor of an electronic device (server or terminal), enables the processor to perform... Figure 1 The data preparation method shown.

[0301] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0302] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0303] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0304] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0305] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0306] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0307] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0308] The above provides a detailed description of the data preparation method, apparatus, electronic device, and storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data preparation method, characterized in that, The method includes: Cut the input video into at least one short video containing only a single sentence; Perform face detection on the short video to identify the people in the short video; The audio information corresponding to the short video is matched with the person information to determine the speaker of the short video; Extract the facial region of the person making the speech from the short video and store it in the first dataset; The target training dataset is determined based on the first dataset.

2. The method according to claim 1, characterized in that, The step of performing face detection on the short video and identifying the people in the short video includes: Using the facial information of the target object in the first frame of the short video as the reference face, the facial information of the target object in the remaining frames of the short video is identified; If the facial information of the target object is not found in any frame of the short video, the detection of the short video is stopped, and the next short video is traversed.

3. The method according to claim 2, characterized in that, The step of identifying the facial information of the target object in the remaining frames of the short video, based on the facial information of the target object in the first frame of the short video, includes: The rectangles corresponding to each detected face in the remaining frames of the short video are compared with the reference face in the first frame to calculate the intersection and union ratio, and the calculation result is obtained. If the calculation result is greater than a preset threshold, the detected face is identified as the face information of the target object; If the calculation results for each detected face in the second frame are all less than or equal to the preset threshold, it is determined that the matching of the person information corresponding to the second frame and the first frame has failed, and the matching of the short video is stopped; the second frame is any frame in the short video other than the first frame.

4. The method according to claim 1, characterized in that, The person information includes at least one candidate face; the step of matching the audio information corresponding to the short video with the person information to determine the speaker corresponding to the short video includes: The facial mouth feature points of each candidate face in the aforementioned person information are statistically analyzed to obtain statistical results; The statistical results are normalized, and the variance corresponding to each candidate face is determined based on the normalized statistical results. Based on a first threshold and the variance corresponding to each candidate face, the speaker corresponding to the short video is determined from the at least one candidate face.

5. The method according to claim 4, characterized in that, The step of determining the speaker corresponding to the short video from the at least one candidate face based on a first threshold and the variance corresponding to each candidate face includes: If there exists a candidate face whose variance is greater than or equal to the first threshold, then the candidate face is identified as the voice-over character corresponding to the short video. If there are at least two candidate faces whose variance is greater than or equal to the first threshold, then the candidate face with the largest variance is determined as the voice of the person in the short video. If there exists a candidate face whose variance is less than the first threshold, stop the first threshold determination of the short video and start iterating through the next short video.

6. The method according to claim 1, characterized in that, The method further includes: Establish the correspondence between lip-shape features and speech features through an audio-video synchronization network; Having obtained the speech features of the speaker, the lip-shape features are traversed based on the correspondence and the speech features; When the lip-shape features match the speech features, it is determined that the speech and lip-shape of the speaker are synchronized.

7. The method according to claim 1, characterized in that, The step of cutting the input video into at least one short video containing only a single sentence includes: Standardize the parameters of the input video; The input video with unified parameters is segmented into sentences to obtain at least one short video containing only a single sentence.

8. The method according to claim 1, characterized in that, Determining the target training dataset based on the first dataset includes: Obtain the confidence level corresponding to each data point in the first dataset; If all confidence levels are greater than or equal to the second threshold, the first dataset is determined to be the target training dataset.

9. A data preparation apparatus, characterized in that, The device includes: The segmentation module is used to segment the input video into at least one short video containing only a single sentence; The recognition module is used to perform face detection on the short video and identify the people in the short video; The determination module is used to match the audio information corresponding to the short video with the character information to determine the speaker corresponding to the short video; A storage module is used to extract the facial region of the person making the speech from the short video and store it in a first dataset; The determining module is further configured to determine the target training dataset based on the first dataset.

10. An electronic device, characterized in that, The electronic device includes a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store executable instructions that cause the processor to perform the data preparation method as described in any one of claims 1 to 8.

11. A readable storage medium, characterized in that, When the instructions in the readable storage medium are executed by the processor of the electronic device, the processor is able to perform the data preparation method as described in any one of claims 1 to 8.

Images