Data processing device and method for automatically training mouth shape model

Abstract

The invention relates to the technical field of data processing, in particular to a data processing device and method for automatically training a mouth shape model, and the device comprises a video input module which is used for receiving a video data stream to be processed; the video segmentation module is used for segmenting an input video data stream into small segments; the single-person and multi-person detection module is used for detecting persons in the video and identifying the appearance of a single person or multiple persons; the video audio and lip synchronization detection module is used for analyzing the audio and the mouth shape in the video so as to detect the audio and lip synchronization condition; the data storage and management module is used for storing and managing all the processed data, including the segmented video clips, the figure detection result and the audio and lip synchronization data; compared with the prior art, efficient data processing, accurate character detection, synchronous sound and lip detection and optimized data management and storage can be realized, progress in the field of mouth shape model training is promoted, and powerful support is provided for development and improvement of voice-related applications.

Description

[Technical field]

[0001] The invention relates to the technical field of data processing, in particular to a data processing device and method for automatically training a lip shape model. [Background Technology]

[0002] At present, the existing video data processing process may have disadvantages such as computing resource requirements, data privacy, data collection difficulty, and technology dependence, as follows:

[0003] (1) Computing resource requirements: Automated processing of video data requires a large amount of computing resources, including high-performance CPUs and GPUs; this results in high costs, not only in terms of hardware procurement costs, but also in terms of electricity and cooling costs.

[0004] (2) Data privacy: During the automated processing, a large amount of voice and lip data may be used; this data may contain personal sensitive information, such as sound, language content, and lip features; therefore, protecting data privacy is crucial; the leakage or abuse of data may lead to legal and trust issues, and strict data privacy protection measures need to be taken, including data desensitization, encryption, and permission control.

[0005] (3) Data collection difficulty: Obtaining enough high-quality speech and lip data can be very challenging; this requires extensive data collection, including diverse voices, accents, languages, and pronunciations; the collection process requires time and resources to ensure the representativeness and accuracy of the dataset; in addition, data quality control is also a challenge, as inconsistent or erroneous data may lead to degraded model performance.

[0006] (4) Technology Dependence: The success of automated video data processing is highly dependent on a variety of technologies, including machine learning algorithms, data preprocessing techniques, and model architectures. These technologies are in constant development and evolution, and the equipment needs to be constantly updated and maintained to adapt to new technological trends and improvements. This may result in additional maintenance costs and complexity, making the maintainability of the equipment a challenge. [Summary of the invention]

[0007] The purpose of the present invention is to solve the above-mentioned shortcomings and provide a data processing device for automatically training lip models, which can achieve efficient data processing, accurate character detection and lip synchronization detection, optimized data management and storage, promote the progress in the field of lip model training, and provide strong support for the development and improvement of speech-related applications.

[0008] In one aspect, the present invention provides a data processing device for automatically training a lip model, comprising:

[0009] A video input module, used for receiving a video data stream to be processed;

[0010] A video segmentation module is used to segment the input video data stream into small segments;

[0011] Single-person and multi-person detection module, used to detect people in the video and identify the presence of a single person or multiple people;

[0012] Video lip sync detection module, used to analyze the audio and lip shape in the video to detect lip sync;

[0013] The data storage and management module is used to store and manage all processed data, including segmented video clips, person detection results, and lip synchronization data.

[0014] In a possible implementation, the video input module includes a camera or a video file interface, and the video input module is used to collect or import video data.

[0015] In another possible embodiment, in the video segmentation module, the video is segmented based on time, including but not limited to segmentation by seconds or frames, or segmentation is performed based on events or actions, and the video segmentation provides an analysis unit for subsequent single-person and multi-person detection and lip synchronization detection.

[0016] In other possible embodiments, the single-person and multi-person detection module adopts a deep learning model, including but not limited to a convolutional neural network CNN or a target detection algorithm, and its detection results include the position coordinates and identifier of each person.

[0017] In another possible embodiment, the video lip sync detection module uses speech processing technology and visual analysis algorithms to associate audio and lip data and determine whether they are synchronized.

[0018] In yet another possible embodiment, the data in the data storage and management module is organized according to person identifiers and timestamps to facilitate retrieval and analysis.

[0019] In another aspect, the present invention provides a data processing method for automatically training a lip model, comprising the following steps:

[0020] 1) The user provides video data through the video input module. The video data is a real-time video stream from a camera or a pre-recorded video file;

[0021] 2) The video segmentation module receives the video data and segments it into small segments according to predetermined rules and durations to form a sequence of video segments;

[0022] 3) The single-person and multi-person detection module receives each video clip and uses deep learning or computer vision algorithms to detect the people in the video;

[0023] 4) The video lip sync detection module receives each video clip simultaneously, extracts audio and lip information from it, and analyzes the audio and lip data to determine the synchronization between them;

[0024] 5) The processed data is stored in the data storage and management module for subsequent training model use.

[0025] In a possible implementation, the following steps are also included: the user configures and controls the entire processing process through the control interface, monitors the progress, and retrieves the processed data for subsequent model training or analysis.

[0026] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, and the program executes the above method.

[0027] In a fourth aspect of the present invention, a computer device is proposed, comprising: a processor, a memory and a bus; the processor and the memory are connected via the bus; the memory is used to store programs, the processor is used to run programs, and the above method is executed when the program is run.

[0028] Compared with the prior art, the present invention has the following advantages:

[0029] (1) Improved data processing efficiency: The present invention significantly improves the processing efficiency of the data required for training the model. With the help of the video segmentation module, large-scale video data can be easily divided into small segments, making it easier to manage and analyze. This high efficiency helps to quickly build and update the training data set, thereby improving the speed and effect of model training.

[0030] (2) Improved accuracy of person detection: The single-person and multi-person detection module of the present invention utilizes advanced deep learning and computer vision algorithms to detect people in a video with high accuracy; this means that users can accurately identify a single person as well as multiple people in a video and provide accurate location information for each person; this is especially important for analyzing the training data of the lip model, as different people may have different lip shapes and pronunciations.

[0031] (3) Improved lip-sync detection accuracy: The video lip-sync detection module of the present invention can more accurately detect the synchronization between audio and lip shape through deep learning and visual analysis technology; this helps to improve the training data of the lip shape model and make it more adaptable to the actual audio and lip shape relationship; this will improve the performance of the lip shape model, especially in applications of simulating and generating lip shapes.

[0032] (4) Data management and storage optimization: The present invention includes an efficient data storage and management module that ensures that processed data is securely stored and easily retrieved as needed; data can be organized by person, timestamp, or other identifiers, allowing users to easily manage and access data, thereby improving data availability and maintainability.

[0033] (5) User-friendly control interface: To ensure that users can easily configure and monitor the entire treatment process, the present invention includes an intuitive control interface; users can use this interface to start the treatment process, monitor progress and retrieve processed data, which makes the operation of the device easier and more controllable.

[0034] (6) Data quality improvement: The present invention helps to improve the quality of data required for training models through accurate segmentation of video data, character detection and lip synchronization detection; high-quality data is one of the key factors for the success of training models, so this technical effect will directly affect the performance and accuracy of the model.

[0035] (7) Accelerate research and application: Due to the efficiency and accuracy of the present invention, researchers and developers can conduct research and application of lip models more quickly; this will help promote technological progress in the fields of speech recognition, sentiment analysis, speech synthesis, etc., while also providing more opportunities for practical applications.

[0036] In summary, the present invention exhibits efficient data processing, accurate character detection and lip synchronization detection, optimized data management and storage, and a user-friendly control interface, which together promote progress in the field of lip model training and provide strong support for the development and improvement of speech-related applications; at the same time, by improving data quality and processing efficiency, the present invention is expected to have a far-reaching impact in multiple fields and improve the performance and usability of lip models. [Drawings]

[0037] Figure 1 It is a schematic diagram of the structure of the device of the present invention;

[0038] Figure 2 It is a schematic diagram of the steps of the method of the present invention;

[0039] Figure 3 is a flow chart of the present invention;

[0040] Figure 4 It is a structural schematic diagram of the computer device in the present invention. [Specific implementation method]

[0041] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present invention. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0042] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described below in conjunction with the accompanying drawings and specific embodiments:

[0043] In one embodiment, as shown in the attached Figure 1 Shown and attached Figure 3 As shown, the present invention provides a data processing device for automatically training a lip model, that is, a video processing device for automatically processing data required for training a lip model. The device is composed of multiple key components, which work closely together to achieve efficient processing of data required for training the model. The following is a detailed description of each component of the device and the connection relationship between them:

[0044] The video input module is the entrance of the device and receives the video data stream to be processed; the video input module usually includes a camera or video file interface for collecting or importing video data.

[0045] The video segmentation module is responsible for segmenting the input video stream into small segments for subsequent processing; segmentation can be based on time, such as by seconds or frames, or performed according to events or actions; this helps organize and process data, and provides an analysis unit for subsequent single-person and multi-person detection and lip synchronization detection.

[0046] Single-person and multi-person detection module: The task of this module is to detect people in the video and identify the appearance of single or multiple people so that each person can be analyzed separately. Deep learning models such as convolutional neural networks (CNNs) or target detection algorithms may be used to achieve person detection. The detection results include the location coordinates and identifier of each person.

[0047] The video lip sync detection module is responsible for analyzing the audio and lip shape in the video to detect lip sync. It may use speech processing technology and visual analysis algorithms to associate audio and lip shape data and determine whether they are synchronized. This is very important for training lip shape models.

[0048] Data storage and management module: All processed data, including segmented video clips, person detection results, lip synchronization data, etc., need to be stored and managed effectively; this can include database systems or cloud storage solutions to ensure data security and accessibility.

[0049] In some other embodiments, as shown in the attached Figure 2 As shown, the present invention provides a data processing method for automatically training a lip model, and the steps are as follows:

[0050] (1) The user provides video data through the video input module. This can be a real-time video stream from a camera or a pre-recorded video file.

[0051] (2) The video segmentation module receives the video data and segments it into small segments according to predetermined rules and durations to form a sequence of video segments.

[0052] (3) The single-person and multi-person detection module receives each video clip and uses deep learning or computer vision algorithms to detect the people in the video. It determines the location and identifier of each person for future analysis.

[0053] (4) The video lip sync detection module receives each video clip simultaneously and extracts audio and lip information from it. It analyzes the audio and lip data to determine the synchronization between them. This helps the training model capture lip sync information more accurately.

[0054] (5) The processed data is stored in the data storage and management module for subsequent training model use. The data can be organized based on person identifiers, timestamps, etc. for easy retrieval and analysis.

[0055] As a further embodiment, in order to ensure that the user can easily configure and monitor the entire processing process, the present invention also includes an intuitive control interface, through which the user can configure and control the entire processing process, monitor progress, and retrieve processed data for subsequent model training or analysis, which makes the operation of the device using the present invention easier and more controllable.

[0056] In some other embodiments, a specific embodiment of a data processing device is provided to demonstrate the effects and influences of the device:

[0057] A certain company is a medium-sized software enterprise with multiple R&D teams and project groups. There are some problems in the R&D process, such as low efficiency in processing the data required for the model, poor data quality, and high R&D costs. In order to solve these problems, the company introduced a data processing device for automatically training the lip model, implementing the following functions:

[0058] Video segmentation: The company uses a video segmentation module to segment videos. The module accurately segments the video, and each segment contains a complete speaker's voice and lip-syncing scenario; the timestamp is generated accurately to ensure the consistency of each segment.

[0059] Single-person and multi-person face detection: The detection module successfully identifies the single or multi-person situations in each short video segment and provides accurate location coordinates and unique identifiers, which helps to distinguish the lip shapes of different speakers.

[0060] Video lip sync detection: Using the video lip sync detection module, the company extracts the audio signal and lip shape data from each short video segment and uses the video processing device’s lip sync detection module to determine the synchronization between the audio and lip shape. The lip sync detection module analyzes the relationship between the audio and lip shape data and provides a synchronization score for each short video segment. This helps understand the lip sync of different speakers and provides more accurate data for the model.

[0061] Data storage and management module: The company's processed data, including segmented video clips, person detection results, and lip synchronization data, are stored in the device's data storage and management module, grouped by timestamp and speaker identifier. The data storage and management module ensures the secure storage and accessibility of the data; training data can be easily retrieved and managed for lip model training and analysis.

[0062] In summary, the video processing device of the present invention is used to process video data of multiple speakers, and the effects of precise segmentation, accurate character detection, lip synchronization analysis and data management are achieved. These effects will provide a high-quality data basis for the training of the lip model, and help improve the performance and accuracy of the lip model.

[0063] In some other embodiments, the Figure 4 The figure is a schematic diagram of a computer device of the present invention. The computer device includes a processor, a memory and a bus; the processor and the memory are connected via a bus, the memory is used to store a program, and the processor is used to run the program. When the program is run, the data processing method for automatically training a lip model provided by the present invention is executed. Furthermore, the present invention also provides a computer-readable storage medium, the computer-readable storage medium includes a stored program, and the program executes the data processing method for automatically training a lip model provided by the present invention.

[0064] The computer device may be a desktop computer, a notebook computer, a PDA, a cloud server, or other computing devices. Figure 4It is only an example of a computer device and does not constitute a limitation on the computer device. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc. The processor may be a central processing unit CPU, or other general-purpose processors, digital signal processors DSP, application-specific integrated circuits ASIC, off-the-shelf programmable gate arrays FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0065] The memory can be used to store computer programs and / or modules / units, and the processor realizes various functions of the computer device by running or executing the computer programs and / or modules / units stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the computer device (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card SMC, a secure digital SD card, a flash memory card, at least one disk storage device, a flash memory device or other volatile solid-state storage device.

[0066] If the functions described in the method of the embodiment of the present invention are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the part of the embodiment of the present invention that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, mobile computing device or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention; and the aforementioned storage medium includes various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories, random access memories, disks or optical disks. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0067] The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit it. Under the idea of ​​the present invention, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present invention as described above. Although the present invention is described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0068] The present invention is not limited to the above-mentioned implementation modes, and any other changes, modifications, substitutions, combinations, and simplifications made without departing from the spirit and principle of the present invention shall be equivalent replacement modes and shall be included in the protection scope of the present invention.

Claims

1. A data processing device for automatically training a lip model, characterized in that: include: A video input module, used for receiving a video data stream to be processed; A video segmentation module is used to segment the input video data stream into small segments; Single-person and multi-person detection module, used to detect people in the video and identify the presence of a single person or multiple people; Video lip sync detection module, used to analyze the audio and lip shape in the video to detect lip sync; The data storage and management module is used to store and manage all processed data, including segmented video clips, person detection results, and lip synchronization data.

2. The device according to claim 1, characterized in that: The video input module includes a camera or video file interface, and the video input module is used to collect or import video data.

3. The device according to claim 1, characterized in that: In the video segmentation module, the video is segmented based on time, including but not limited to segmentation by seconds or frames, or segmented according to events or actions. The video segmentation provides an analysis unit for subsequent single-person and multi-person detection and lip synchronization detection.

4. The device according to claim 1, characterized in that: The single-person and multi-person detection module adopts a deep learning model, including but not limited to a convolutional neural network (CNN) or a target detection algorithm, and its detection results include the position coordinates and identifier of each person.

5. The device according to claim 1, characterized in that: The video lip sync detection module uses speech processing techniques and visual analysis algorithms to correlate audio and lip data and determine whether they are synchronized.

6. The device according to claim 1, characterized in that: The data in the data storage and management module is organized according to person identifiers and timestamps to facilitate retrieval and analysis.

7. A data processing method for a data processing device for automatically training a lip model according to any one of claims 1 to 6, characterized in that: The following steps are involved: 1) The user provides video data through the video input module. The video data is a real-time video stream from a camera or a pre-recorded video file; 2) The video segmentation module receives the video data and segments it into small segments according to predetermined rules and durations to form a sequence of video segments; 3) The single-person and multi-person detection module receives each video clip and uses deep learning or computer vision algorithms to detect the people in the video; 4) The video lip sync detection module receives each video clip simultaneously, extracts audio and lip information from it, and analyzes the audio and lip data to determine the synchronization between them; 5) The processed data is stored in the data storage and management module for subsequent training model use.

8. The method according to claim 7, characterized in that The following steps are also included: The user configures and controls the entire processing process through the control interface, monitors the progress, and retrieves the processed data for subsequent model training or analysis.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program that executes the method of claim 7 .

10. A computer device, characterized in that: include: processor, memory, and bus; The processor is connected to the memory via the bus; The memory is used to store a program, and the processor is used to run the program. When the program is run, the method according to claim 7 is executed.

Images