Data processing method and equipment

Abstract

The invention provides a data processing method and equipment. The method comprises the following steps: acquiring first audio data and first image data corresponding to the same time period; extracting feature data containing the same object from the first audio data and the first image data to obtain corresponding second audio data and second image data; comparing the second audio data with the second image data, and determining a data time difference between the first audio data and the first image data; and aligning the first audio data and the first image data based on the data time difference.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a data processing method and device. BACKGROUND

[0002] In the field of multimedia data processing, especially in the application involving audio and video synchronization, how to realize the accurate alignment of sound and image is a key technical problem. With the popularization of virtual human, animation, video conference and other applications, the synchronization of audio and image data directly affects user experience and system performance.

[0003] In the related art, audio-visual synchronization is usually realized by adding time stamps in audio and video data. However, in actual application, due to the insufficient insertion accuracy of time stamps or the inconsistent time reference of different data sources, the problem of audio-visual asynchronization often occurs. In addition, in some application scenarios such as virtual human interaction, the time stamp mechanism is not adopted, which further aggravates the synchronization difficulty.

[0004] The existing method has obvious deficiencies in the accuracy, compatibility and cross-platform consistency of time stamps, and cannot effectively solve the timing deviation between audio and image data, thereby affecting the accuracy and stability of the overall data processing. SUMMARY

[0005] The embodiment of the present application provides a data processing method and device.

[0006] The technical scheme of the embodiment of the present application is realized as follows: The embodiment of the present application provides a data processing method, which comprises the following steps: Obtaining first audio data and first image data corresponding to the same time period; Extracting feature data containing the same object from the first audio data and the first image data respectively to obtain corresponding second audio data and second image data; Comparing the second audio data and the second image data to determine the data time difference between the first audio data and the first image data; Aligning the first audio data and the first image data based on the data time difference.

[0007] The embodiment of the present application provides a data processing device, which comprises a processor and a memory. The memory is used for storing audio data and image data. The processor is configured to retrieve first audio data and first image data corresponding to the same time period from memory; extract feature data containing the same object from the first audio data and first image data respectively to obtain corresponding second audio data and second image data; compare the second audio data and second image data to determine the data time difference between the first audio data and the first image data; and align the first audio data and the first image data based on the data time difference.

[0008] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the data processing method provided in this application when executed by a processor.

[0009] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the data processing method provided in this application. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 A flowchart illustrating an exemplary video playback method provided in this application embodiment; Figure 3 A flowchart illustrating an exemplary virtual human animation playback method provided in this application embodiment; Figure 4 A flowchart illustrating an exemplary data processing method provided in this application embodiment; Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] Audio-visual asynchrony is a common problem in related technologies, especially in virtual human animation and video playback scenarios.

[0013] For video playback scenarios: Traditional solutions rely on timestamp mechanisms. However, the addition of timestamps is not always accurate, meaning that the embedding of timestamps itself may contain errors. For example, in the acquisition, encoding, and packaging stages, if the timestamp does not record the actual generation or capture time of the data frame, then the synchronization based on it is wrong from the source. In addition, since timestamps only provide an ideal target of "when to play", they cannot control and handle the actual processing delays caused by uneven computational load in subsequent decoding, rendering, and other stages. Or, when the system is busy, even if there are correct timestamps, video or audio frames may be forced to be discarded because there is not enough time to process them, thus causing synchronization failure.

[0014] For virtual human animation scenarios: In the virtual human animation pipeline, from sound acquisition to virtual animation generation, the virtual images do not originate from a pre-generated, timestamped video file. Instead, they are a series of continuous, untimestamped image frames generated in real-time by audio. There is a lack of a benchmark mechanism to assign precise timing to these newly generated images. Furthermore, the virtual human generation path suffers from algorithmic biases (fluctuations in the inference time of the Artificial Intelligence (AI) model) and rendering biases (variations in the graphics engine load). These biases are dynamic and unpredictable. Traditional timestamping schemes for fixed media cannot measure or compensate for the random delays generated during this real-time generation process.

[0015] In summary, the fundamental contradiction faced by the relevant technologies lies in: For fixed media files, the synchronization mechanism (timestamp) itself may be inaccurate and cannot effectively cope with dynamic changes in the playback environment; For real-time generated virtual human animations, there is no synchronization benchmark and compensation mechanism that spans the entire link from audio input to video output.

[0016] To address the aforementioned problems, embodiments of this application provide a data processing method, implemented by a data processing device, such as... Figure 1 As shown, the process includes the following steps S101 to S104: Step S101: Obtain the first audio data and the first image data corresponding to the same time period.

[0017] In the embodiments of this application, the data processing device is an electronic device with data processing function, which may be a tablet computer, a laptop computer, a handheld computer, a personal digital assistant (PDA), a desktop computer, etc. No specific data processing device is limited here.

[0018] In the embodiments of this application, the same time period means that the first audio data and the first image data have the same starting point and duration on the time axis, usually a time window of 1 second, 5 seconds or 10 seconds.

[0019] For example, in a video playback scenario, the first audio data is the raw audio signal extracted from the player, such as user voice; the first image data is a segment of continuous frames extracted from the video stream, such as frames showing people in the video. Because the first audio data and the first image data lack a unified and precise timing reference in the encoding, encapsulation, or decoding stages, there is a timing deviation between the two during rendering, resulting in audio and video desynchronization.

[0020] For example, such as Figure 2 As shown, the timestamps in the image and audio data in the video source file 21 are not necessarily accurate (i.e., they have not been timestamp calibrated), which will cause a deviation between the audio data and the video data (corresponding to the original video deviation in the figure). In addition, during the playback of the video, the video will undergo video decoding 22 and the audio data will undergo sound decoding 23. During the video decoding 22 and sound decoding 23, the audio data and video data may be deviated again, so that there will be a time difference between the decoded video data displayed on the monitor 24 and the decoded audio data played by the speaker (e.g., loudspeaker) 25.

[0021] For example, such as Figure 3 As shown, in the display scenario of virtual human animation, the first audio data can be obtained when the user issues a voice command. The data processing device collects the user's voice command (31) and uses it as the first audio data. Simultaneously, the virtual human animation production process uses the Audio2Face (A2F) algorithm to process the first audio data based on the voice content, obtaining facial expression feature parameters. This drives and renders the virtual animation (33), which is then used as the first image data. This results in a time difference between the virtual animation displayed on the monitor (24) and the sound played back (34) through a speaker (e.g., a loudspeaker) (25). This is because converting sound into image data takes time during the virtual human animation production process, leading to a certain deviation between the first audio data and the second image data.

[0022] Step S102: Extract feature data containing the same object from the first audio data and the first image data respectively to obtain the corresponding second audio data and second image data.

[0023] In the embodiments of this application, since the video data and audio data played by the player may not match, that is, there is a phenomenon of audio and video being out of sync, the data processing device can extract feature data containing the same object from the first audio data and the first image data to obtain the corresponding second audio data and second image data.

[0024] In the embodiments of this application, "same object" refers to the common target described in the first audio data and the first image data, and "feature data" refers to information that can reflect the characteristics of the same object. For example, phoneme information and intonation rhythm in the first audio data, and facial key points and expression changes in the first image data; or, feature data containing the same type of the same object can be extracted from the first audio data and the first image data. For example, both the first audio data and the first image data can be converted into facial blend shape (BS) data to obtain second audio data and second image data. In this way, the second audio data and the second image data are converted into data of the same dimension, ensuring the comparability of the data. Among them, blend shape data (BS data) is the core parameter set driving shape blending in 3D animation. Its essence is a digital asset that realizes the dynamic change of the model's appearance through mathematical interpolation. Its mathematical essence is a weighted summation of vertex-level offset vectors, and the weights are usually adjusted between 0 and 1 to ensure smooth deformation.

[0025] For example, the second audio data can be generated by processing the first audio data using the Audio2Face algorithm, mapping it to facial expression parameters. The second image data can be generated by identifying a face region from the first image data and calculating the facial expression parameters corresponding to that face region. Audio2Face is a deep learning-based audio-driven facial animation technology that analyzes the acoustic features of speech (such as phonemes, intonation, and emotion) to generate accurate lip movements, facial expressions, and facial muscle movements of a 3D virtual character in real time, achieving automated animation generation of "sound-driven expressions."

[0026] For example, taking a virtual human animation as an example, when a user says "hello," Audio2Face analyzes each phoneme in the sentence and converts it into a series of facial blending shape weights, representing actions such as mouth opening, closing, and upturned corners of the mouth. Simultaneously, the Facial Expression (FE) algorithm analyzes the current facial image in the first image data, detecting information such as the position and shape of the lips, and generates a set of facial blending shape weights. Both sets of data generated by Audio2Face and Facial Expression reflect the same object—the speaker's mouth state. The FE algorithm is a core technology in the fields of computer vision and affective computing; it identifies and infers human emotional states (such as happiness, anger, surprise, etc.) by analyzing features such as muscle movement and texture changes in facial images or videos.

[0027] Step S103: Compare the second audio data and the second image data to determine the data time difference between the first audio data and the first image data.

[0028] In the embodiments of this application, after obtaining second audio data and second image data containing feature information of the same object state, the data processing device can compare the feature data of the same object in the two data, and then match the audio data and image data with the same feature data of the same object to obtain the corresponding data time difference.

[0029] For example, the data processing device can perform frame-by-frame or time-by-time similarity comparisons on the second audio data and the second image data to find the closest time point. The data time difference refers to the offset between the first audio data and the first image data on the time axis, which may manifest as the first audio data leading the first image data, or the first image data leading the first audio data.

[0030] For example, if the facial blending shape weight of a certain phoneme in the first audio data reaches its maximum value in the 5th frame, while the same action in the first image data does not appear until the 7th frame, it indicates that the first audio data is 2 frames ahead. When the first audio data is 2 frames ahead of the first image data, the data processing device can determine that there is a 2-frame delay and needs to compensate for and adjust the synchronization relationship between the audio and the image.

[0031] Step S104: Align the first audio data and the first image data based on the data time difference.

[0032] In the embodiments of this application, alignment refers to adjusting the playback time of the first audio data or the first image data according to the calculated data time difference, so that the two remain synchronized on the timeline. For example, audio-visual synchronization can be achieved without destroying the original content through frame-by-frame fine-tuning, interpolation, or other adjustment methods.

[0033] For example, taking virtual human animation as an example, assuming the first audio data is 2 frames ahead of the first image data, then the first audio data can be played 2 frames ahead while the first image data is playing, or the first image data can be played 2 frames ahead while the first audio data is playing. These playback methods can effectively eliminate the delay phenomenon and improve the viewing experience.

[0034] Thus, by extracting feature data of the same object from the first audio data and the first image data respectively, and performing comparative analysis, the delay between the first audio data and the first image data is calculated. Based on this, compensation and adjustment are made, achieving audio-visual alignment without relying on timestamps. This data processing method is applicable to various scenarios such as virtual human animation and video playback, and can significantly improve the accuracy of audio-visual synchronization and playback quality in these scenarios.

[0035] In some embodiments, when performing step S101, the data processing device may also perform the following steps S201 and S202: Step S201: Obtain video source data, and from the video source data, obtain audio data to be aligned and image data to be aligned, or obtain audio data to be aligned and generate image data to be aligned based on the audio data to be aligned.

[0036] In the embodiments of this application, video source data refers to the original media files or real-time streaming data used to extract audio and video information, such as video files read from local storage, live streams transmitted over a network, screen recordings, etc. Video source data contains audio signals and image frame sequences, providing basic material for subsequent alignment operations. In practical applications, video source data may originate from various scenarios such as players, virtual human systems, and online conferencing platforms.

[0037] In the embodiments of this application, the audio data to be aligned is the audio portion extracted from the video source data, which is typically in PCM format or other audio encoding formats.

[0038] In the embodiments of this application, the image data to be aligned is an image portion extracted from video source data, typically a continuous sequence of image frames, or possibly captured through methods such as screenshots. For ordinary videos, human faces may occasionally appear in the image data to be aligned; however, for virtual human animations, the image data to be aligned can be entirely generated through synthesis.

[0039] In embodiments of this application, the data processing device can acquire only the audio data to be aligned and generate corresponding image data to be aligned based on the audio data. When there is no readily available image data, Audio2Face (A2F) technology can be used to convert the input audio signal into facial blend shape (BS) data, thereby generating an image output that matches the audio content. The use of Audio2Face (A2F) technology can be applied to situations where existing image data is lacking, such as in voice-driven virtual human animation generation tasks.

[0040] Step S202: Sample data from the audio data to be aligned and the image data to be aligned for the same time period to obtain the corresponding first audio data and first image data.

[0041] In embodiments of this application, a data processing device can sample data from the audio data to be aligned and the image data to be aligned, respectively, for the same time period to obtain corresponding first audio data and first image data.

[0042] For example, the audio data to be aligned is 30 minutes long, and the video data to be aligned is also 30 minutes long. Sampling data within the same time period can be done by sampling the audio data from the 10th to the 15th minute of the audio data to be aligned as the first audio data, and sampling the video data from the 10th to the 15th minute of the video data to be aligned as the first image data; or, sampling data within the same time period can be done by sampling the audio data from the 0th to the 5th minute of the audio data to be aligned as the first audio data, and sampling the video data from the 0th to the 5th minute of the video data to be aligned as the first image data. Which specific portion of the entire time period is sampled can be determined based on actual needs and application scenarios, and this application does not limit this.

[0043] For example, if the first audio data is audio data from minute 0 to minute 5, and the first image data is image data from minute 0 to minute 5, then after performing the above steps S102 to S104, the remaining audio data from minute 5 to minute 30 and the image data will also have a corresponding alignment operation.

[0044] For example, if the video source data is a 2-hour movie, and the audio data to be aligned and the image data to be aligned are 2 hours of audio data and audio data respectively, the first audio data and the first image data can be 5 minutes of first audio data and first image data sampled every half hour from the audio data to be aligned and the image data to be aligned, so as to achieve alignment every half hour during the playback of the video source data. In this way, it is not necessary to perform alignment every 5 minutes for 2 hours, and the computing power can be reduced. Of course, data alignment can also be performed every 5 minutes in a video source data, which has higher accuracy.

[0045] In some embodiments, when performing step S102, the data processing device may perform the following steps S301 to S304: Step S301: Extract audio features from the first audio data to obtain audio feature information, and map the audio feature information to the facial morphology parameter space to obtain the first facial feature weight.

[0046] In the embodiments of this application, audio feature extraction refers to identifying feature parameters related to speech content from audio signals, such as spectrum, energy, fundamental frequency, etc., to describe the speaker's speech state. The feature parameters obtained by audio feature extraction can reflect semantic, tone, emotion, and other information in the audio data.

[0047] In the embodiments of this application, audio feature extraction is achieved through an algorithmic model (e.g., the A2F algorithm). The data obtained from the audio feature extraction is used to generate facial blending shape weights, thereby driving changes in the facial expressions of virtual humans or characters. Different facial blending shape weight values ​​correspond to different facial expressions, and the content expressed by the audio data is also different. For example, in the expression of smiling, features such as upturned corners of the mouth and squinting eyes can be represented by combinations of facial blending shape weights corresponding to features such as upturned corners of the mouth and squinting eyes. Alternatively, during the process of a character speaking, different facial expressions result in different facial blending shape weights.

[0048] In the embodiments of this application, facial blending shape weight is a set of values ​​representing the state of facial muscle movement. It is usually composed of multiple facial key points, each key point corresponding to a facial blending shape weight value, ranging from 0 to 1. The facial blending shape weight typically involves shape weight values ​​corresponding to 54 facial key points, each facial key point corresponding to a different facial position.

[0049] Step S302: Determine the second audio data based on the first facial feature weight.

[0050] In the embodiments of this application, the data processing device can directly determine the first facial feature weight as the second audio data, or it can determine the shape weight value corresponding to certain fixed key points in the first facial feature weight as the second audio data.

[0051] Step S303: Extract facial key points from the first image data to obtain facial feature information, and generate a second facial feature weight based on the facial expression represented by the facial feature information.

[0052] In the embodiments of this application, facial key point extraction involves identifying important anatomical locations of the face from image data, such as key coordinate points of the eyes, nose, mouth, and eyebrows. These key coordinate points form the basis of the facial contour and can be used to calculate facial expression deformations.

[0053] For example, the data processing device can use a facial expression recognition algorithm to process image data, extract key coordinate points of the face, and generate facial feature information.

[0054] In the embodiments of this application, the second facial blending shape weight is calculated based on the aforementioned facial feature information. The second facial blending shape weight reflects the state of the facial expression of the character or virtual human in the current image. Similar to the first facial blending shape weight in step 501, the second facial blending shape weight is a set of values ​​used to drive the facial expression of the virtual human or character. The second facial blending shape weight is extracted from image data, not from audio data. By extracting facial blending shape weights based on image data, facial blending shape weights corresponding to the facial expressions in the actual image can be obtained.

[0055] Step S304: Determine the second image data based on the second facial feature weights.

[0056] In the embodiments of this application, the data processing device can directly determine the second facial feature weight as the second image data, or it can determine the shape weight value corresponding to certain fixed key points in the second facial feature weight as the second image data. If the shape weight value corresponding to certain fixed key points in the first facial feature weight is determined as the second audio data, then correspondingly, the shape weight value corresponding to the fixed key points in the second facial feature weight is also determined as the second image data.

[0057] For example, if the shape weight value corresponding to the key point of the mouth part in the first facial feature weight is determined as the second image data, then the shape weight value of the key point of the eye part in the second facial feature weight is also determined as the second image data. Of course, the fixed key points can also be the eyes, cheeks, or other facial areas, and can be set according to actual needs and application scenarios. This application does not limit this.

[0058] In this way, both the first audio data and the second image data are converted into facial feature weights, thus transforming the data into the same dimension and laying the foundation for subsequent comparison.

[0059] In some embodiments, when performing step S302 above, the data processing device may also perform the following step: determining the weights in the first facial blending shape weights that match the mouth movements as the second audio data.

[0060] In the embodiments of this application, the weights matching mouth movements refer to selecting only those parts related to lip movements among all facial movements. Typically, mouth movements correspond to certain action units in a Facial Action Coding System (FACS), such as lip corner lifting and lower lip lowering, generally involving 17 key points. The data processing device can extract and filter the facial blending shape weights of certain action units in the Facial Action Coding System (FACS) to obtain the weights matching mouth movements.

[0061] In the embodiments of this application, the mouth is the most active and information-rich area in facial expressions, and its weight adjustment plays a core role in mouth shape expression. By obtaining the weight of the mouth blend shape, it is possible to accurately identify everything from subtle lip movements to exaggerated mouth shapes. Therefore, the weight based on the mouth movement is also the most representative of the entire facial expression, making the comparison based on the mouth movement weight more accurate.

[0062] Accordingly, when the data processing device performs the above step S504, it may also perform the following step: determine the weight in the second facial blending shape weight that matches the mouth movement as the second image data.

[0063] In embodiments of this application, similar to audio data, extracting a set of facial blending shape weights related to lip movements from image data refers to obtaining weights that match mouth movements. The purpose of this step is to filter out irrelevant facial expression information (such as blinking, frowning, etc.) and retain only speech-related action information. By filtering out irrelevant facial expression information and retaining only speech-related action information, data redundancy can be reduced, computational efficiency improved, and comparability with audio data enhanced.

[0064] In the embodiments of this application, mouth motion data in image data can be directly compared with mouth motion data extracted from audio data to determine their synchronization. Since both mouth motion data in image data and mouth motion data extracted from audio data are numerical representations based on facial motion units, they are directly comparable without the need for additional timestamp calibration.

[0065] In this way, both audio and image data are converted into facial motion data in a unified format, focusing on the key feature of mouth movements. By focusing on driving the mixed shape weights of the mouth region, precise automatic alignment of lip shape and sound is achieved. Simultaneously, by concentrating computational resources to analyze subtle lip movement features, complex articulation movements such as consonant plosives and vowel transitions in speech can be accurately reproduced, effectively avoiding the detail blurring caused by global facial processing. This targeted processing not only significantly improves the visual accuracy of lip-syncing but also demonstrates significant advantages in computational efficiency and anti-interference capabilities. This makes subsequent comparison processes more efficient and accurate, thus achieving automatic alignment of audio and image data.

[0066] In some embodiments, when performing step S103, the data processing device may perform the following steps S401 to S404: Step S401: Divide the second audio data and the second image data into multiple sets of audio data and multiple sets of image data according to the same time interval.

[0067] In the embodiments of this application, the same time interval refers to segmenting the second audio data and the second image data into segments with the same fixed duration (e.g., 50 milliseconds, 33 milliseconds). For example, if the segmentation interval is 50 milliseconds and the total duration of the second audio data and the second image data is 10 seconds, it can be divided into 200 groups. This ensures that the second audio data and the second image data have a unified time reference in the subsequent comparison process, facilitating accurate calculation of the time deviation between them.

[0068] For example, if the second image data has 600 image frames in 10 seconds, then the second audio data is also processed by sampling 60 audio data per second, resulting in 600 sets of audio data and 600 sets of video data.

[0069] Step S402: From multiple sets of image data, determine the target image data corresponding to each set of audio data.

[0070] In the embodiments of this application, the target image data refers to the most matching image data at the time corresponding to a set of audio data. Since there may be a certain delay between the second audio data and the second image data, it is necessary to find the frame image that is most relevant to the current audio data among multiple sets of image data.

[0071] For example, the data processing device can determine the target image data that matches each set of audio data based on the similarity of blend shape data of facial feature points (such as mouth shape) in audio data and image data, and then determine the target image data that matches each set of audio data in multiple sets of audio data based on the similarity.

[0072] Step S403: For each set of audio data, determine the time difference between it and the corresponding target image data to obtain multiple time differences corresponding to multiple sets of audio data.

[0073] In the embodiments of this application, the time difference refers to the time offset between each set of audio data and the corresponding image data. Since the sampling and division of audio and image data are based on time, and multiple sets of audio and image data are actually time-series data, each set of audio or image data will also have a corresponding timestamp. Therefore, for each set of audio data, the time difference between it and the corresponding target data can be obtained. For example, if a set of audio data corresponds to the sound signal at 1.5 seconds, and the image data matched by this set of audio data is a scene captured at 1.45 seconds, then there is a time difference of 0.05 seconds between these two sets of data.

[0074] For example, if the third set of audio data corresponds to the first set of image data, the time difference between the first set of audio data and the second set of audio data can be set to the same time difference as the time difference between the third set of audio data and the first set of image data. In this way, multiple time differences corresponding to multiple sets of audio data can be obtained.

[0075] In the embodiments of this application, by calculating the time difference of multiple sets of data, more comprehensive synchronization information can be obtained, rather than relying on the judgment of a single time point. This method improves robustness and is especially suitable for situations where the video content has slight jitter or discontinuous playback.

[0076] Step S404: Determine the data time difference based on multiple time differences.

[0077] In the embodiments of this application, the data time difference refers to the final calculated overall time deviation value. The data time difference is used to adjust the playback timing of the second audio data and the second image data. The data time difference can be obtained by statistically processing multiple time differences (such as taking the average, weighted average, or median). Statistical processing can eliminate the influence of individual outliers and improve the reliability of the results. Of course, multiple time differences can also be directly determined as the data time difference, so that there is a time difference for each group of data, which can improve accuracy.

[0078] In the embodiments of this application, the second audio data and the second image data are grouped at equal time intervals, and the matching relationship between each group of audio data and image data is compared one by one. The time difference between each group is calculated and then statistically processed to obtain the overall audio-visual time difference. In this way, the delay between the second audio data and the second image data can be accurately identified, and the playback rhythm in the playback software can be adjusted accordingly to achieve audio-visual synchronization.

[0079] This reduces the lip-syncing issue that users experience when watching virtual human animations or regular videos, thereby improving the user experience and enhancing product competitiveness.

[0080] In some embodiments, when performing step S602 above, the data processing device may also perform the following steps S501 and S502: Step S501: For each set of audio data, compare it with multiple sets of image data to obtain the corresponding multiple similarities.

[0081] In the embodiments of this application, in this step, similarity refers to the degree of matching between the blend shape data converted from the audio signal and the blend shape data extracted from the image signal, calculated by the algorithm. The higher the similarity, the closer the timing of the lip-sync expression between the blend shape data converted from the audio signal and the blend shape data extracted from the image signal. For example, mathematical methods such as Euclidean distance and cosine similarity are used to quantify the temporal alignment between the blend shape data converted from the audio signal and the blend shape data extracted from the image signal.

[0082] For example, the data processing device can use a cyclic comparison method to compare multiple sets of audio data and multiple sets of image data. If there are 100 sets of audio data and 100 sets of video data, then the similarity obtained is 100*100, with each set of audio data and 100 sets of image data having multiple similarities of 100.

[0083] In the embodiments of this application, by calculating the similarity between the Blend Shape data converted from the audio signal and the Blend Shape data extracted from the image signal, the system can accurately identify which image data and audio data have a high temporal matching, thereby providing a basis for subsequent audio-visual synchronization adjustment.

[0084] Step S502: For each set of audio data, identify the image data with similarity values ​​greater than a preset threshold from the multiple sets of image data as the corresponding target image data.

[0085] In the embodiments of this application, the preset threshold is a judgment standard set according to the actual application scenario, used to filter out image data that has a sufficiently high similarity to audio data as target image data. For example, the preset threshold can be set to 0.9. Only when the similarity is greater than or equal to 0.9 is the corresponding image data considered to be consistent with the audio data in time. By setting a reasonable preset threshold, invalid matches with low similarity can be filtered out.

[0086] In embodiments of this application, if multiple sets of image data have similarities exceeding a preset threshold, adjacent sets of audio data can be used for judgment to determine the corresponding target image data. For example, if the first set of audio data corresponds to the fifth and second set of image data, and the second set of audio data corresponds to the third set of image data, then it can be inferred that the first set of audio data matches the second set of image data, and the second set of image data is the target image data corresponding to the first set of audio data.

[0087] In this way, by comparing each set of audio data with multiple sets of image data and calculating the similarity, and by filtering the image data based on a preset threshold to obtain the target image data, a more accurate audio-visual synchronization effect can be achieved, thereby reducing the audio-visual desynchronization phenomenon during playback and ultimately improving the user's viewing experience.

[0088] In some embodiments, when performing step S104 above, the data processing device may perform the following steps: determining the frame difference between the first audio data and the first image data based on the data time difference.

[0089] In the embodiments of this application, the frame rate difference refers to the number of frames corresponding to the time difference between the first audio data and the first image data during playback due to acquisition, transmission, or processing delays. For example, if the first audio data is 0.5 seconds faster than the first image data, and there are 30 frames per second, then the frame rate difference is 15 frames. The calculation of the frame rate difference depends on the system clock synchronization mechanism and the precise control of the sampling rate of the first audio data and the frame rate of the first image data.

[0090] For example, if the data time difference is the overall time difference between the first audio data and the first image data (e.g., determined based on the mode, mean, etc. of multiple time differences), then the frame difference between the first audio data and the first image data can be determined based on the data time difference and the number of frames corresponding to the data time difference. If the data time difference includes multiple time differences, then a frame difference will be determined for each group of audio data in the first audio data, resulting in multiple frame differences.

[0091] In some embodiments, when performing the "aligning the first audio data and the first image data" step S104 above, the data processing device may perform the following steps: based on the frame difference, compensate and adjust the first audio data to obtain compensated audio data.

[0092] In embodiments of this application, compensation adjustment is a method that synchronizes the first audio data with the first image data in time by increasing or decreasing the length of the first audio data or changing its playback speed. For example, when the first audio data precedes the first image data, the two can be aligned by decreasing the playback speed of the first audio data or cropping a portion of the first audio data. Conversely, if the first image data precedes the first audio data, the length of the first audio data can be increased or its playback speed increased to match the first image data.

[0093] For example, if the frame difference between the first audio data and the first image data is a statistically calculated overall frame difference, such as the first audio data being 15 frames ahead of the first image data, then it can be set to compensate 15 frames for the first 90 frames, and the first audio data can be compensated as a whole.

[0094] An exemplary compensation method could be to advance one frame every six frames in the first audio data, thus completing the compensation within the first 90 frames, or to advance the first frame of the first audio data by six frames.

[0095] In the embodiments of this application, the compensation and adjustment method should, as far as possible, ensure that the quality of the first audio data is not affected, and avoid noise or distortion caused by the adjustment. A common practice is to use an interpolation algorithm to smoothly transition changes in the first audio data signal, thereby reducing auditory abruptness.

[0096] In the embodiments of this application, by compensating and adjusting the first audio data based on the frame difference, the first audio data can be aligned with the first image data in time, thereby improving the synchronization between the first audio data and the first image data, and thus enhancing the user's viewing experience.

[0097] In some embodiments, when performing the "aligning the first audio data and the first image data" step S104 above, the data processing device may perform the following steps: based on the frame difference, compensate and adjust the first image data to obtain compensated image data.

[0098] In the embodiments of this application, the compensation adjustment of the first image data typically includes methods such as inserting adjacent image frames, skipping certain image frames, or adjusting the playback speed of the first image data. For example, if the first image data lags behind the first audio data by 15 frames, the first image data can be synchronized with the first audio data by repeatedly playing the most recent frames of the first image data. Furthermore, a frame-by-frame fine-tuning approach can be used to gradually adjust the playback rhythm of the first image data, making the changes in the first image data more natural.

[0099] In the embodiments of this application, the first image data compensation and adjustment method should be designed with consideration for screen smoothness and visual continuity in order to avoid screen stuttering or jumping due to the implementation of the method. For example, the first image data compensation and adjustment method can be combined with an intelligent prediction algorithm to predict the screen content of the next frame and generate an intermediate frame to fill the gap.

[0100] For example, if the frame difference between the first audio data and the first image data is a statistically calculated overall frame difference, such as the first audio data being 15 frames ahead of the first image data, then it can be set to compensate 15 frames for the first 90 frames, and the first image data can be compensated as a whole.

[0101] An exemplary compensation method could be to compensate one frame every six frames in the first image data, thus completing the compensation in the first 90 frames, or to compensate these six frames directly in the first frame of the first image data.

[0102] In this way, by determining the frame difference based on the data time difference and compensating and adjusting the first audio data or the first image data according to the frame difference, the first audio data and the first image data can be accurately aligned, thereby effectively solving the problem of asynchrony between the first audio data and the first image data, and thus improving the presentation quality of multimedia content.

[0103] In the embodiments of this application, compensation can also be applied to the first image data and the first audio data simultaneously. For example, if the first image data is 15 frames ahead of the first audio data, then the first image data can be pushed back by 7 frames and the first audio data can be advanced by 8 frames. Alternatively, the first image data can be pushed back by 5 frames and the first audio data can be advanced by 10 frames. Of course, other combinations are also possible, as long as these 15 frames are made up.

[0104] In some embodiments, when the data processing device performs the above step: "compensate and adjust the first audio data based on the frame difference to obtain compensated audio data", it may also perform the following steps: compensate and adjust each group of audio data in the first audio data based on the frame difference to obtain each group of audio data after compensation and adjustment, and obtain compensated audio data based on each group of audio data after compensation and adjustment.

[0105] For example, if each group of audio data in the first audio data is determined to have a frame difference, compensation can be made for each group of audio data. If the first group of audio data differs from the second group of image data by 15 frames, the second group of audio data differs from the third group of image data by 15 frames, and so on, then compensating the first group of audio data by 15 frames will compensate the subsequent audio data.

[0106] For example, if each group of audio data in the first audio data is determined to have a frame difference, then for each group of audio data, if the first group of audio data is 15 frames ahead of the second group of image data, the second group of audio data is 16 frames ahead of the third group of image data, and all subsequent groups are 16 frames ahead, then the first group of audio data is compensated by 15 frames and the second group of audio data is compensated by 1 frame.

[0107] For example, if each group of audio data in the first audio data is determined to have a frame difference, compensation can be made for each group of audio data. If the first group of audio data is 15 frames ahead of the second group of image data, the second group of audio data is 16 frames ahead of the third group of image data, and each subsequent group of audio data differs from the corresponding image data by a different number of frames, then corresponding supplementation can be made based on the aforementioned compensation situation. For example, the first group of audio data is compensated for 15 frames, the second group of audio data is compensated for 1 frame, and the number of frames for subsequent compensation needs to include the already compensated 16 frames.

[0108] In some embodiments, when the data processing device performs the above step: "compensate and adjust the first image data based on the frame difference to obtain compensated audio data", it may also perform the following steps: compensate and adjust each group of audio data in the first image data based on the frame difference to obtain each group of compensated and adjusted image data, and obtain compensated image data based on each group of compensated and adjusted image data.

[0109] For example, if each group of image data in the first image data is determined to have a frame difference, compensation can be made for each group of image data. If the first group of image data differs from the second group of audio data by 15 frames, the second group of image data differs from the third group of audio data by 15 frames, and so on, then compensating the first group of image data by 15 frames will compensate the subsequent image data.

[0110] For example, if each group of image data in the first image data is determined to have a frame difference, compensation can be made for each group of image data. If the first group of image data differs from the second group of audio data by 15 frames, the second group of image data differs from the third group of audio data by 16 frames, and the subsequent groups differ by 16 frames, then the first group of image data can be compensated by 15 frames, and the second group of image data can be compensated by 1 frame.

[0111] For example, if each group of image data in the first image data is determined to have a frame difference, compensation can be made for each group of image data. If the first group of image data differs from the second group of audio data by 15 frames, and the second group of image data differs from the third group of audio data by 16 frames, and each subsequent group of image data differs from the corresponding audio data by a different number of frames, then corresponding supplementation can be made based on the aforementioned compensation situation. For example, the first group of image data is compensated for 15 frames, the second group of image data is compensated for 1 frame, and the subsequent compensation frame count needs to include the already compensated 16 frames.

[0112] In some embodiments, the data processing device may further perform the following steps: generating aligned video data based on aligned first audio data and aligned second image data, and playing the aligned video data.

[0113] In the embodiments of this application, generating aligned video data refers to merging aligned first audio data and aligned second image data into a complete video stream. The generated aligned video data is not only synchronized in time but also coordinated in space, thereby creating a natural and smooth audiovisual experience.

[0114] In the embodiments of this application, a processing system generates aligned video data based on aligned first audio data and aligned second image data, and then plays the aligned video data. This achieves precise synchronization between audio and images, thereby improving the naturalness and realism of the video, and ultimately enhancing the user's viewing experience and interactive effects.

[0115] For example, such as Figure 4 As shown, a flowchart of an exemplary data processing method is provided, including steps S701 to S703: Step S701: Obtain data.

[0116] Here, for the scenario of video source files, such as Figure 2As shown, video source file 21 is decoded (22) to obtain image data, and video source file 101 is decoded (23) to obtain audio data. Then, BS data (corresponding to the second image data) is output from the image data (corresponding to the first image data) using the FE algorithm; and BS data (corresponding to the second audio data) is identified from the audio data (corresponding to the first audio data) using the A2F algorithm. At this point, there will be no decoding deviation. For virtual human animation scenes, such as... Figure 3 As shown, the virtual animation image after driving and rendering 33 is captured to obtain the first image data, and then the FE algorithm is used to output BS data (corresponding to the pre-made BS in the figure). At this time, there is no delay in the generation of virtual human animation. Then, the audio data is directly converted into BS data (corresponding to the recognized BS in the figure) through the A2F algorithm.

[0117] Step S702: Compare the data.

[0118] Here, multiple sets of BS data are collected, and 17 data points (float data between 0 and 1) of the mouth are selected from the BS data. A cyclic comparison method is used to calculate that data with a similarity greater than 90% between two sets of data are considered to be data from the same time. Multiple sets of data are compared and statistically analyzed to improve the accuracy of the comparison. The two sets of data that are calculated to be from the same time are marked.

[0119] Step S703: Playback data.

[0120] Here, audio and video signals are played according to the markers, the frame difference between the two markers is calculated, and compensation adjustments are made. The compensation method uses frame-by-frame fine-tuning, and the adjustment is completed after multiple frames. This is used to reduce abrupt changes in the image or sound.

[0121] In the above method, the collection time for multiple sets of BS data can be set according to requirements. This method can reduce the sampling rate of BS data and save computing power.

[0122] This application provides a data processing method, which includes: acquiring first audio data and first image data corresponding to the same time period; extracting feature data containing the same object from the first audio data and first image data respectively to obtain corresponding second audio data and second image data; comparing the second audio data and second image data to determine the data time difference between the first audio data and the first image data; and aligning the first audio data and the first image data based on the data time difference. The data processing method provided in this application, by extracting feature data of the same object from the first audio data and first image data respectively and performing comparative analysis, calculates the delay between the first audio data and the first image data, and performs compensation adjustments accordingly, achieving audio-visual alignment without relying on timestamps. This data processing method is applicable to various scenarios such as virtual human animation and video playback, and can significantly improve the accuracy of audio-visual synchronization and playback quality in these scenarios.

[0123] like Figure 5 As shown, this application embodiment provides a data processing device 5, which includes: The acquisition module 51 is used to acquire the first audio data and the first image data corresponding to the same time period; Extraction module 52 is used to extract feature data containing the same object from the first audio data and the first image data respectively, to obtain the corresponding second audio data and second image data; Comparison module 53 is used to compare the second audio data and the second image data to determine the data time difference between the first audio data and the first image data; Alignment module 54 is used to align the first audio data and the first image data based on the data time difference.

[0124] In one embodiment of this application, the acquisition module 51 is further configured to acquire video source data, and acquire audio data to be aligned and image data to be aligned from the video source data, or acquire audio data to be aligned and generate image data to be aligned based on the audio data to be aligned; sample data of the same time period from the audio data to be aligned and the image data to be aligned to obtain corresponding first audio data and first image data.

[0125] In one embodiment of this application, the comparison module 53 is further configured to divide the second audio data and the second image data into multiple sets of audio data and multiple sets of image data according to the same time interval; determine the target image data corresponding to each set of audio data from the multiple sets of image data; determine the time difference between each set of audio data and the corresponding target image data to obtain multiple time differences corresponding to the multiple sets of audio data; and determine the data time difference based on the multiple time differences.

[0126] In one embodiment of this application, the comparison module 53 is further configured to compare each set of audio data with multiple sets of image data to obtain multiple similarities; and for each set of audio data, determine the image data with multiple similarity values ​​greater than a preset threshold in the multiple sets of image data as the corresponding target image data.

[0127] In one embodiment of this application, the extraction module 52 is further configured to extract audio features from the first audio data to obtain audio feature information, and map the audio feature information to a facial morphology parameter space to obtain a first facial feature weight; determine the second audio data based on the first facial feature weight; extract facial key points from the first image data to obtain facial feature information, and generate a second facial feature weight based on the facial expression represented by the facial feature information; and determine the second image data based on the second facial feature weight.

[0128] In one embodiment of this application, the extraction module 52 is further configured to determine the weights in the first facial blending shape weights that match the mouth movements as second audio data; and to determine the weights in the second facial blending shape weights that match the mouth movements as second image data.

[0129] In one embodiment of this application, the alignment module 54 is further configured to determine the frame difference between the first audio data and the first image data based on the data time difference; to compensate and adjust the first audio data based on the frame difference to obtain compensated audio data; and to compensate and adjust the first image data based on the frame difference to obtain compensated image data.

[0130] In one embodiment of this application, the alignment module 54 is further configured to compensate and adjust each group of audio data in the first audio data based on the frame difference, to obtain each group of audio data after compensation and adjustment, and to obtain compensated audio data based on each group of audio data after compensation and adjustment; and to compensate and adjust each group of image data in the first image data based on the frame difference, to obtain each group of image data after compensation and adjustment, and to obtain compensated image data based on each group of image data after compensation and adjustment.

[0131] In one embodiment of this application, the alignment module 54 is further configured to generate aligned video data based on the aligned first audio data and the aligned second image data, and to play the aligned video data.

[0132] like Figure 6 As shown, this application embodiment provides a data processing device 6, including: a processor 61 and a memory 62; The memory 62 is used to store audio data and image data; the processor 61 is used to retrieve first audio data and first image data corresponding to the same time period from the memory; extract feature data containing the same object from the first audio data and first image data respectively to obtain corresponding second audio data and second image data; compare the second audio data and second image data to determine the data time difference between the first audio data and the first image data; and align the first audio data and the first image data based on the data time difference.

[0133] This application provides a computer-readable storage medium storing one or more computer programs, which can be executed by one or more processors to implement the above-described data processing method. The computer-readable storage medium can be transient or non-transient.

[0134] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described data processing method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied as a computer storage medium; in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0135] In some embodiments, the storage medium may be a computer-readable storage medium, which may be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), ferromagnetic random access memory (FRAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disk-read-only memory (CD-ROM); or it may be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0136] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0137] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxesFigure 1 A device that provides the functions specified in one or more boxes.

[0138] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0140] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0141] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0142] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

[0143] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0144] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0145] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A data processing method, comprising: obtaining first audio data and first image data corresponding to a same time period; extracting feature data containing a same object from the first audio data and the first image data respectively to obtain corresponding second audio data and second image data; comparing the second audio data and the second image data to determine a data time difference between the first audio data and the first image data; and aligning the first audio data and the first image data based on the data time difference.

2. The data processing method of claim 1, wherein the obtaining first audio data and first image data corresponding to a same time period comprises: obtaining video source data, and obtaining to-be-aligned audio data and to-be-aligned image data from the video source data, or obtaining to-be-aligned audio data and generating the to-be-aligned image data based on the to-be-aligned audio data; sampling data of a same time period from the to-be-aligned audio data and the to-be-aligned image data to obtain corresponding first audio data and first image data.

3. The data processing method of claim 1, wherein the comparing the second audio data and the second image data to determine a data time difference between the first audio data and the first image data comprises: dividing the second audio data and the second image data into multiple groups of audio data and multiple groups of image data respectively according to a same time interval; determining, from the multiple groups of image data, target image data corresponding to each group of audio data in the multiple groups of audio data; determining, for each group of audio data, a time difference between the corresponding target image data to obtain multiple time differences corresponding to the multiple groups of audio data; and determining the data time difference based on the multiple time differences.

4. The data processing method of claim 3, wherein the determining, from the multiple groups of image data, target image data corresponding to each group of audio data in the multiple groups of audio data comprises: comparing, for each group of audio data, the multiple groups of image data to obtain multiple similarity values corresponding thereto; and determining, for each group of audio data, image data corresponding to a similarity value greater than a preset threshold in the multiple groups of image data as the corresponding target image data.

5. The data processing method of claim 1, wherein the extracting feature data containing a same object from the first audio data and the first image data respectively to obtain corresponding second audio data and second image data comprises: performing audio feature extraction on the first audio data to obtain audio feature information, and mapping the audio feature information to a face shape parameter space to obtain first face feature weights; determining the second audio data based on the first face feature weights; performing face key point extraction on the first image data to obtain face feature information, and generating second face feature weights based on a face expression represented by the face feature information; and determining the second image data based on the second face feature weights.

6. The data processing method of claim 5, wherein the determining the second audio data based on the first facial blend shape weight comprises: determining a weight in the first facial blend shape weight that matches a mouth movement as the second audio data. Accordingly, the determining the second image data based on the second facial blend shape weight comprises: determining a weight in the second facial blend shape weight that matches the mouth movement as the second image data.

7. The data processing method of claim 3, wherein the aligning the first audio data and the first image data based on the data time difference comprises: determining a frame number difference between the first audio data and the first image data based on the data time difference; and the aligning the first audio data and the first image data comprises at least one of: compensating the first audio data based on the frame number difference to obtain compensated audio data; and compensating the first image data based on the frame number difference to obtain compensated image data.

8. The data processing method of claim 7, wherein the method comprises at least one of: the compensating the first audio data based on the frame number difference to obtain compensated audio data comprises: compensating each group of audio data in the first audio data based on the frame number difference to obtain compensated each group of audio data, and obtaining the compensated audio data based on the compensated each group of audio data; and the compensating the first image data based on the frame number difference to obtain compensated image data comprises: compensating each group of image data in the first image data based on the frame number difference to obtain compensated each group of image data, and obtaining the compensated image data based on the compensated each group of image data.

9. The data processing method of any one of claims 1 to 8, further comprising: generating aligned video data based on the aligned first audio data and the aligned second image data, and playing the aligned video data.

10. A data processing device comprising: a processor and a memory; the memory is configured to store audio data and image data; the processor is configured to: obtain first audio data and first image data corresponding to a same time period from the memory; extract feature data containing a same object from the first audio data and the first image data respectively to obtain second audio data and second image data; compare the second audio data and the second image data to determine a data time difference between the first audio data and the first image data; and align the first audio data and the first image data based on the data time difference. ​

Images