Audio and video synchronization detection method and device, storage medium and computer program product
By acquiring and comparing the frame numbers corresponding to the audio signal frequency and video image content, the synchronization of audio and video data is automatically judged, and the problems of low efficiency and low accuracy of audio and video synchronization detection in the prior art are solved, and efficient and accurate Synchronous detection.
Patent Information
- Application Number
- CN202411013389.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art audio and video synchronization detection efficiency is low and relies on manual detection, and the accuracy is not high, so it cannot effectively solve the audio and video synchronization problem of multimedia equipment.
By obtaining the target audio data and video data in the preset test media data, determine the frame number corresponding to the audio signal frequency and the frame number corresponding to the video image content, and automatically compare the frame number consistency. If it is inconsistent, judge that the data is not synchronized and improve detection Efficiency and accuracy.
It realizes the automation of audio and video synchronous detection, improves detection efficiency and accuracy, and reduces dependence on artificial sensitivity and experience.
Smart Images

Figure CN118945390A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to an audio and video synchronization detection method, device, storage medium and computer program product. Background Art
[0002] With the development of multimedia technology, the application of multimedia devices has penetrated into various fields of human life, such as communication, industry, medicine, teaching, etc., bringing great convenience to people's lives. In addition, multimedia devices often involve the processing of audio and video, which easily leads to the problem of audio and video being out of sync. Therefore, measuring the synchronization effect of audio and video of multimedia devices has become one of the evaluation indicators of multimedia devices.
[0003] Currently, the audio and video output by multimedia devices are manually watched to manually detect whether the output audio and video are synchronized. It can be seen that this manual detection method is time-consuming and labor-intensive, has low detection efficiency, and depends on personal experience and sensitivity to audio and video synchronization, and has low detection accuracy.
[0004] Therefore, how to improve the detection efficiency and detection accuracy of audio and video synchronization detection is a technical problem that needs to be solved urgently in this technical field. Summary of the invention
[0005] The main purpose of this application is to provide an audio and video synchronization detection method, device, storage medium and computer program product, aiming to solve the technical problem of how to improve the detection efficiency and detection accuracy of audio and video synchronization detection.
[0006] To achieve the above object, the present application proposes an audio and video synchronization detection method, the audio and video synchronization detection method comprising:
[0007] Acquire target audio data and target video data to be synchronously detected, wherein the target audio data and the target video data are data output when a preset test end plays preset test media data, the preset test media data includes test audio data and test video data, the signal frequency of each frame of audio signal in the test audio data is different, and the image content of each frame of video image in the test video data is different;
[0008] For each frame of the target audio signal in the target audio data, determining a first theoretical frame number corresponding to the signal frequency of the target audio signal;
[0009] Determine a target video image corresponding to a target audio signal in the target video data, and determine a second theoretical frame number corresponding to an image content of the target video image, wherein the target audio signal is consistent with a timestamp of the target video image;
[0010] If the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchronism between the target audio data and the target video data.
[0011] In one embodiment, before the step of determining, for each frame of the target audio signal in the target audio data, a first theoretical frame number corresponding to the signal frequency of the target audio signal, the method further comprises:
[0012] Detecting all transition timestamps contained in the target audio data, wherein the transition timestamps include timestamps of rising edges of pulse signals in the target audio data;
[0013] The target audio data is divided into at least one frame of target audio signal according to each of the jump time stamps, wherein the target audio data between two adjacent jump time stamps constitutes one frame of the target audio signal.
[0014] In one embodiment, after the step of determining a second theoretical frame number corresponding to the image content of the target video image, the method further includes:
[0015] Determine an error time corresponding to the target audio signal based on the first theoretical frame number and the second theoretical frame number;
[0016] Combining the error times corresponding to the target audio signals to obtain an error value queue, and determining the total number of error times in the error value queue that are greater than or equal to a preset time threshold;
[0017] If the total number is greater than a preset threshold, the preset test end is determined to be an unqualified product.
[0018] In one embodiment, the step of determining the error time corresponding to the target audio signal based on the first theoretical frame number and the second theoretical frame number includes:
[0019] Calculating an absolute value of a frame number difference between the first theoretical frame number and the second theoretical frame number, and calculating a product of the absolute value of the frame number difference and a preset frame duration to obtain a calculation result;
[0020] The calculation result is determined to be an error time corresponding to the target audio signal.
[0021] In one embodiment, before the step of obtaining the target audio data and the target video data to be synchronously detected, the method further includes:
[0022] Generating test audio data, wherein the signal frequency of each frame of audio signal in the test audio data is different;
[0023] Generate test video data, wherein the frame rate and number of frames of the test video data and the test audio data are consistent, and the image content of each frame of the video image in the test video data is different;
[0024] The test audio data and the test video data are synthesized to obtain preset test media data.
[0025] In one embodiment, the step of generating test audio data includes:
[0026] Acquire preset reference audio data, wherein the reference audio data includes a preset frame reference audio signal;
[0027] For each frame of the reference audio signal in the reference audio data, determine a corresponding signal frequency based on the frame number of the reference audio signal, and frequency-modulate the reference audio signal based on the signal frequency to obtain a frequency-modulated audio signal, wherein the signal frequency is positively correlated with the frame number of the audio signal;
[0028] Each frame of the FM audio signal is combined to obtain test audio data.
[0029] In one embodiment, the step of generating test video data includes:
[0030] Generate a frame of video image containing a target number of preset graphics in sequence until the preset frame of the video image is generated, wherein the target number is positively correlated with the frame number of the video image;
[0031] Each frame of the video image is combined to obtain test video data.
[0032] In addition, to achieve the above-mentioned purpose, the present application also provides an audio and video synchronization detection device, which includes an image acquisition unit, a sound pickup unit and a processor, the image acquisition unit and the sound pickup unit are respectively connected to the processor, and the processor is used to execute the steps of the audio and video synchronization detection method as described above.
[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the audio and video synchronization detection method described above are implemented.
[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the audio and video synchronization detection method described above are implemented.
[0035] One or more technical solutions proposed in this application have at least the following technical effects:
[0036] Obtain target audio data and target video data to be synchronously detected, wherein the target audio data and the target video data are data output when a preset test end plays preset test media data, and the preset test media data include test audio data and test video data, and the signal frequency of each frame of audio signal in the test audio data is different, and the image content of each frame of video image in the test video data is different; thus, there is a one-to-one correspondence between the frame number and the signal frequency of the audio signal in the test audio data, so that the corresponding frame number can be determined based on the signal frequency, and similarly, there is a one-to-one correspondence between the frame number and the image content of the video image in the test video data, so that the corresponding frame number can be determined based on the image content. For each frame of the target audio signal in the target audio data, determine the first theoretical frame number corresponding to the signal frequency of the target audio signal; determine the target video image corresponding to the target audio signal in the target video data, and determine the second theoretical frame number corresponding to the image content of the target video image, wherein the timestamps of the target audio signal and the target video image are consistent; if the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchrony between the target audio data and the target video. For the target audio signal and the target video image belonging to the same timestamp, if the data is synchronized, under the constraint of consistent timestamps, the corresponding frame numbers of the two should also be consistent, that is, the first theoretical frame number is consistent with the second theoretical frame number. Conversely, if the data is not synchronized, the first theoretical frame number and the second theoretical frame number will also be inconsistent. Based on this, the theoretical frame numbers corresponding to the target audio signal and the target video image with consistent timestamps are obtained. When the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchrony between the target audio data and the target video. The entire detection process is completed automatically without manual participation, thereby improving the detection efficiency of audio and video synchronization detection. Compared with the manual detection method, it does not rely on subjective sensitivity and personal experience, thereby improving the detection accuracy of audio and video synchronization detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0039] Figure 1 A flowchart of the first embodiment of the audio and video synchronization detection method of the present application is provided;
[0040] Figure 2 A schematic diagram of a scenario of an existing audio and video synchronization detection method;
[0041] Figure 3 This is a schematic diagram of the audio and video data collection scenario of the audio and video synchronization detection method of this application;
[0042] Figure 4 A schematic diagram of a brief process of generating preset media data involved in an embodiment of an audio and video synchronization detection method of the present application;
[0043] Figure 5 A schematic diagram of test audio data involved in an embodiment of an audio and video synchronization detection method of the present application;
[0044] Figure 6 This is a video image simulation experiment diagram related to an embodiment of the audio and video synchronization detection method of the present application;
[0045] Figure 7 This is a schematic diagram of target audio data involved in an embodiment of the audio and video synchronization detection method of the present application;
[0046] Figure 8 This is a schematic diagram of a brief process involved in an embodiment of an audio and video synchronization detection method of the present application;
[0047] Fig. 9 It is a structural diagram of an audio and video synchronization detection device of the hardware operating environment involved in the audio and video synchronization detection method in the embodiment of the present application.
[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0049] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.
[0050] Audio and video synchronization is also known as lip synchronization, which means that the playback time of the image and sound is consistent during the playback process, and it is perceived that the image and sound are emitted at the same time. In reality, since audio and video signals have different compression and encoding standards, multiple media are actually independent of each other during playback, which may cause time errors during playback and affect the viewing experience. Streaming media is also easily affected by bandwidth and the computing power of the playback device during network transmission, resulting in audio and video freezes.
[0051] The lack of synchronization between audio and video will seriously affect the user's viewing experience. If the lip movements and sounds of the characters in the picture are not synchronized, the user will feel very uncomfortable and have cognitive problems. The problem of audio and video synchronization may occur in many fields. For example, in TV broadcasting, live streaming, video conferencing and other scenarios, maintaining synchronization is an important issue in audio and video processing technology.
[0052] When solving the problem of audio and video synchronization, some synchronization strategies are usually adopted, such as timestamp-based synchronization strategy, frame rate-based synchronization strategy, etc. The goal of these strategies is to make the audio and video data as consistent as possible during playback, but the synchronization of the input audio and video during playback cannot guarantee that the output audio and video after the device plays is also synchronized.
[0053] Most people are less sensitive to video freezes than audio freezes. For continuously played videos, freezes are generally noticeable if the error exceeds -125 to 45ms, while audio freezes only take 15ms, and some sensitive people can even detect 5ms freezes.
[0054] Therefore, before multimedia devices leave the factory, users need to check whether the audio and video output of the product are synchronized. The current audio and video synchronization detection methods are as follows: Figure 2 As shown, the multimedia device to be tested plays a piece of audio and video data, and humans watch and listen to whether the audio and video output by the device are synchronized. This manual detection method requires a lot of manpower and has low detection efficiency. It is heavily dependent on personal sensitivity to audio and video, has high requirements on personnel capabilities, and has low detection accuracy.
[0055] Based on this, the main solution of the present application is: obtain the target audio data and target video data to be synchronously detected, wherein the target audio data and the target video data are data output when the preset test end plays the preset test media data, and the preset test media data includes test audio data and test video data, and the signal frequency of each frame of audio signal in the test audio data is different, and the image content of each frame of video image in the test video data is different; for each frame of target audio signal in the target audio data, determine the first theoretical frame number corresponding to the signal frequency of the target audio signal; determine the target video image corresponding to the target audio signal in the target video data, and determine the second theoretical frame number corresponding to the image content of the target video image, wherein the timestamp of the target audio signal is consistent with that of the target video image; if the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchrony between the target audio data and the target video data.
[0056] The present application obtains the theoretical frame number corresponding to the target audio signal and the target video image with consistent timestamps. When the first theoretical frame number is inconsistent with the second theoretical frame number, it determines that there is data asynchrony between the target audio data and the target video. The entire detection process is completed automatically without human participation, thereby improving the detection efficiency of audio and video synchronization detection. Compared with manual detection methods, it does not rely on subjective sensitivity and personal experience, thereby improving the detection accuracy of audio and video synchronization detection.
[0057] It should be noted that the execution subject of each embodiment of the audio and video synchronization detection method of the present application can be a computing service device that realizes the above functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an audio and video synchronization detection device that can realize the above functions, such as a television, a watch, a game console, a navigation device, etc. The embodiments of the audio and video synchronization detection method of the present application do not make specific restrictions on this.
[0058] Based on this, the first embodiment of the audio and video synchronization detection method of the present application is proposed below. Figure 1 , Figure 1 This is a flow chart of the first embodiment of the audio and video synchronization detection method of the present application.
[0059] In this embodiment, the audio and video synchronization detection method includes steps S10 to S40:
[0060] Step S10, obtaining target audio data and target video data to be synchronously detected, wherein the target audio data and the target video data are data output when a preset test end plays preset test media data, the preset test media data includes test audio data and test video data, the signal frequency of each frame of audio signal in the test audio data is different, and the image content of each frame of video image in the test video data is different;
[0061] The preset test end can be any device that can output audio and video data, such as a television, a computer, a tablet computer, a mobile phone, a watch, a game console, etc., and this embodiment does not impose specific restrictions on this. It should be noted that the preset test end can specifically be a multimedia device that needs to detect whether the audio and video output is qualified. In this embodiment, whether the audio and video output is qualified refers to whether the audio and video data output after the synchronized audio and video data is played by the device is still synchronized. In a specific implementation, before TV A is put on the market, the tester needs to test whether the audio and video output of TV A is synchronized, and the preset test media data can be played through TV A.
[0062] The preset test media data includes test audio data and test video data. It should be noted that the test audio data and the test video data are time-synchronized data. The test audio data includes at least one frame of audio signal. The signal frequency of each frame of audio signal is different. In this way, the frame number of the audio signal corresponds to the signal frequency one by one, so that the corresponding frame number can be determined by the signal frequency. Similarly, the test video data includes at least one frame of video image. The image content of each frame of video image is different. In this way, the frame number of the video image corresponds to the image content one by one, so that the corresponding frame number can be determined by the image content. It is easy to understand that the frame number refers to the frame number, which is used to identify the frame number of all data that the current data is. For example, if the frame number of a frame of audio signal in the test audio data is i, then the audio signal is the i-th frame of audio signal in the test audio data. For example, if the frame number of a frame of video image in the test video data is s, then the video image is the s-th frame of video image in the test video data.
[0063] It should be noted that, considering that the audio signal is easily affected during the transmission process, thereby affecting the characteristics such as amplitude and phase, but the signal frequency is relatively stable and usually does not change during the transmission process. Based on this, this embodiment selects the signal frequency as the basis for determining the frame number, and the signal frequency of each frame of the audio signal in the test audio data is different, so that the corresponding frame number can be determined based on the signal frequency.
[0064] It should be noted that video images are easily affected by external brightness during transmission, and high-frequency parts of the image are easily lost. Based on this, the image content of the target video image can be specifically a simple graphic block (such as a rectangular block, a circular block, an elliptical block, etc.), and different image contents can specifically be different numbers of graphic blocks.
[0065] The preset test terminal plays the preset test media data. Specifically, the preset test media data can be input into the preset test terminal, and the test audio data in the preset test media data can be played through the first output unit in the preset test terminal, and the test video data in the preset test media data can be played through the second output unit in the preset test terminal. The first output unit includes but is not limited to a speaker, and the second output unit includes but is not limited to a display.
[0066] After the preset test end plays the preset test media data, the data output by the preset test end can be picked up to obtain the target audio data and the target video data. Figure 3 As shown, the video data output by the preset test end is picked up by the camera, and the audio data output by the preset test end is picked up by the microphone. The video data picked up by the camera is the target video data, and the audio data picked up by the microphone is the target audio data.
[0067] It should be noted that the target audio data and the target video data are time-synchronized data. Specifically, the time-synchronized target audio data and the target video data can be obtained by synchronously picking up the audio data and the video data output by the preset test end. Alternatively, after picking up the audio data and the video data output by the preset test end, the picked up audio data and the video data can be time-aligned to obtain the time-synchronized target audio data and the target video data.
[0068] Step S20, for each frame of the target audio signal in the target audio data, determining a first theoretical frame number corresponding to the signal frequency of the target audio signal;
[0069] The target audio signal is an audio signal included in the target audio data to be distinguished from the audio signal in the test audio data.
[0070] It is easy to understand that there is a corresponding relationship between the signal frequency of the audio signal and the frame number, and the signal frequency does not change during the transmission process. Therefore, the frame number corresponding to the signal frequency can be determined based on the signal frequency of the target audio signal, that is, the first theoretical frame number.
[0071] Step S30, determining a target video image corresponding to a target audio signal in the target video data, and determining a second theoretical frame number corresponding to an image content of the target video image, wherein the target audio signal has a consistent timestamp with the target video image;
[0072] It should be noted that the target audio signal and the target video image timestamp are consistent, and specifically, the target audio signal and the target video image timestamp overlap. If there are multiple target video images with overlapping timestamps of the target audio signal, the last target video image with overlapping timestamps of the target audio signal in the target video data may be selected, or the first target video image with overlapping timestamps of the target audio signal in the target video data may be selected, or the target video image with completely overlapping timestamps of the target audio signal in the target video data may be selected. For example, a camera collects image data at a frame rate of 30fps (Frames Per Second) to obtain target video data, then the duration of a frame of video image in the target video data is about 33 milliseconds. Assuming that the target video data contains 3 frames of target video images, namely [target video image 1, target video image 2, target video image 3], then the timestamps of the 3 frames of target video images are [0-32 milliseconds, 33-65 milliseconds, 66-98 milliseconds]. The target audio data contains 3 frames of target audio signals, namely [target audio signal 1, target audio signal 2, target audio signal 3], and the corresponding timestamps are [0-32 milliseconds, 33-92 milliseconds, 93-98 milliseconds]. The last target video image in the target video data that overlaps with the timestamp of the target audio signal is selected, then the target video image corresponding to the target audio signal 1 is target video image 1, the target video image corresponding to the target audio signal 2 is target video image 3, and the target video image corresponding to the target audio signal 3 is target video image 3.
[0073] It is easy to understand that there is a corresponding relationship between the image content of a video image and the frame number, and image recognition can be used to identify the image content contained in the target video image, so that the frame number corresponding to the image content can be determined based on the image content of the target video image, that is, the second theoretical frame number.
[0074] Step S40: If the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchronism between the target audio data and the target video data.
[0075] The first theoretical frame number is compared with the second theoretical frame number. If the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchrony between the target audio data and the target video data, which can be understood as the target audio signal and the target video image are not synchronized. Further, the next frame of the target audio signal and the corresponding target video image can be detected to see whether they are synchronized, or the audio and video synchronization detection process can be terminated. This embodiment does not impose any specific restrictions on this.
[0076] It should be noted that if the first theoretical frame number is consistent with the second theoretical frame number, the next target audio signal and its corresponding target video image can be detected to see if they are synchronized until all target audio signals are detected, or a target audio signal and its corresponding target video image are out of synchronization.
[0077] This embodiment obtains target audio data and target video data to be synchronously detected, wherein the target audio data and the target video data are data output when the preset test end plays preset test media data, and the preset test media data include test audio data and test video data. The signal frequency of each frame of audio signal in the test audio data is different, and the image content of each frame of video image in the test video data is different; thus, there is a one-to-one correspondence between the frame number and the signal frequency of the audio signal in the test audio data, so that the corresponding frame number can be determined based on the signal frequency, and similarly, there is a one-to-one correspondence between the frame number and the image content of the video image in the test video data, so that the corresponding frame number can be determined based on the image content. For each frame of the target audio signal in the target audio data, determine the first theoretical frame number corresponding to the signal frequency of the target audio signal; determine the target video image corresponding to the target audio signal in the target video data, and determine the second theoretical frame number corresponding to the image content of the target video image, wherein the timestamps of the target audio signal and the target video image are consistent; if the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchrony between the target audio data and the target video. For the target audio signal and the target video image belonging to the same timestamp, if the data is synchronized, under the constraint of consistent timestamps, the corresponding frame numbers of the two should also be consistent, that is, the first theoretical frame number is consistent with the second theoretical frame number. Conversely, if the data is not synchronized, the first theoretical frame number and the second theoretical frame number will also be inconsistent. Based on this, the theoretical frame numbers corresponding to the target audio signal and the target video image with consistent timestamps are obtained. When the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchrony between the target audio data and the target video. The entire detection process is completed automatically without manual participation, thereby improving the detection efficiency of audio and video synchronization detection. Compared with the manual detection method, it does not rely on subjective sensitivity and personal experience, thereby improving the detection accuracy of audio and video synchronization detection.
[0078] Based on the above first embodiment, a second embodiment of the audio and video synchronization detection method of the present application is proposed. In this embodiment, the same or similar contents as those of the above first embodiment can be referred to the above description, and will not be repeated later. In this embodiment, before the step of determining the first theoretical frame number corresponding to the signal frequency of the target audio signal for each frame of the target audio signal in the target audio data, the method further includes:
[0079] Step A10, detecting all transition timestamps contained in the target audio data, wherein the transition timestamp includes a timestamp of a rising edge of a pulse signal in the target audio data;
[0080] It should be noted that the screened transition timestamp is related to the way the test audio data is divided into signal frames. If the test audio data is divided into signal frames by the rising edge of the pulse signal, the screened transition timestamp is the timestamp of the rising edge of the pulse signal. If the test audio data is divided into signal frames by the falling edge of the pulse signal, the screened transition timestamp is the timestamp of the falling edge of the pulse signal. If the test audio data is divided into signal frames by the mutation point where the pulse value is greater than or equal to the preset threshold (from less than the preset threshold to greater than or equal to the threshold), the screened transition timestamp is the timestamp of the mutation point where the pulse value is greater than or equal to the preset threshold. Exemplarily, the various embodiments of the present application are explained and illustrated by taking the transition timestamp as the timestamp of the rising edge of the pulse signal in the target audio data as an example.
[0081] Step A20: dividing the target audio data into at least one frame of target audio signal according to each of the jump time stamps, wherein the target audio data between two adjacent jump time stamps constitutes one frame of the target audio signal.
[0082] The audio data is divided into at least one frame of target audio signal according to each jump time stamp. Specifically, the target audio data between two adjacent jump time stamps is divided into one frame of target audio signal. Figure 7 The local target audio data shown, Figure 7 The horizontal axis represents time (unit: ms), and the vertical axis represents signal value. There is a jump timestamp 101 between 3.926731 milliseconds and 3.93 milliseconds, and there is a jump timestamp 102 between 3.96 milliseconds and 3.965 milliseconds. Then, there is a frame of target audio signal between the jump timestamp 101 and the jump timestamp 102.
[0083] It should be noted that the target audio data between the last jump timestamp of the target audio data and the end time of the target audio data is the last frame of the target audio signal.
[0084] During the encoding and transmission process of audio data, the signal itself does not carry time information or frame identification information. Therefore, this embodiment uses the rising edge of the pulse signal as the basis for dividing the signal frame, and divides the target audio data into at least one frame of target audio signal, thereby effectively dividing the signal frame of the target audio data without transmitting additional frame identification information. The signal frame division is realized by the pulse signal that constitutes the audio data, thereby reducing the implementation cost of signal frame division.
[0085] In a possible implementation manner, after the step of determining a second theoretical frame number corresponding to the image content of the target video image, the method further includes:
[0086] Step B10, determining an error time corresponding to the target audio signal based on the first theoretical frame number and the second theoretical frame number;
[0087] Step B20, combining the error times corresponding to the target audio signals to obtain an error value queue, and determining the total number of error times in the error value queue that are greater than or equal to a preset time threshold;
[0088] It can be understood that if the first theoretical frame number is consistent with the second theoretical frame number, the error time is zero, and the error time corresponding to each target audio signal is calculated, and each error time is combined in sequence according to the frame order of each target audio signal to obtain an error value queue. Exemplarily, assuming that the target audio data has 10 frames of target audio signals, namely [target audio signal 1, target audio signal 2, target audio signal 4, target audio signal 5...target audio signal 10], and the corresponding error times are respectively [0, 30 milliseconds, 20 milliseconds, 10 milliseconds, 0, 0, 30 milliseconds, 0, 0, 0], then the error value queue is {0, 30 milliseconds, 20 milliseconds, 10 milliseconds, 0, 0, 30 milliseconds, 0, 0, 0}.
[0089] After obtaining the error value queue, determine the total number of error times greater than or equal to the preset time threshold in the error value queue. Specifically, the total number may be the cumulative number of error times greater than or equal to the preset time threshold, such as in the error value queue of {0, 30 milliseconds, 20 milliseconds, 10 milliseconds, 0, 0, 30 milliseconds, 0, 0, 0}, if the preset time threshold is 20 milliseconds, the total number of error times greater than or equal to 20 milliseconds is 3; the total number may also be the maximum cumulative number of error times continuously greater than or equal to the preset time threshold, such as in the error value queue of {0, 30 milliseconds, 20 milliseconds, 10 milliseconds, 0, 0, 30 milliseconds, 0, 0, 0}, if the preset time threshold is 20 milliseconds, the total number of error times continuously greater than or equal to 20 milliseconds is 2.
[0090] Step B30: If the total number is greater than a preset threshold, the preset test end is determined to be an unqualified product.
[0091] The preset threshold value may be any value set in advance according to actual needs, and this embodiment does not impose any specific limitation on this.
[0092] In this embodiment, the error time is calculated based on the first theoretical frame number and the second theoretical frame number, thereby realizing the quantification of the audio and video synchronization error, and the error value queue is obtained based on the error time combination of each target audio signal, which provides effective reference data for determining whether the preset test end is qualified.
[0093] In a possible implementation, the step of determining the error time corresponding to the target audio signal based on the first theoretical frame number and the second theoretical frame number includes:
[0094] Step C10, calculating the absolute value of the frame number difference between the first theoretical frame number and the second theoretical frame number, and calculating the product of the absolute value of the frame number difference and a preset frame duration to obtain a calculation result;
[0095] The preset frame duration may specifically be a frame duration of an audio signal frame in the test audio data (hereinafter referred to as a single frame duration).
[0096] Step C20: determining the calculation result as the error time corresponding to the target audio signal.
[0097] The first theoretical frame number is S1, the second theoretical frame number is S2, and the single frame duration is t. The error time can be expressed by the formula |S2-S1|*t. In this way, the error time can be obtained by simple arithmetic operations, and the calculation amount of the error time is small and the calculation complexity is low.
[0098] For example, in order to help understand the technical concept or technical principle of audio and video synchronization detection after the present embodiment is combined with the above-mentioned embodiment 1, a specific embodiment is listed, referring to Figure 8 As shown, in this specific embodiment, the audio and video synchronization detection process is:
[0099] The preset test end plays the preset test media data in the AVI (Audio Video Interleave) file format, synchronously collects and obtains the target audio data and the target video data output by the preset test end, the image content in the target video data is a graphic block, the target audio data is divided into at least one frame of target audio signal by the rising edge of the pulse signal (referred to as the pulse signal rising edge), the time interval between the rising edges of the pulse signal in the audio data is synchronously analyzed, and the audio frame time interval sequence {33ms, 33ms, 34ms.......} is obtained, and the signal frequency carried in the target audio signal is demodulated to obtain a demodulated information sequence B {1000 Hz, 1100 Hz, 1200 Hz..... .}, take the rising edge of the pulse signal in each frame of the target audio signal as the trigger signal for parsing the collected video image, collect the target video image corresponding to each frame of the target audio signal, and obtain the number of image blocks in the image by analyzing the target video image to obtain the number of graphic blocks sequence C{1, 2, 3...}, calculate the signal frequency obtained from the audio and video analysis, convert the signal frequency into the first theoretical frame number F(B), convert the number of graphics obtained in the target video image into the second theoretical frame number F(C), compare F(B) with F(C), and based on the comparison result, we can know whether the two are synchronized and the error time, and combine the error times to get the error value queue ( Figure 8 The audio and video time difference sequence shown in ).
[0100] Based on the above-mentioned first and / or second embodiments, a third embodiment of the audio and video synchronization detection method of the present application is proposed. In this embodiment, the same or similar contents as those of the above-mentioned first and second embodiments can be referred to the above introduction, and will not be repeated later. In this embodiment, before the step of obtaining the target audio data and the target video data to be synchronously detected, the method further includes:
[0101] Step D10, generating test audio data, wherein the signal frequency of each frame of audio signal in the test audio data is different;
[0102] Step D20, generating test video data, wherein the frame rate and frame number of the test video data are consistent with those of the test audio data, and the image content of each frame of the test video data is different;
[0103] The frame rate and frame number of the test audio data and the test video data are consistent to ensure synchronization between the generated test audio data and the test video data. Figure 4 As shown, the frame rates of the test audio data and the test video data are both 30fps, and the number of frames is both 64 frames.
[0104] Step D30: synthesize the test audio data and the test video data to obtain preset test media data.
[0105] Specifically, the test audio data and the test video data can be synthesized in a known manner, and the synthesis process of the preset test media data is not described in detail in this embodiment. Figure 4 As shown, the test audio data and the test video data are synthesized into preset test media data in AVI format.
[0106] Furthermore, after obtaining the preset test media data, the preset test media data may be played through the preset test end to detect whether the audio and video data outputted by the preset test end after playing are synchronized.
[0107] In a possible implementation, the step of generating test audio data includes:
[0108] Step E10, obtaining preset reference audio data, wherein the reference audio data includes a preset frame reference audio signal;
[0109] In a preferred embodiment, referring to Figure 4 As shown, the signal pulse of each frame of the audio signal of the reference audio data is a rectangular pulse to facilitate subsequent frequency modulation.
[0110] Step E20, for each frame of the reference audio signal in the reference audio data, determining a corresponding signal frequency based on the frame number of the reference audio signal, and frequency modulating the reference audio signal based on the signal frequency to obtain a frequency modulated audio signal, wherein the signal frequency is positively correlated with the frame number of the audio signal;
[0111] In a preferred embodiment, referring to Figure 4 As shown, considering that the preset test end is usually a personal electronic consumer product, the frequency response curve of the product speaker needs to be considered. Based on this, the 1K to 10K range with a relatively smooth frequency response is selected as the modulation frequency range, corresponding to the frame number of 0 to 625, and the frame number and signal frequency related transformation relationship f(x) = 14.4x + 1000 is obtained, where x represents the frame number and f(x) represents the signal frequency.
[0112] The reference audio signal is modulated based on the signal frequency to obtain an FM audio signal. Specifically, a sine wave having the same frequency as the signal frequency may be generated, and the sine wave is superimposed on the reference audio signal to obtain an FM audio signal. For example, the reference audio signal is Figure 4 When the signal is shown, the reference audio signal is frequency modulated to obtain an FM audio signal such as Figure 5 As shown, Figure 5 The horizontal axis represents time (unit: ms), and the vertical axis represents signal value.
[0113] Step E30, combining each frame of the FM audio signal to obtain test audio data.
[0114] It can be understood that the test audio data is obtained by combining each frame of the frequency modulated audio signal, that is, the test audio data is obtained after the reference audio data is frequency modulated.
[0115] Since the audio signal does not carry timestamp information during the encoding process, this embodiment obtains the target audio data by frequency modulating the reference audio data, and uses the pulses of the audio signal in the reference audio data as the time-stamp synchronization signal. By measuring the pulse interval, it can be determined whether the audio and video are simultaneously stuck during the playback process, thereby reducing the cumulative error and achieving the purpose of calculating the specific frame number based on the specific signal frequency demodulated from the pulse.
[0116] In a possible implementation, the step of generating test video data includes:
[0117] Step F10, sequentially generating a frame of video image containing a target number of preset graphics until the preset frame of the video image is generated, wherein the target number is positively correlated with the frame number of the video image;
[0118] The video image is easily affected by the external brightness during transmission, and the high-frequency part of the image is easily lost. Therefore, the preset graphic can be a simple graphic block, such as a rectangular block, a circular block, etc., and this embodiment does not make any specific restrictions on this.
[0119] It can be that the generated video image includes a target number of preset graphics, that is, the image content of the generated video image is the target number of preset graphics, and different image contents specifically mean different numbers of preset graphics.
[0120] The number of targets is positively correlated with the frame number of the video image. In a preferred embodiment, the number of targets is equal to the frame number of the video image, so that the number of graphics included in the video image is the frame number of the video image, so that the frame number of the video image can be quickly determined based on the image content.
[0121] Furthermore, when there are too many graphics, on the one hand, the resolution of the preset test end is not supported, and on the other hand, there are too many graphics to quickly and accurately identify the number of graphics included in the video image. Figure 6 As shown, in a preferred embodiment, the maximum value of the target number is 624 to ensure that each graphic can be displayed clearly and the boundaries between the graphics are obvious, so as to facilitate the subsequent rapid and accurate identification of the number of graphics in the video image and then determine the corresponding frame number.
[0122] Step F20, combining each frame of the video image to obtain test video data.
[0123] It is understandable that the test video data is obtained by sequentially combining each frame of video image at a preset frame rate. For example, if the preset frame rate is 30fps, the test video data is obtained by combining a frame of video image approximately every 33 milliseconds.
[0124] In order to help understand the above embodiments of the audio and video synchronization detection method of the present application, an example of the audio and video synchronization detection method combined with the above embodiments is proposed. The specific steps are as follows:
[0125] 1. Modulation audio frequency selection: select the 1K-10K range with a relatively smooth frequency response as the modulation frequency range, corresponding to the frame number 0-625, so as to obtain the frame number and frequency related transformation relationship f(x)=14.4x+1000.
[0126] 2. Considering the actual screen size of the product and the processing speed of the analysis algorithm, the number of graphics selected in the video cannot be too many. The transformation relationship between the frame number and the number is selected as f(x)=1x, where x is the frame number and f(x) is the number of graphics. In order to enhance recognition at the detection end, the graphics will be randomly displayed in a circle or square at the position with the largest free area.
[0127] 3. A graphic is added to each frame of the generated video image to obtain test video data. At the same time, a pulse is added to each frame of the corresponding audio signal. A sine wave with a frequency of (frame number*14.4+1000) is modulated in the pulse to obtain test audio data.
[0128] 4. Synthesize the test audio data and the test video data into a preset media file in AVI format according to the frames.
[0129] 5. Import the preset media file into the preset test terminal for playback.
[0130] 6. Use the camera and microphone to receive audio data and video data and analyze them separately.
[0131] 7. Calculate the number of graphics contained in each frame according to the frame rate of the received target video data and obtain the number queue FCCA. Specifically, perform color plane extraction on the acquired image, then perform binarization operation, and finally identify the number of graphics in the video image.
[0132] 8. Analyze the received target audio data to obtain the inter-pulse interval time queue PTA and the sine frequency (signal frequency) PFA contained in each pulse. Specifically, obtain the maximum value of the audio waveform, perform threshold processing according to the maximum value of 80%, and perform alignment according to the phase difference corresponding to the threshold of 80% to obtain the pulse time interval.
[0133] 9. According to the corresponding relationship, PFA is reversely transformed back to the frame number FPFA (first theoretical frame number) using the inverse function f(y)=0.0694y-69.444.
[0134] 10. Adjust FCCA according to the interval data of PTA. Specifically, if the PTA time is greater than the single-frame time, it means that the audio is stuck during playback, and it is necessary to skip the FCCA data rounded up to the integer number ((interval-single-frame time) / single-frame time).
[0135] 11. After the adjustment is completed, subtract FCCA (the second theoretical frame number) from FPFA, and obtain the error time by multiplying the difference by single frame time. Combine all the error times to obtain the error value queue during the preset test single playback. By analyzing the total number of error times that continuously exceed the preset time threshold, the degree of jamming in the product's audio and video synchronization and whether the preset test end is a qualified product can be evaluated.
[0136] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the audio and video synchronization detection method of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.
[0137] An embodiment of the present application provides an audio and video synchronization detection device, which includes: an image acquisition unit, a sound pickup unit and a processor, wherein the image acquisition unit and the sound pickup unit are respectively connected to the processor, and the processor is used to execute the steps of the audio and video synchronization detection method in the above embodiment.
[0138] The image acquisition unit includes but is not limited to a camera, and the sound pickup unit includes but is not limited to a microphone.
[0139] Reference below Fig. 9 , which shows a schematic diagram of the structure of an audio and video synchronization detection device suitable for implementing the embodiment of the present application. The audio and video synchronization detection device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The audio and video synchronization detection device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0140] like Fig. 9As shown, the audio and video synchronization detection device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the audio and video synchronization detection device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the audio and video synchronization detection device to communicate wirelessly or wired with other devices to exchange data. Although the figure shows an audio and video synchronization detection device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.
[0141] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.
[0142] Compared with the prior art, the beneficial effects of the extended reality device provided in the embodiment of the present application are the same as the beneficial effects of the audio and video synchronization detection method provided in the above embodiment, and other technical features in the extended reality device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.
[0143] It should be understood that the various parts disclosed in the embodiments of the present application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.
[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
[0145] An embodiment of the present application provides a computer-readable storage medium having computer-readable program instructions (ie, a computer program) stored thereon, wherein the computer-readable program instructions are used to execute the audio and video synchronization detection method in the above-mentioned embodiment.
[0146] The computer-readable storage medium provided in the embodiment of the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM: Random Access Memory), a read-only memory (ROM: Read Only Memory), an erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency: Radio Frequency), etc., or any suitable combination of the above.
[0147] The computer-readable storage medium may be included in the extended reality device; or may exist independently without being assembled into the extended reality device.
[0148] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the extended reality device, the extended reality device performs the functions defined in the method of the embodiment disclosed in the present application.
[0149] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0150] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0151] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.
[0152] The readable storage medium provided in the embodiment of the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned audio and video synchronization detection method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in the embodiment of the present application are the same as the beneficial effects of the audio and video synchronization detection method provided in the above-mentioned embodiment, which will not be repeated here.
[0153] An embodiment of the present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned audio and video synchronization detection method when executed by a processor.
[0154] Compared with the prior art, the beneficial effects of the computer program product provided in the embodiment of the present application are the same as the beneficial effects of the audio and video synchronization detection method provided in the above embodiment, which will not be repeated here.
[0155] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.
Claims
1. A method for detecting audio and video synchronization, characterized in that: The audio and video synchronization detection method comprises the following steps: Acquire target audio data and target video data to be synchronously detected, wherein the target audio data and the target video data are data output when a preset test end plays preset test media data, the preset test media data includes test audio data and test video data, the signal frequency of each frame of audio signal in the test audio data is different, and the image content of each frame of video image in the test video data is different; For each frame of the target audio signal in the target audio data, determining a first theoretical frame number corresponding to the signal frequency of the target audio signal; Determine a target video image corresponding to a target audio signal in the target video data, and determine a second theoretical frame number corresponding to an image content of the target video image, wherein the target audio signal is consistent with a timestamp of the target video image; If the first theoretical frame number is inconsistent with the second theoretical frame number, it is determined that there is data asynchronism between the target audio data and the target video data.
2. The method according to claim 1, characterized in that Before the step of determining, for each frame of the target audio signal in the target audio data, a first theoretical frame number corresponding to the signal frequency of the target audio signal, the method further comprises: Detecting all transition timestamps contained in the target audio data, wherein the transition timestamps include timestamps of rising edges of pulse signals in the target audio data; The target audio data is divided into at least one frame of target audio signal according to each of the jump time stamps, wherein the target audio data between two adjacent jump time stamps constitutes one frame of the target audio signal.
3. The method according to claim 1, characterized in that After the step of determining a second theoretical frame number corresponding to the image content of the target video image, the method further includes: Determine an error time corresponding to the target audio signal based on the first theoretical frame number and the second theoretical frame number; Combining the error times corresponding to the target audio signals to obtain an error value queue, and determining the total number of error times in the error value queue that are greater than or equal to a preset time threshold; If the total number is greater than a preset threshold, the preset test end is determined to be an unqualified product.
4. The method according to claim 3, characterized in that The step of determining the error time corresponding to the target audio signal based on the first theoretical frame number and the second theoretical frame number includes: Calculating an absolute value of a frame number difference between the first theoretical frame number and the second theoretical frame number, and calculating a product of the absolute value of the frame number difference and a preset frame duration to obtain a calculation result; The calculation result is determined to be an error time corresponding to the target audio signal.
5. The method according to claim 1, characterized in that Before the step of acquiring the target audio data and the target video data to be synchronously detected, the method further includes: Generating test audio data, wherein the signal frequency of each frame of audio signal in the test audio data is different; Generate test video data, wherein the frame rate and frame number of the test video data are consistent with those of the test audio data, and the image content of each frame of the video image in the test video data is different; The test audio data and the test video data are synthesized to obtain preset test media data.
6. The method according to claim 5, characterized in that The step of generating test audio data comprises: Acquire preset reference audio data, wherein the reference audio data includes a preset frame reference audio signal; For each frame of the reference audio signal in the reference audio data, determine a corresponding signal frequency based on the frame number of the reference audio signal, and frequency-modulate the reference audio signal based on the signal frequency to obtain a frequency-modulated audio signal, wherein the signal frequency is positively correlated with the frame number of the audio signal; Each frame of the FM audio signal is combined to obtain test audio data.
7. The method according to claim 5, characterized in that The step of generating test video data comprises: Generate a frame of video image containing a target number of preset graphics in sequence until the preset frame of the video image is generated, wherein the target number is positively correlated with the frame number of the video image; Each frame of the video image is combined to obtain test video data.
8. An audio and video synchronization detection device, characterized in that: The audio and video synchronization detection device includes an image acquisition unit, a sound pickup unit and a processor, wherein the image acquisition unit and the sound pickup unit are respectively connected to the processor, and the processor is used to execute the steps of the audio and video synchronization detection method according to any one of claims 1 to 7.
9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the audio and video synchronization detection method according to any one of claims 1 to 7 are implemented.
10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the audio and video synchronization detection method according to any one of claims 1 to 7 are implemented.
