Audio-video synchronization automatic detection method and device

Abstract

The application provides an audio-video synchronization automatic detection method and device, the method comprising: decoding a video stream and an audio stream of a media file to be processed to obtain video data and audio data; performing motion detection on the video data to generate a video motion event stream with time information, and performing sound activity detection on the audio data with a fixed time length analysis window to generate an audio sound event stream with time information, wherein the video motion event stream comprises motion start events and motion end events, and the audio sound event stream comprises sound start events and sound end events; tracking motion scene states based on the video motion event stream, matching the sound start events with the tracked motion scenes, and calculating audio-video synchronization deviations of each matched motion scene, so that audio-video synchronization detection can be realized.

Description

Technical Field

[0001] This invention relates to the field of multimedia processing and audio / video quality detection technology, and in particular to an automatic audio / video synchronization detection method and apparatus. Background Technology

[0002] Audio and video synchronization is one of the core indicators of multimedia content quality. In cloud gaming scenarios, game visuals and sound effects undergo independent acquisition, encoding, transmission, decoding, and rendering processes, and each stage may introduce varying degrees of timing deviation. When the audio and video timing deviation exceeds the threshold perceptible to the human eye / ear (generally considered to be about 100ms), users will clearly perceive quality issues such as unsynchronized game visuals and sound effects, and misaligned operations and feedback, severely impacting the user experience.

[0003] Therefore, how to achieve synchronized audio and video detection has become an urgent technical problem to be solved. Summary of the Invention

[0004] To address the technical problems existing in the prior art, embodiments of this application provide an automatic audio and video synchronization detection method and apparatus.

[0005] In a first aspect, embodiments of this application provide an automatic audio-video synchronization detection method, including: The media file to be processed is decoded into video stream and audio stream to obtain video data and audio data respectively. Motion detection is performed on video data to generate a video motion event stream with time information, and sound activity detection is performed on audio data in fixed-duration analysis windows to generate an audio sound event stream with time information. The video motion event stream includes motion start event and motion end event, and the audio sound event stream includes sound start event and sound end event. Motion scene state tracking is performed based on video motion event streams. The sound start event is matched with the tracked motion scene, and the audio-video synchronization deviation of each matched motion scene is calculated.

[0006] Secondly, embodiments of this application also provide an automatic audio-video synchronization detection device, comprising: The decoding unit is used to decode the video stream and audio stream of the media file to be processed, so as to obtain video data and audio data respectively. The detection unit is used to perform motion detection on video data, generate a video motion event stream with time information, and perform sound activity detection on audio data in units of fixed-duration analysis windows, generating an audio sound event stream with time information. The video motion event stream includes motion start event and motion end event, and the audio sound event stream includes sound start event and sound end event. The calculation unit is used to track the state of motion scene based on video motion event stream, match the sound start event with the tracked motion scene, and calculate the audio and video synchronization deviation of each matched motion scene.

[0007] The audio-video synchronization automatic detection method and apparatus provided in this application first decodes the media file to be processed to obtain video data and audio data; then, motion detection and sound activity detection are performed on the video data and audio data respectively to generate a video motion event stream and an audio sound event stream with time information; finally, motion scene state tracking is performed based on the video motion event stream, the sound start event in the audio sound event stream is matched with the tracked motion scene, and the audio-video synchronization deviation of each matched motion scene is calculated. The entire scheme can realize audio-video synchronization detection. Attached Figure Description

[0008] Figure 1 This is a flowchart illustrating an embodiment of an automatic audio-video synchronization detection method provided in this application. Figure 2 This is a schematic diagram of an embodiment of an automatic audio-video synchronization detection device provided in this application. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0010] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0011] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0012] Reference Figure 1 The diagram shown is a flowchart illustrating an automatic audio-video synchronization detection method provided in an embodiment of this application. The method includes: S10. Decode the video stream and audio stream of the media file to be processed to obtain video data and audio data respectively; In this embodiment, the media file to be processed can be a media file obtained by recording the cloud game's running screen and sound, or a media file input by the user. In specific implementation, the video stream can be decoded into a raw frame sequence with a video pixel format of RGB24, and the audio stream can be decoded into raw sampled data encoded by linear PCM (Pulse Code Modulation), with a sampling rate of 16kHz, a mono channel, and each sample point represented by a 16-bit signed integer.

[0013] S11. Perform motion detection on the video data to generate a video motion event stream with time information, and perform sound activity detection on the audio data in units of fixed-duration analysis windows to generate an audio sound event stream with time information. The video motion event stream includes motion start event and motion end event, and the audio sound event stream includes sound start event and sound end event. S12. Based on the video motion event stream, perform motion scene state tracking, match the sound start event with the tracked motion scene, and calculate the audio-video synchronization deviation of each matched motion scene.

[0014] The automatic audio-video synchronization detection method provided in this application first decodes the media file to be processed to obtain video data and audio data; then, it performs motion detection and sound activity detection on the video data and audio data respectively to generate a video motion event stream and an audio sound event stream with time information; finally, it tracks the motion scene state based on the video motion event stream, matches the sound start event in the audio sound event stream with the tracked motion scene, and calculates the audio-video synchronization deviation of each matched motion scene. The entire scheme can realize audio-video synchronization detection.

[0015] Based on the aforementioned method embodiments, the step of performing motion detection on video data and generating a video motion event stream with time information may include: If it is determined that a motion initiation event is pending, and the current video frame is not the last video frame in the video data, and the current video frame is a frame with significant motion, then the motion initiation event is triggered; or If it is determined that the current waiting motion end event is triggered, and the current video frame is not the last video frame in the video data, and the current video frame is a non-significant motion frame, and the N1 consecutive video frames before the current video frame are all non-significant motion frames, then the motion end event is triggered. Among them, the motion ratio of significant motion frames is greater than the dynamic threshold, and the motion ratio of non-significant motion frames is less than or equal to the dynamic threshold. The dynamic threshold is calculated based on the dynamic baseline value, and the dynamic baseline value is calculated based on the sliding window. The sliding window stores the motion ratio of the latest N2 non-significant motion frames, where N1 and N2 are both positive integers.

[0016] In this embodiment, "currently awaiting motion start event triggering" means that the most recently triggered motion event is a motion end event, and "currently awaiting motion end event triggering" means that the most recently triggered motion event is a motion start event. Motion start and motion end events always trigger alternately. If the current awaiting motion start event is triggered, the motion start event is triggered only if the current video frame is not the last video frame in the video data and the current video frame is a significant motion frame; otherwise, the motion start event is not triggered. If the current awaiting motion end event is triggered, the motion end event is triggered only if the current video frame is the last video frame in the video data, or if the current video frame is not the last video frame in the video data, the current video frame is a non-significant motion frame, and the N1 consecutive video frames preceding the current video frame are all non-significant motion frames; otherwise, the motion end event is not triggered. Ultimately, pairs of motion start and motion end events are obtained.

[0017] The value of N1 can be set as needed, for example, to 4. The relationship between the motion ratio and the dynamic threshold determines whether the current video frame is a significant motion frame or a non-significant motion frame: if the motion ratio of the current video frame is greater than the dynamic threshold, then the current video frame is determined to be a significant motion frame; otherwise, the current video frame is determined to be a non-significant motion frame.

[0018] Motion ratio represents the percentage of pixels that have changed compared to the previous video frame. The percentage of pixels whose brightness has changed refers to the proportion of pixels whose brightness has changed. Before calculating the motion ratio of a video frame P, it can be determined whether the width and height of video frame P exceed a set threshold. If they do, video frame P is scaled down proportionally to ensure that its width and height do not exceed the threshold; otherwise, no scaling down is performed. Then, video frame P is converted to grayscale to obtain a single-channel grayscale image. Gaussian blur is applied to the grayscale image, and the absolute value of the brightness difference between the Gaussian-blurred image (denoted as P1) and the image P2 corresponding to the previous video frame (which has undergone the same processing as video frame P) is calculated. Next, for each absolute value, calculate whether it is greater than a threshold pixel_threshold (the value can be set as needed, for example, 20): if the absolute value is greater than the threshold pixel_threshold, then the pixel in the video frame P corresponding to that absolute value is marked as a changed pixel; otherwise, the pixel in the video frame P corresponding to that absolute value is marked as an unchanged pixel. Finally, calculate the ratio of the number of changed pixels in video frame P to the number of pixels in video frame P; this ratio is the motion ratio of video frame P.

[0019] The sliding window stores the motion proportions of the latest N² insignificant motion frames. Initially, if the number of motion proportions in the sliding window is less than N², the dynamic baseline value can be 0. Once the number of motion proportions in the sliding window reaches N², the dynamic baseline value is calculated based on these N² motion proportions; for example, the dynamic baseline value can be the median of these N² motion proportions. After the number of motion proportions in the sliding window reaches N², the motion proportions of subsequent insignificant motion frames will update the motion proportions in the sliding window to ensure that the sliding window stores the motion proportions of the latest N² insignificant motion frames. The value of N² can be set as needed, for example, to 30.

[0020] The formula for calculating the dynamic threshold (dynamic_threshold) is as follows: dynamic_threshold=max(baseline×sensitivity,min_threshold); Wherein, baseline is the dynamic baseline value, sensitivity is the preset sensitivity multiplier, min_threshold is the preset threshold lower limit, and the values ​​of sensitivity and min_threshold can be set as needed, for example, 2.5 and 0.01 respectively, and max(x, y) means taking the larger value of x and y.

[0021] The timing information for the motion start event can be the display time of the video frame corresponding to the motion start event. If the video frame corresponding to the motion end event is the last video frame in the video data, then the timing information for the motion end event can be the display time of the video frame corresponding to the motion end event; however, if the video frame corresponding to the motion end event is not the last video frame in the video data, then the timing information for the motion end event can be the display time of the first video frame among the N1 consecutive video frames preceding the video frame corresponding to the motion end event. The display time of the video frame can be the actual display timestamp of the video frame.

[0022] Based on the aforementioned method embodiments, the step of performing sound activity detection on the audio data in units of fixed-duration analysis windows to generate an audio sound event stream with time information may include: If it is determined that the currently waiting sound start event is triggered, and the current analysis window in the audio data is not the last analysis window in the audio data, and the current analysis window satisfies the first and second conditions, then the sound start event is triggered; or If it is determined that the current waiting sound end event has been triggered, and the current analysis window in the audio data is not the last analysis window in the audio data, and the current analysis window meets the third condition, and the N3 consecutive analysis windows before the current analysis window all meet the third condition, then the sound end event is triggered. The first condition is that the decibel level of the analysis window is greater than the first threshold. The second condition is that the average decibel level of the N4 consecutive analysis windows before the analysis window is less than or equal to the first threshold, or the average decibel level of the N4 consecutive analysis windows before the analysis window is greater than the first threshold and the difference between the decibel level of the analysis window and the average decibel level is greater than or equal to the second threshold. The third condition is that the decibel level of the analysis window is less than or equal to the first threshold. N3 and N4 are both positive integers.

[0023] In this embodiment, "currently waiting for the sound start event to be triggered" means that the most recently triggered sound event is the sound end event, and "currently waiting for the sound end event to be triggered" means that the most recently triggered sound event is the sound start event. The sound start event and the sound end event always trigger alternately.

[0024] If the current waiting sound start event is to be triggered, the sound start event will only be triggered if the current analysis window in the audio data is not the last analysis window in the audio data, and the current analysis window meets the first and second conditions. Otherwise, the sound start event will not be triggered. The first condition is that the decibel level of the analysis window is greater than a first threshold. The second condition is that the average decibel level of the N4 consecutive analysis windows before the current analysis window is less than or equal to the first threshold, or the average decibel level of the N4 consecutive analysis windows before the current analysis window is greater than the first threshold and the difference between the decibel level of the current analysis window and the average decibel level is greater than or equal to the second threshold. When calculating the decibel level of the analysis window, the root mean square (RMS) value corresponding to the analysis window can be calculated first, and then the RMS value can be converted to decibels. The RMS value corresponding to the analysis window is the RMS value of the normalized amplitude values ​​of all sampling points of the analysis window, and the normalized amplitude value is the result obtained by normalizing the amplitude value. The decibel conversion formula is: energy_db=max(20×log 10 (rms / 1.0), -100), energy_db represents the decibels of the analysis window, and rms represents the normalized value of the amplitude of the sampling point. The values ​​of the first and second thresholds can be set as needed, for example, -70dB and 14dB respectively. The value of N4 can be set as needed, for example, 10.

[0025] If the currently waiting sound end event is triggered, the sound end event is triggered if the current analysis window in the audio data is the last analysis window, or if the current analysis window is not the last analysis window in the audio data, and the current analysis window meets the third condition, and the N3 consecutive analysis windows preceding the current analysis window also meet the third condition. Otherwise, the sound end event is not triggered. The third condition is that the decibel level of the analysis window is less than or equal to a first threshold. The value of N3 can be set as needed, for example, a value of 4. Ultimately, paired sound start and sound end events can be obtained.

[0026] The timing information for the sound start event can be the playback time of the analysis window corresponding to the sound start event. If the analysis window corresponding to the sound end event is the last analysis window in the audio data, then the timing information for the sound end event can be the playback time of the analysis window corresponding to the sound end event; otherwise, if the analysis window corresponding to the sound end event is not the last analysis window in the audio data, then the timing information for the sound end event can be the playback time of the first analysis window among the N3 consecutive analysis windows preceding the analysis window corresponding to the sound end event. The playback time of the analysis window can be the actual playback timestamp of the analysis window, calculated using the formula: Playback timestamp = Analysis window sequence number × Number of samples / Sampling rate of the analysis window + Audio stream start offset. The duration of the analysis window can be set as needed, for example, setting it to 20ms, which means dividing the audio data into 20ms units, resulting in multiple analysis windows.

[0027] Based on the foregoing method embodiments, the step of tracking motion scene state based on video motion event streams and matching the sound initiation event with the tracked motion scene may include: After each motion start event is generated, if it is determined that the time since the end of the last motion scene exceeds the preset cooldown time, a new motion scene is started, and the time information of the generated motion start event is used as the video-side start time reference first_motion_ms for the new motion scene. If it is determined that the audio state is in a sound state when the new motion scene is opened, then the time information of the most recent sound start event is used as the audio-side start time base first_sound_ms of the new motion scene; or if it is determined that a sound start event was generated during the opening of the new motion scene and the new motion scene was not associated with audio when the sound start event was generated, then the time information of the generated sound start event is used as the audio-side start time base first_sound_ms of the new motion scene. If it is determined that a motion end event was generated during the opening of the new motion scene, then the new motion scene is closed.

[0028] In this embodiment, after each motion start event is triggered, it is determined whether the time remaining since the end of the last motion scene exceeds a preset cooldown time. If the cooldown time has not been exceeded, the currently triggered motion start event is skipped; otherwise, a new motion scene is started, and the time information of the triggered motion start event is used as the video-side start time base first_motion_ms for the new motion scene. For the first triggered motion start event, the result of determining whether the time remaining since the end of the last motion scene exceeds the preset cooldown time is true by default. That is, each motion start event that is not skipped is accompanied by the opening of a motion scene, and the time information of the motion end event corresponding to the last motion start event among several consecutive skipped motion start events is the actual closing time of the corresponding motion scene. For example, suppose the motion events triggered in chronological order are motion start event A1, motion end event B1, motion start event A2, motion end event B2, motion start event A3, motion end event B3, motion start event A4, motion end event B4... After motion start event A1 is triggered, motion scene 1 is opened, and the time information of motion start event A1 is used as the video-side start time base first_motion_ms for motion scene 1. After motion end event B1 is triggered, motion scene 1 is closed. If motion start event A2 is triggered before the end of motion scene 1 (within the cooldown time), motion start event A2 is ignored, and the time of motion end event B2 is used as the actual closing time of motion scene 1. If motion start event A3 is triggered before the end of motion scene 1 (i.e., the actual closing time of motion scene 1) (within the cooldown time), motion start event A3 is ignored, and the time of motion end event B3 is used as the actual closing time of motion scene 1. If motion start event A4 is triggered before the end of motion scene 1 (i.e., the actual closing time of motion scene 1) (exceeds the cooldown time), a new motion scene 2 is started, and the time of motion start event A4 is used as the video-side start time base `first_motion_ms` for motion scene 2. The motion start event corresponding to motion scene 1 is motion start event A1, and the corresponding motion end event is motion end event B1. The cooldown time can be set as needed, for example, to 3000ms.

[0029] Each motion scene needs to be associated with audio during its activation phase, as illustrated by the previous example. Assume the sound events triggered in chronological order are: sound start event C1, sound end event D1, sound start event C2, sound end event D2… The motion events and sound events are merged to form a unified timeline event sequence: sound start event C1, motion start event A1, motion end event B1, motion start event A2, motion end event B2, motion start event A3, sound end event D1, motion end event B3, motion start event A4, sound start event C2, motion end event B4, sound end event D2… For motion scene 1, the nearest sound event when motion scene 1 is activated is sound start event C1. Therefore, the audio state is confirmed to be audible. The time information of sound start event C1 (i.e., the time information of the most recent sound start event) is used as the audio-side start time base first_sound_ms for motion scene 1, thus associating motion scene 1 with the audio corresponding to sound start event C1. For motion scene 2, if a sound start event C2 is generated during the opening of motion scene 2 and motion scene 2 is not associated with audio when the sound start event C2 is generated, then the time information of the sound start event C2 is used as the audio-side start time base first_sound_ms of motion scene 2 to associate motion scene 2 with the audio corresponding to the sound start event C2.

[0030] If no audio is associated during the opening of a motion scene, the target sound start event is determined from the candidate sound start events. The time information of the target sound start event is used as the audio-side start time reference `first_sound_ms` for that motion scene. Here, a candidate sound start event is a sound start event whose absolute value of the difference between its time information and the video-side start time reference `first_motion_ms` for the motion scene is less than or equal to a third threshold. The target sound start event is the candidate sound start event with the shortest time interval between its time information and the video-side start time reference `first_motion_ms` for the motion scene. The third threshold can be set as needed, for example, to 2000ms.

[0031] Based on the aforementioned method embodiments, calculating the audio-visual synchronization deviation of each matched motion scene may include: For each matched motion scene, calculate the absolute value of the difference between the video-side start time reference first_motion_ms and the audio-side start time reference first_sound_ms. Determine whether the absolute value is less than or equal to the fourth threshold. If the absolute value is less than or equal to the fourth threshold, then the audio and video of the motion scene are determined to be synchronized; otherwise, the audio and video of the motion scene are determined to be out of sync.

[0032] In this embodiment, the fourth threshold can be set as needed, for example, to 100ms. This value is determined with reference to the perceptible threshold of audio-visual synchronization deviation in human visual and auditory perception.

[0033] Reference Figure 2 The diagram shown is a structural schematic of an automatic audio-visual synchronization detection device provided in an embodiment of this application. The device includes: Decoding unit 20 is used to perform video stream decoding and audio stream decoding on the media file to be processed, so as to obtain video data and audio data respectively; The detection unit 21 is used to perform motion detection on video data, generate a video motion event stream with time information, and perform sound activity detection on audio data in units of fixed-duration analysis windows, generating an audio sound event stream with time information. The video motion event stream includes motion start event and motion end event, and the audio sound event stream includes sound start event and sound end event. The calculation unit 22 is used to track the state of motion scene based on the video motion event stream, match the sound start event with the tracked motion scene, and calculate the audio and video synchronization deviation of each matched motion scene.

[0034] The audio-video synchronization automatic detection device provided in this application first uses a decoding unit 20 to decode the media file to be processed to obtain video data and audio data; then, a detection unit 21 performs motion detection and sound activity detection on the video data and audio data respectively to generate a video motion event stream and an audio sound event stream with time information; finally, a calculation unit 22 performs motion scene state tracking based on the video motion event stream, matches the sound start event in the audio sound event stream with the tracked motion scene, and calculates the audio-video synchronization deviation of each matched motion scene. The entire scheme can realize audio-video synchronization detection.

[0035] The automatic audio-video synchronization detection device provided in this application embodiment is implemented in the same way as the automatic audio-video synchronization detection method provided in this application embodiment, and the effect it can achieve is also the same as the automatic audio-video synchronization detection method provided in this application embodiment, so it will not be described again here.

[0036] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An automatic audio-video synchronization detection method, characterized in that, include: The media file to be processed is decoded into video stream and audio stream to obtain video data and audio data respectively. Motion detection is performed on video data to generate a video motion event stream with time information, and sound activity detection is performed on audio data in fixed-duration analysis windows to generate an audio sound event stream with time information. The video motion event stream includes motion start event and motion end event, and the audio sound event stream includes sound start event and sound end event. Motion scene state tracking is performed based on video motion event streams. The sound start event is matched with the tracked motion scene, and the audio-video synchronization deviation of each matched motion scene is calculated. The motion scene state tracking based on video motion event stream, which matches the sound initiation event with the tracked motion scene, includes: After each motion start event is generated, if it is determined that the time since the end of the last motion scene exceeds the preset cooldown time, a new motion scene is started, and the time information of the generated motion start event is used as the video-side start time reference first_motion_ms for the new motion scene. If it is determined that the audio state is in a sound state when the new motion scene is opened, then the time information of the most recent sound start event is used as the audio-side start time base first_sound_ms of the new motion scene; or if it is determined that a sound start event was generated during the opening of the new motion scene and the new motion scene was not associated with audio when the sound start event was generated, then the time information of the generated sound start event is used as the audio-side start time base first_sound_ms of the new motion scene. If it is determined that a motion end event was generated during the opening of the new motion scene, then the new motion scene is closed.

2. The method as described in claim 1, characterized in that, The step of performing motion detection on video data and generating a video motion event stream with time information includes: If it is determined that a motion initiation event is pending, and the current video frame is not the last video frame in the video data, and the current video frame is a frame with significant motion, then the motion initiation event is triggered; or If it is determined that the current waiting motion end event is triggered, and the current video frame is not the last video frame in the video data, and the current video frame is a non-significant motion frame, and the N1 consecutive video frames before the current video frame are all non-significant motion frames, then the motion end event is triggered. Among them, the motion ratio of significant motion frames is greater than the dynamic threshold, and the motion ratio of non-significant motion frames is less than or equal to the dynamic threshold. The dynamic threshold is calculated based on the dynamic baseline value, and the dynamic baseline value is calculated based on the sliding window. The sliding window stores the motion ratio of the latest N2 non-significant motion frames, where N1 and N2 are both positive integers.

3. The method as described in claim 2, characterized in that, The time information for the motion start event is the display time of the video frame corresponding to the motion start event, and the time information for the motion end event is the display time of the first video frame among the N1 consecutive video frames preceding the video frame corresponding to the motion end event.

4. The method as described in claim 2, characterized in that, The formula for calculating the dynamic threshold (dynamic_threshold) is as follows: dynamic_threshold=max(baseline×sensitivity,min_threshold); Where baseline is the dynamic baseline value, sensitivity is the preset sensitivity multiplier, min_threshold is the preset lower threshold, and max(x, y) means taking the larger value between x and y.

5. The method as described in claim 1, characterized in that, The process of performing sound activity detection on audio data in fixed-duration analysis windows to generate an audio event stream with time information includes: If it is determined that the currently waiting sound start event is triggered, and the current analysis window in the audio data is not the last analysis window in the audio data, and the current analysis window satisfies the first and second conditions, then the sound start event is triggered; or If it is determined that the current waiting sound end event has been triggered, and the current analysis window in the audio data is not the last analysis window in the audio data, and the current analysis window meets the third condition, and the N3 consecutive analysis windows before the current analysis window all meet the third condition, then the sound end event is triggered. The first condition is that the decibel level of the analysis window is greater than the first threshold. The second condition is that the average decibel level of the N4 consecutive analysis windows before the analysis window is less than or equal to the first threshold, or the average decibel level of the N4 consecutive analysis windows before the analysis window is greater than the first threshold and the difference between the decibel level of the analysis window and the average decibel level is greater than or equal to the second threshold. The third condition is that the decibel level of the analysis window is less than or equal to the first threshold. N3 and N4 are both positive integers.

6. The method as described in claim 5, characterized in that, The timing information for the sound start event is the playback time of the analysis window corresponding to the sound start event, and the timing information for the sound end event is the playback time of the first analysis window among the N3 consecutive analysis windows preceding the analysis window corresponding to the sound end event.

7. The method as described in claim 1, characterized in that, Also includes: If it is determined that no audio is associated during the opening of the new motion scene, the target sound start event is identified from the candidate sound start events, and the time information of the target sound start event is used as the audio-side start time reference first_sound_ms of the new motion scene. The candidate sound start event is the sound start event in which the absolute value of the difference between the time information of the unassociated sound start event and the video-side start time reference first_motion_ms of the new motion scene is less than or equal to the third threshold. The target sound start event is the candidate sound start event in which the time interval between the time information of the candidate sound start event and the video-side start time reference first_motion_ms of the new motion scene is the shortest.

8. The method as described in claim 1, characterized in that, The calculation of audio-visual synchronization deviation for each matched motion scene includes: For each matched motion scene, calculate the absolute value of the difference between the video-side start time reference first_motion_ms and the audio-side start time reference first_sound_ms. Determine whether the absolute value is less than or equal to the fourth threshold. If the absolute value is less than or equal to the fourth threshold, then the audio and video of the motion scene are determined to be synchronized; otherwise, the audio and video of the motion scene are determined to be out of sync.

9. An automatic audio-video synchronization detection device, characterized in that, include: The decoding unit is used to decode the video stream and audio stream of the media file to be processed, so as to obtain video data and audio data respectively. The detection unit is used to perform motion detection on video data, generate a video motion event stream with time information, and perform sound activity detection on audio data in units of fixed-duration analysis windows, generating an audio sound event stream with time information. The video motion event stream includes motion start event and motion end event, and the audio sound event stream includes sound start event and sound end event. The calculation unit is used to track the state of motion scene based on video motion event stream, match the sound start event with the tracked motion scene, and calculate the audio and video synchronization deviation of each matched motion scene. The motion scene state tracking based on video motion event stream, which matches the sound initiation event with the tracked motion scene, includes: After each motion start event is generated, if it is determined that the time since the end of the last motion scene exceeds the preset cooldown time, a new motion scene is started, and the time information of the generated motion start event is used as the video-side start time reference first_motion_ms for the new motion scene. If it is determined that the audio state is in a sound state when the new motion scene is opened, then the time information of the most recent sound start event is used as the audio-side start time base first_sound_ms of the new motion scene; or if it is determined that a sound start event was generated during the opening of the new motion scene and the new motion scene was not associated with audio when the sound start event was generated, then the time information of the generated sound start event is used as the audio-side start time base first_sound_ms of the new motion scene. If it is determined that a motion end event was generated during the opening of the new motion scene, then the new motion scene is closed.

Images