Video data processing method and apparatus, and electronic device

Abstract

The present application relates to the technical field of video data processing, and especially relates to a video data processing method, device and electronic equipment, the method comprising: inputting video data by user interaction, and preprocessing the video data; performing continuity analysis on the frame images of the video data, and performing scene classification on the frame images of the video data according to the continuity analysis result, so as to obtain each video scene; performing behavior recognition on each video scene, so as to obtain a behavior breakpoint; performing audio detection on the behavior breakpoint, identifying abnormal change points in an audio file, and matching the audio detection result of the behavior breakpoint with the abnormal change points in the audio file, so as to take the abnormal change points that fail to match as audio abnormal breakpoints in the audio file; and outputting the detection result to the user according to the judgment result of the audio abnormal breakpoints and the audio detection result of the behavior breakpoint. The present application effectively improves the processing efficiency of video data.

Description

Technical Field

[0001] This invention relates to the field of video data processing technology, and in particular to a video data processing method, apparatus, and electronic device. Background Technology

[0002] Currently, video data processing technology plays a crucial role in video quality inspection and post-production. Traditional methods for detecting video data defects mainly rely on manual review or simple image processing techniques, which are not only inefficient but also prone to overlooking detailed issues, making them unsuitable for large-scale video production.

[0003] Existing technologies lack effective audiovisual fusion analysis methods, making it difficult to accurately identify common problems such as audio-visual asynchrony and abnormal noise. Analysis models based solely on video or audio struggle to accurately match behavioral breakpoints with audio anomalies, resulting in low detection accuracy and hindering the improvement of automated video data processing. Summary of the Invention

[0004] The purpose of this invention is to provide a video data processing method, apparatus, and electronic device to solve at least one of the problems existing in the prior art.

[0005] To achieve the above objectives, according to one aspect of this application, the present invention provides a video data processing method, comprising:

[0006] The process involves performing continuous analysis on the frame-by-frame images of the video data, classifying the video scenes based on the results of the continuous analysis, obtaining each video scene, fusing and analyzing each video scene using audio files, and updating the scene classification process based on the analysis results.

[0007] Behavior recognition is performed on each video scene to obtain behavior breakpoints, and audio detection is performed on behavior breakpoints based on audio files, frame frequency, and the corresponding time of frame images and audio files.

[0008] Identify abnormal change points in audio files and match the audio detection results of behavioral breakpoints with the abnormal change points in the audio files. Abnormal change points that fail to match are taken as audio abnormal breakpoints in the audio files.

[0009] Optionally, a continuity analysis model is established, and the framed images of the video data are input into the continuity analysis model to obtain the continuity analysis results of the framed images of the video data;

[0010] Based on the continuity analysis results of the frame images of the video data, the frame images of the video data are classified into scenes to obtain each video scene;

[0011] The audio files are used to perform fusion analysis on each video scene, and the scene classification process is updated based on the analysis results.

[0012] Optionally, the image similarity between adjacent frame images is calculated and denoted as β(i,i+1); where i represents the sequence number of the frame image in the video data, and β(i,i+1) represents the image similarity between the i-th frame image and the (i+1)-th frame image.

[0013] A continuity judgment threshold is set to judge the continuity of consecutive frame images: when β(i,i+1) is less than the continuity judgment threshold, the i-th frame image and the (i+1)-th frame image are judged to be continuous; otherwise, the i-th frame image and the (i+1)-th frame image are judged to be discontinuous.

[0014] Optionally, based on the continuity analysis results between adjacent frame images, a sequence of consecutive frame images can be statistically analyzed to divide the consecutive frame image sequence into various video scenes.

[0015] Optionally, the audio files can be divided into audio ranges corresponding to each video scene.

[0016] Calculate the playback duration t of the frame image and set the interval detection duration T. Then, combine the interval detection duration at the end of the j-th video scene and the interval detection duration at the beginning of the (j+1)-th video scene to form the audio continuous verification interval of the j-th video scene and the (j+1)-th video scene. Calculate the fluctuation index of each audio continuous verification interval and denote it as α(j,j+1).

[0017] An audio fluctuation threshold is set and compared with the fluctuation index of each audio continuous verification interval. The scene classification process is then updated based on the comparison result: when α(j,j+1) is less than the audio fluctuation threshold, the j-th video scene and the (j+1)-th video scene are determined to be continuous video scenes and are merged into one video scene; when α(j,j+1) is greater than or equal to the audio fluctuation threshold, the j-th video scene and the (j+1)-th video scene are determined to be non-continuous video scenes and no update is performed.

[0018] Optionally, the time of the audio file corresponding to the behavior breakpoint is obtained, and the breakpoint detection interval corresponding to the behavior breakpoint is extracted using the time of the audio file corresponding to the behavior breakpoint as the midpoint of the interval. The extraction process of the breakpoint detection interval is as follows: select the interval from the audio file with the behavior breakpoint as the midpoint of the interval and the interval detection duration T as the interval radius.

[0019] The process of audio detection for each breakpoint detection interval is as follows;

[0020] If ZF[a(k)] = max{ZF[a(k)-T,a(k)]}, then the audio detection of the breakpoint is considered normal; if ZF[a(k)] ≠ max{ZF[a(k)-T,a(k)]}, then the audio detection of the breakpoint is considered abnormal.

[0021] Where a(k) is the time of the midpoint of the interval within the detection interval of the kth breakpoint, and ZF[a(k)] represents the amplitude of point a(k).

[0022] Optionally, the process of identifying abnormal change points in the audio file is as follows: the amplitude of each time point in the audio file is judged, and if ZF(b) is greater than PJ{[bT,b)}, then the time point is regarded as an abnormal change point in the audio file; where PJ{[bT,b)} represents the average amplitude within the interval [bT,b), and ZF(b) represents the amplitude of the audio file at time point b.

[0023] Optionally, abnormal audio breakpoints can be output to the user as abnormal noise points in the audio file;

[0024] The abnormal behavior breakpoints detected in the audio are output to the user as suspicious points of audio-visual desynchronization.

[0025] According to another aspect of this application, a video data processing apparatus is provided, comprising:

[0026] The data preprocessing unit is used to interactively input video data and preprocess the video data to obtain the video data's frame images, audio files, frame rate, and the corresponding time of the frame images and audio files;

[0027] The scene classification unit is used to perform continuity analysis on the frame images of the video data, and to classify the scene images of the video data according to the continuity analysis results, so as to obtain each video scene.

[0028] The audio detection unit is used to perform behavior recognition on each video scene to obtain behavior breakpoints, and to perform audio detection on the behavior breakpoints based on the audio file, frame frequency, and the corresponding time of the frame image and the audio file.

[0029] The breakpoint analysis unit is used to identify abnormal change points in the audio file and match the audio detection results of the behavioral breakpoints with the abnormal change points in the audio file, so that the abnormal change points that fail to match are taken as audio abnormal breakpoints in the audio file.

[0030] The detection output unit is used to output the detection results to the user with the judgment results of audio abnormal breakpoints and the audio detection results of behavioral breakpoints.

[0031] According to another aspect of this application, an electronic device is provided, the electronic device comprising:

[0032] One or more processors;

[0033] Storage device for storing one or more programs;

[0034] When the one or more programs are executed by the one or more processors, the one or more processors implement the video data processing method.

[0035] Compared with existing technologies, the advantages of this invention are as follows: By combining the continuity analysis of video frame images with the fusion verification of audio information, accurate segmentation and dynamic optimization of video scenes are achieved. Through behavior recognition of each scene and temporal matching analysis of behavior breakpoints using audio detection technology, abnormal situations such as audio-visual asynchrony are effectively identified. Simultaneously, by utilizing the detection and matching mechanism of abnormal audio change points, normal sound behavior is accurately distinguished from real audio noise, improving detection accuracy. The overall solution achieves collaborative analysis of audiovisual information, not only improving the automation level and efficiency of video data processing but also significantly enhancing the accuracy and reliability of defect identification. It is suitable for quality monitoring scenarios in video production and has good practical value and promising prospects for promotion. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a flowchart illustrating the video data processing method of this embodiment.

[0038] Figure 2 This is a flowchart illustrating the video scene classification method in this embodiment.

[0039] Figure 3 This is a flowchart illustrating the audio detection method in this embodiment.

[0040] Figure 4 This is a schematic diagram of the video data processing device provided in this embodiment.

[0041] Figure 5 This is a schematic diagram of the structure of the electronic device provided in this embodiment. Detailed Implementation

[0042] To more clearly illustrate the present invention, the following description, in conjunction with preferred embodiments and accompanying drawings, further explains the invention. Similar components in the drawings are indicated by the same reference numerals. Those skilled in the art should understand that the specific description below is illustrative rather than restrictive and should not be construed as limiting the scope of protection of the present invention.

[0043] It should be noted that although the terms first, second, third, etc., may be used in the embodiments of this application for description, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of this application, first can also be referred to as second, and similarly, second can also be referred to as first.

[0044] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0045] Specifically, the video data processing method described in this application can be applied to the defect monitoring of video data, so as to realize the quality monitoring of video data and mark the defect location through the technical means of behavior recognition and audio-visual analysis; the video data described in this application is a video that does not have background music added during the video production process and only contains environmental audio and human audio.

[0046] To apply the above-mentioned application scenarios, this application provides a video data processing method, the flowchart of which can be found in the document. Figure 1 The above includes:

[0047] Step S101: The user interactively inputs video data and preprocesses the video data to obtain the frame images, audio files, frame frequency, and the corresponding time between the frame images and the audio files. The corresponding time between the frame images and the audio files is specifically the correspondence between each frame of the audio file and each frame of the frame images, and the correspondence is a one-to-one mapping relationship.

[0048] Specifically, this application does not specify the process of preprocessing video data, and the technical means are those disclosed in the prior art, which will not be described in detail in this application.

[0049] Specifically, by preprocessing the user-input video data, frame-by-frame images, audio files, and their corresponding temporal relationships are effectively extracted, providing a standardized data foundation for subsequent analysis. Step S101 ensures data consistency and integrity, improving the stability and accuracy of the overall processing flow.

[0050] Please continue reading. Figure 1 As shown, the video data processing method further includes:

[0051] Step S102: Perform continuity analysis on the frame images of the video data, and classify the scene of the frame images of the video data based on the continuity analysis results to obtain each video scene.

[0052] Please see Figure 2 As shown, it is a flowchart illustrating the video scene classification method provided in this application, including:

[0053] Step S201: Establish a continuity analysis model and input the framed images of the video data into the continuity analysis model to obtain the continuity analysis results of the framed images of the video data.

[0054] Specifically, in step S201, the process of performing continuity analysis on the frame images of the video data is as follows:

[0055] Calculate the image similarity between adjacent frames and denote it as β(i,i+1); where i represents the sequence number of the frame image in the video data, and β(i,i+1) represents the image similarity between the i-th frame image and the (i+1)-th frame image.

[0056] A continuity judgment threshold is set to judge the continuity of consecutive frame images: when β(i,i+1) is less than the continuity judgment threshold, the i-th frame image and the (i+1)-th frame image are judged to be continuous; otherwise, the i-th frame image and the (i+1)-th frame image are judged to be discontinuous.

[0057] Specifically, the continuity judgment threshold in this application is 0.85; the calculation process of the image similarity β(i,i+1) between adjacent frame images in this application is a prior art well known to those skilled in the art. In this application, the feature vector of the image can be extracted using a pre-trained model based on deep learning, and then the cosine similarity can be used as the image similarity of the group.

[0058] Please continue reading. Figure 1 As shown, the video scene classification method further includes:

[0059] Step S202: Based on the continuity analysis results of the frame images of the video data, the frame images of the video data are classified into scenes to obtain each video scene.

[0060] Specifically, in step S202, the process of classifying the scene of the frame images of the video data is as follows:

[0061] Based on the continuity analysis results between adjacent frame images, a sequence of consecutive frame images is statistically analyzed to divide the consecutive frame image sequence into various video scenes.

[0062] For example, the phrase "taking consecutive frame images as a type of video scene" in this application means taking a segment of consecutive frame images as a type of video scene.

[0063] Specifically, by incorporating audio information to fuse and dynamically update the preliminary scene classification results, the potential for misjudgments arising from relying solely on image analysis was effectively corrected. Through audiovisual collaborative verification, the accuracy of scene boundary determination was improved, enhancing the robustness and rationality of scene segmentation.

[0064] Please continue reading. Figure 2 As shown, the video scene classification method further includes:

[0065] Step S203: Perform fusion analysis on each video scene using audio files, and update the scene classification process based on the analysis results.

[0066] Specifically, in step S203, the process of updating the scene classification process is as follows:

[0067] The audio files are divided into audio ranges corresponding to each video scene.

[0068] Calculate the playback duration t of the frame images and set the interval detection duration T. Then, combine the interval detection duration at the end of the j-th video scene and the interval detection duration at the beginning of the (j+1)-th video scene to form the audio continuous verification interval of the j-th video scene and the (j+1)-th video scene. Calculate the fluctuation index of each audio continuous verification interval and denote it as α(j,j+1). Set α(j,j+1)=CV(j,j+1); where CV(j,j+1) represents the coefficient of variation of the audio amplitude in the j-th video scene and the (j+1)-th video scene.

[0069] An audio fluctuation threshold is set and compared with the fluctuation index of each audio continuous verification interval. The scene classification process is then updated based on the comparison result: when α(j,j+1) is less than the audio fluctuation threshold, the j-th video scene and the (j+1)-th video scene are determined to be continuous video scenes and are merged into one video scene; when α(j,j+1) is greater than or equal to the audio fluctuation threshold, the j-th video scene and the (j+1)-th video scene are determined to be non-continuous video scenes and no update is performed.

[0070] Specifically, the calculation process of the playback duration t of the framed image described in this application is set by taking the value of the reciprocal of the frame frequency. At the same time, the value of the audio fluctuation threshold described in this application is 30%. The coefficient of variation in "CV(j,j+1) represents the coefficient of variation of audio amplitude in the j-th video scene and the (j+1)-th video scene" is a publicly available technology, and its calculation process will not be described in detail.

[0071] Specifically, by performing continuity analysis on frame-by-frame images and classifying scenes, logically coherent video scenes can be automatically segmented. This process improves the accuracy of scene segmentation, provides clear structured units for subsequent behavior recognition and audio analysis, and enhances the system's ability to understand video content.

[0072] Please continue reading. Figure 1 As shown, the video data processing method further includes:

[0073] Step S103: Perform behavior recognition on each video scene to obtain behavior breakpoints, and perform audio detection on the behavior breakpoints based on the audio file, frame frequency, and the corresponding time of the frame image and the audio file.

[0074] Please see Figure 3 The diagram shown is a flowchart of the audio detection method provided in this application, including:

[0075] Step S301: Perform behavior recognition on each video scene and obtain behavior breakpoints.

[0076] Specifically, in step S301, the process of obtaining the behavior breakpoint is as follows:

[0077] Behavior recognition is performed on each video scene, and the recognized behaviors are classified and labeled to classify them into audio behaviors and non-audio behaviors. The starting frame image of the audio behavior is used as the behavior breakpoint.

[0078] For example, this application does not specifically limit the implementation process of "behavior recognition for each video scene". Those skilled in the art can freely set it as long as it meets their needs. In this application, the specific implementation process of image classification and object detection based on convolutional neural networks and the establishment of behavior database to identify objects and scenes in the video scene is a prior art well known to those skilled in the art. This application does not describe its implementation process in detail.

[0079] Please continue reading. Figure 3 As shown, the audio detection method further includes:

[0080] Step S302: Audio detection is performed on the behavior breakpoints based on the audio file, frame frequency, and the corresponding time of the framed image and the audio file.

[0081] Specifically, in step S302, the process of audio detection for behavioral breakpoints is as follows:

[0082] Obtain the time of the audio file corresponding to the behavior breakpoint, and use the time of the audio file corresponding to the behavior breakpoint as the midpoint of the interval to extract the breakpoint detection interval corresponding to the behavior breakpoint. The process of extracting the breakpoint detection interval is as follows: select the interval from the audio file with the behavior breakpoint as the midpoint of the interval and the interval detection duration T as the interval radius.

[0083] The process of performing audio detection on each breakpoint detection interval is as follows;

[0084] If ZF[a(k)] = max{ZF[a(k)-T,a(k)]}, then the audio detection of the breakpoint is considered normal; if ZF[a(k)] ≠ max{ZF[a(k)-T,a(k)]}, then the audio detection of the breakpoint is considered abnormal.

[0085] Where a(k) is the time of the midpoint of the interval within the detection interval of the kth breakpoint, and ZF[a(k)] represents the amplitude of point a(k).

[0086] It is worth noting that the operation “max{ZF[a(k)-T,a(k)]}” in this application calculates the maximum value of the amplitude within the time interval [a(k)-T,a(k)].

[0087] Specifically, by performing behavior recognition on various video scenes and combining it with audio detection, abnormal situations such as audio-visual asynchrony can be accurately located. This step achieves temporal alignment analysis of visual behavior and audio signals, improving the accuracy of recognizing the matching state of key actions and sounds.

[0088] Please continue reading. Figure 1 As shown, the video data processing method further includes:

[0089] Step S104: Identify abnormal change points in the audio file, and match the audio detection results of the behavior breakpoints with the abnormal change points in the audio file, so that the abnormal change points that fail to match are taken as audio abnormal breakpoints in the audio file.

[0090] Specifically, in step S104, the process of identifying abnormal change points in the audio file is as follows: the amplitude of each time point in the audio file is judged, and if ZF(b) is greater than PJ{[bT,b)}×η, then the time point is regarded as an abnormal change point in the audio file; where PJ{[bT,b)} represents the average amplitude in the interval [bT,b), ZF(b) represents the amplitude of the audio file at time point b, and η represents the standard offset coefficient.

[0091] It is understood that the abnormal change points mentioned in this application refer to the time in the audio file; at the same time, the standard offset coefficient mentioned in this application is 120%.

[0092] Specifically, in step S104, the process of matching abnormal change points is as follows: if the abnormal change point does not belong to the set of times of the audio file corresponding to the normal behavior breakpoint of the audio detection, the matching is determined to be unsuccessful, and the unmatched abnormal change point is taken as the audio abnormal breakpoint in the audio file; otherwise, the matching is determined to be successful.

[0093] Specifically, by identifying abnormal changes in audio and matching them with behavioral breakpoints, the system effectively distinguishes normal sound behavior from real audio noise. This mechanism avoids misclassifying valid sound as abnormal, significantly improving the accuracy and reliability of audio defect detection.

[0094] Please continue reading. Figure 1 As shown, the video data processing method further includes:

[0095] Step S105: Output the detection results to the user based on the judgment results of the audio abnormal breakpoints and the audio detection results of the behavioral breakpoints.

[0096] Specifically, in step S105, the process of outputting the detection results to the user is as follows:

[0097] The abnormal breakpoints in the audio file are output to the user as abnormal noise points in the audio file;

[0098] The abnormal behavior breakpoints detected in the audio are output to the user as suspicious points of audio-visual desynchronization.

[0099] Specifically, by combining the results of abnormal audio breakpoints and behavioral audio detection, a clear detection report is output to the user, facilitating the rapid identification of quality issues. This step enables the visualization and categorization of detection results, improves human-computer interaction efficiency, and provides strong support for video post-processing.

[0100] Please see Figure 4 As shown, it is a structural schematic diagram of the video data processing apparatus provided in this application, including:

[0101] The data preprocessing unit is used to interactively input video data and preprocess the video data to obtain the video data's frame images, audio files, frame rate, and the corresponding time of the frame images and audio files;

[0102] The scene classification unit is used to perform continuity analysis on the frame images of the video data, and to classify the scene images of the video data according to the continuity analysis results, so as to obtain each video scene.

[0103] The audio detection unit is used to perform behavior recognition on each video scene to obtain behavior breakpoints, and to perform audio detection on the behavior breakpoints based on the audio file, frame frequency, and the corresponding time of the frame image and the audio file.

[0104] The breakpoint analysis unit is used to identify abnormal change points in the audio file and match the audio detection results of the behavioral breakpoints with the abnormal change points in the audio file, so that the abnormal change points that fail to match are taken as audio abnormal breakpoints in the audio file.

[0105] The detection output unit is used to output the detection results to the user with the judgment results of audio abnormal breakpoints and the audio detection results of behavioral breakpoints.

[0106] The video data processing apparatus provided in this application embodiment can execute the video data processing method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the method.

[0107] From a hardware perspective, to enable the video data processing method to function in a computer, this application also provides an electronic device; please refer to [link to relevant documentation]. Figure 5 As shown, it is a schematic diagram of the structure of the electronic device described in this application, including:

[0108] The system comprises a processor 1, a memory 2, a communication interface 3, and a bus 4; wherein the processor 1 and the memory 2, and the memory 2 and the communication interface 3, transmit data via the bus 4; the processor is used to process data in the memory and generate commands, the memory is used to store data, the communication interface is used to receive and send data, and the bus is used to realize data transmission between the processor, the memory, and the communication interface.

[0109] In this embodiment, the video data processing method can be implemented as a runnable computer program. When the computer program is loaded into the processor or into the memory and processed by the processor via the bus, one or more steps of the video data processing method can be executed.

[0110] This embodiment also provides a computer-readable storage medium for storing the computer-executable instructions. The computer-readable storage medium is a tangible physical storage medium that can store the computer program and various types of data used in the program. The physical storage medium includes, but is not limited to, existing physical storage media or combinations thereof, such as random access memory, read-only memory, optical disk, and hard disk.

[0111] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A video data processing method, characterized in that, include: The process involves performing continuous analysis on the frame-by-frame images of the video data, classifying the video scenes based on the results of the continuous analysis, obtaining each video scene, fusing and analyzing each video scene using audio files, and updating the scene classification process based on the analysis results. Behavior recognition is performed on each video scene, and the recognized behaviors are classified and labeled to classify the recognized behaviors into sound behaviors and non-sound behaviors. The starting frame image of the sound behavior is used as the behavior breakpoint, and audio detection is performed on the behavior breakpoint based on the audio file, frame frequency, and the corresponding time of the frame image and the audio file. Identify abnormal change points in the audio file and match the normal audio detection results of the behavioral breakpoints with the abnormal change points in the audio file. The abnormal change points that fail to match are taken as audio abnormal breakpoints in the audio file. The process of identifying abnormal change points in an audio file is as follows: the amplitude of each time point in the audio file is judged. If ZF(b) is greater than PJ{[bT,b)}×η, then the time point is regarded as an abnormal change point in the audio file; where PJ{[bT,b)} represents the average amplitude in the interval [bT,b), ZF(b) represents the amplitude of the audio file at time point b, and η represents the standard offset coefficient. The abnormal breakpoints in the audio file are output to the user as abnormal noise points in the audio file; The abnormal behavior breakpoints detected in the audio are output to the user as suspicious points of audio-visual desynchronization.

2. The video data processing method according to claim 1, characterized in that, A continuity analysis model is established, and the framed images of the video data are input into the continuity analysis model to obtain the continuity analysis results of the framed images of the video data. Based on the continuity analysis results of the frame images of the video data, the frame images of the video data are classified into scenes to obtain each video scene; The audio files are used to perform fusion analysis on each video scene, and the scene classification process is updated based on the analysis results.

3. The video data processing method according to claim 2, characterized in that, Calculate the image similarity between adjacent frames and denote it as β(i,i+1); where i represents the sequence number of the frame image in the video data, and β(i,i+1) represents the image similarity between the i-th frame image and the (i+1)-th frame image. A continuity judgment threshold is set to judge the continuity of consecutive frame images: when β(i,i+1) is less than the continuity judgment threshold, the i-th frame image and the (i+1)-th frame image are judged to be continuous; otherwise, the i-th frame image and the (i+1)-th frame image are judged to be discontinuous.

4. The video data processing method according to claim 3, characterized in that, Based on the continuity analysis results between adjacent frame images, a sequence of consecutive frame images is statistically analyzed to divide the consecutive frame image sequence into various video scenes.

5. The video data processing method according to claim 4, characterized in that, The audio files are divided into audio ranges corresponding to each video scene. Calculate the playback duration t of the frame images and set the interval detection duration T. Then, combine the interval detection duration at the end of the j-th video scene and the interval detection duration at the beginning of the (j+1)-th video scene to form the audio continuous verification interval of the j-th video scene and the (j+1)-th video scene. Calculate the fluctuation index of each audio continuous verification interval and denote it as α(j,j+1). Set α(j,j+1)=CV(j,j+1); where CV(j,j+1) represents the coefficient of variation of the audio amplitude in the j-th video scene and the (j+1)-th video scene. Set an audio fluctuation threshold and compare it with the fluctuation index of each audio continuous verification interval. Then update the scene classification process with the comparison result: when α(j,j+1) is less than the audio fluctuation threshold, determine that the j-th video scene and the j+1-th video scene are continuous video scenes and merge the j-th video scene and the j+1-th video scene into one video scene. When α(j,j+1) is greater than or equal to the audio fluctuation threshold, the j-th video scene and the (j+1)-th video scene are determined to be non-continuous video scenes and no update is performed.

6. The video data processing method according to claim 5, characterized in that, Obtain the time of the audio file corresponding to the behavior breakpoint, and use the time of the audio file corresponding to the behavior breakpoint as the midpoint of the interval to extract the breakpoint detection interval corresponding to the behavior breakpoint. The process of extracting the breakpoint detection interval is as follows: select the interval from the audio file with the behavior breakpoint as the midpoint of the interval and the interval detection duration T as the interval radius. The process of performing audio detection on each breakpoint detection interval is as follows; If ZF[a(k)] = max{ZF[a(k)-T,a(k)]}, then the audio detection of the breakpoint is considered normal; if ZF[a(k)] ≠ max{ZF[a(k)-T,a(k)]}, then the audio detection of the breakpoint is considered abnormal. Where a(k) is the time of the midpoint of the interval within the detection interval of the kth breakpoint, ZF[a(k)] represents the amplitude of point a(k), and the operation "max{ZF[a(k)-T,a(k)]}" calculates the maximum value of the amplitude within the time interval [a(k)-T,a(k)].

7. A video data processing apparatus, applied to the video data processing method as described in any one of claims 1-6, characterized in that, include: The data preprocessing unit is used to interactively input video data and preprocess the video data to obtain the video data's frame images, audio files, frame rate, and the corresponding time of the frame images and audio files; The scene classification unit is used to perform continuity analysis on the frame images of the video data, and to classify the scene images of the video data according to the continuity analysis results, so as to obtain each video scene. The audio detection unit is used to perform behavior recognition on each video scene to obtain behavior breakpoints, and to perform audio detection on the behavior breakpoints based on the audio file, frame frequency, and the corresponding time of the frame image and the audio file. The breakpoint analysis unit is used to identify abnormal change points in the audio file and match the audio detection results of the behavioral breakpoints with the abnormal change points in the audio file, so that the abnormal change points that fail to match are taken as audio abnormal breakpoints in the audio file. The detection output unit is used to output the detection results to the user with the judgment results of audio abnormal breakpoints and the audio detection results of behavioral breakpoints.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video data processing method according to any one of claims 1-6.

Images