Video editing method and device, electronic equipment and nonvolatile storage medium

Abstract

The invention discloses a video editing method and device, electronic equipment and a nonvolatile storage medium. The method comprises the following steps: acquiring an audio data stream of a target video, and segmenting the audio data stream into a plurality of audio clips; determining a classification result corresponding to the audio clip, and determining a time period corresponding to the audio clip as a candidate ball hitting time period when the classification result is that the audio clip contains the ball hitting sound; acquiring a video frame corresponding to the candidate ball hitting time period in the target video, and judging that the candidate ball hitting time period is a real ball hitting time period under the condition that the visual feature of the video frame is matched with a preset ball hitting rule; and determining an editing time point according to the real ball hitting time period, and editing the target video according to the editing time point to obtain a ball hitting round video clip. The technical problem that the accuracy of ball hitting detection segment detection is low due to the fact that ball hitting detection is conducted only through sound and is easily interfered by environmental noise in the prior art is solved.

Description

Technical Field

[0001] This application relates to the field of video analysis technology, and more specifically, to a video editing method, apparatus, electronic device, and non-volatile storage medium. Background Technology

[0002] In live or recorded sports broadcasts, especially for net sports like badminton and tennis, post-production video editing typically involves manually selecting and cutting shot rallies. This process is not only time-consuming and labor-intensive but also inefficient. Current intelligent shot detection and editing methods primarily rely on single-modal information, such as audio-based shot detection. This makes them susceptible to ambient noise, leading to a high false detection rate and severely impacting the accuracy of shot event recognition.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a video editing method, apparatus, electronic device, and non-volatile storage medium to at least solve the technical problem of low accuracy in detecting ball-hitting segments due to the fact that related technologies rely solely on sound for ball-hitting detection, which is easily affected by environmental noise interference.

[0005] According to one aspect of the embodiments of this application, a video editing method is provided, comprising: acquiring an audio data stream of a target video and dividing the audio data stream into multiple audio segments; determining the classification result corresponding to the audio segments, and if the classification result indicates that the audio segment contains a hitting sound, determining the time period corresponding to the audio segment as a candidate hitting time period; acquiring video frames in the target video corresponding to the candidate hitting time periods, and determining the candidate hitting time period as a real hitting time period if the visual features of the video frames match a preset hitting rule, wherein the preset hitting rule is used to characterize the spatial positional relationship that the visual features should satisfy when performing a hitting action; determining the editing time point based on the real hitting time period, and editing the target video according to the editing time point to obtain a video segment of the hitting round.

[0006] Optionally, dividing the audio data stream into multiple audio segments includes: performing preprocessing operations on the audio data stream, wherein the preprocessing operations include at least one of the following: noise reduction processing and normalization processing; determining the window length of the time window and the corresponding movement step size of the time window, wherein the movement step size is not greater than the window length; starting from the beginning position of the audio data stream, dividing the audio data stream into multiple audio segments by gradually moving the time window according to the movement step size.

[0007] Optionally, determining the classification result corresponding to the audio segment includes: obtaining the audio features corresponding to the audio segment, wherein the audio features include at least one of the following: Mel frequency cepstral coefficient features, spectrogram features; using a sound classification model to analyze the audio features and obtain the classification result corresponding to the audio features, wherein the sound classification model is trained based on a target training set, the target training set includes: multiple audio segment samples, and the classification result label corresponding to each audio segment sample, the classification result including: near ball hitting sound, far ball hitting sound, ball hitting sound from other venues, and others.

[0008] Optionally, after acquiring the video frames corresponding to the candidate hitting time periods in the target video, the method further includes: using a target detection model to extract the first image features of the video frames, and determining the position of key target objects in the video frames based on the first image features, wherein the key target objects include: the hitter and the racket; using a ball tracking model to extract the second image features of the video frames, and determining the position and trajectory of the ball in the video frames based on the second image features, wherein the model structure and computational complexity of the target detection model are both lower than those of the ball tracking model.

[0009] Optionally, the visual features of the video frame include: the position of the key target object in the video frame, the position and trajectory of the ball in the video frame; the preset hitting rules include at least one of the following: in the video frame, the hitter and the racket appear simultaneously, and the hitter's body movements conform to the characteristics of a swing action; in the video frame, the ball is detected, and the position of the ball is within the hitting range of the racket; the key target object is located within the area corresponding to the target field.

[0010] Optionally, determining whether a candidate hitting time period is a real hitting time period further includes: determining a first time point corresponding to the hitting sound in the audio segment corresponding to the candidate hitting time period; determining a second time point corresponding to the video frame where the hitting event occurs, wherein the video frame where the hitting event occurs is a video frame containing the moment when the hitter swings the racket and hits the ball; if the first time point and the second time point are the same, and the visual features of the video frame meet the preset hitting rules, then the candidate hitting time period is determined to be a real hitting time period.

[0011] Optionally, the target video is edited according to the editing time points to obtain the video clip of the hitting round, including: determining the first real hitting time period corresponding to the first hitting event of the hitting round and the second real hitting time period corresponding to the last hitting event of the hitting round; determining the first hitting time point within the first real hitting time period and the second hitting time point within the second real hitting time period; determining the time point with a preset duration before the first hitting time point as the start editing time point and the time point with a preset duration after the second hitting time point as the end editing time point; and editing the target video according to the start editing time point and the end editing time point to obtain the video clip of the hitting round.

[0012] According to another aspect of the embodiments of this application, a video editing apparatus is also provided, comprising: an audio segmentation module, configured to acquire an audio data stream of a target video and segment the audio data stream into multiple audio segments; an audio classification module, configured to determine the classification result corresponding to the audio segments, and, if the classification result is that the audio segment contains a hitting sound, determine the time period corresponding to the audio segment as a candidate hitting time period; a visual assistance module, configured to acquire video frames in the target video corresponding to the candidate hitting time periods, and, if the visual features of the video frames match a preset hitting rule, determine that the candidate hitting time period is a real hitting time period, wherein the preset hitting rule is used to characterize the spatial positional relationship that the visual features should satisfy when performing a hitting action; and a round editing module, configured to determine the editing time point based on the real hitting time period, and edit the target video according to the editing time point to obtain a hitting round video segment.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, the processor being configured to run a program stored in the memory, wherein the program executes a video editing method during runtime.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes a video editing method by running the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of a video editing method.

[0016] In this embodiment, the method involves acquiring the audio data stream of the target video and dividing it into multiple audio segments; determining the classification result corresponding to each audio segment; and, if the classification result indicates that the audio segment contains the sound of hitting the ball, identifying the time segment corresponding to the audio segment as a candidate hitting time segment; acquiring video frames in the target video that correspond to the candidate hitting time segments; and, if the visual features of the video frames match the preset hitting rules, determining the candidate hitting time segment as the actual hitting time segment. The preset hitting rules characterize the spatial relationship that the visual features should satisfy during the hitting action. Based on the actual hitting time segment, the method determines the editing time point and edits the target video according to the editing time point to obtain the hitting round video segment. By fusing audio and visual information, the method improves the accuracy of hitting round recognition, thereby solving the technical problem of low accuracy in hitting detection segments caused by relying solely on sound for hitting detection and being susceptible to environmental noise interference. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal (or electronic device) for implementing a video editing method according to an embodiment of this application;

[0019] Figure 2 This is a schematic diagram of a video editing method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a method for intelligent ball-hitting round editing based on audio and video multimodal analysis according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of a video editing device according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] Intelligent hit detection and editing methods in related technologies mainly rely on single-modal information, such as relying solely on audio or solely on video images. However, these methods have significant limitations and shortcomings, primarily including:

[0025] 1) Error problems caused by single-modal detection: When detecting a ball hit based solely on audio signals, it is easily affected by ambient noise, such as cheers from the audience or sounds of other movements in the field, resulting in a high false detection rate. For example, it seriously affects the accuracy of ball hit event recognition.

[0026] 2) Lack of spatiotemporal correlation analysis: The methods in the related technologies failed to effectively map the sound of hitting the ball in the audio signal to the actual hitting action in the video in space and synchronize them in time. This resulted in the inability to accurately locate the corresponding hitting event in the video even if the sound of hitting the ball was detected, which reduced the accuracy of editing.

[0027] 3) Resource consumption and efficiency issues: While relying entirely on visual models for ball detection and editing can improve accuracy, it consumes enormous computational resources and has excessively long inference times, making it difficult to meet the needs of real-time or rapid editing. Furthermore, this increases editing costs, limiting its application in large-scale video data processing.

[0028] To address the aforementioned issues, this application provides a solution that firstly utilizes an audio classification model for sliding window classification to perform coarse screening, significantly reducing the need for judgments in non-hitting rallies; then, a visual model is used for further motion screening and classification, ultimately efficiently obtaining accurate temporal intervals for hitting rallies, as detailed below.

[0029] According to an embodiment of this application, a method embodiment for video editing is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0030] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or electronic device) for implementing a video editing method is shown. Figure 1 As shown, the computer terminal 10 (or electronic device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0031] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or electronic device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video editing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned video editing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0034] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or electronic device).

[0035] Under the above operating environment, this application provides a video editing method. Figure 2 This is a schematic diagram of a video editing method flow according to an embodiment of this application, such as... Figure 2 As shown, the method includes the following steps:

[0036] Step S202: Obtain the audio data stream of the target video and divide the audio data stream into multiple audio segments;

[0037] Step S204: Determine the classification result corresponding to the audio segment, and if the classification result is that the audio segment contains the sound of hitting the ball, determine the time period corresponding to the audio segment as the candidate hitting time period;

[0038] Step S206: Obtain the video frame corresponding to the candidate hitting time period in the target video, and determine the candidate hitting time period as the real hitting time period if the visual features of the video frame match the preset hitting rules. The preset hitting rules are used to characterize the spatial positional relationship that the visual features should satisfy when performing the hitting action.

[0039] Step S208: Based on the actual hitting time, determine the editing time point, and edit the target video according to the editing time point to obtain the video segment of the hitting round.

[0040] By combining audio and visual information through the above steps, the accuracy of ball-hitting round recognition is improved, thereby solving the technical problem of low accuracy in ball-hitting segment detection caused by the fact that related technologies rely solely on sound for ball-hitting detection and are easily affected by environmental noise.

[0041] The video editing method in steps S202 to S208 of the embodiments of this application will be further described below.

[0042] Figure 3 This is a schematic diagram of a method for intelligent ball-hitting round editing based on audio and video multimodal analysis according to an embodiment of this application, as shown below. Figure 3 As shown, after inputting the original audio and video data (target video), a sliding window classification is first performed on the audio channel to extract candidate hitting time periods. Then, within the interval of the candidate hitting time periods, the video channel is extracted for YOLO target detection and TrackNet ball detection and rule judgment to verify the authenticity of the hitting. At the same time, the timestamp information is combined to match sound and visual events. Finally, the edited video clip of the hitting round is output. A detailed introduction follows.

[0043] First, after acquiring the target video, its audio channel is processed. Using a time window, the audio data stream of the target video is divided into multiple audio segments and sound classification and detection are performed. The specific steps are as follows.

[0044] In some embodiments of this application, segmenting an audio data stream into multiple audio segments includes the following steps: performing preprocessing operations on the audio data stream, wherein the preprocessing operations include at least one of the following: noise reduction processing and normalization processing; determining the window length of the time window and the corresponding movement step size of the time window, wherein the movement step size is not greater than the window length; starting from the beginning position of the audio data stream, segmenting the audio data stream by gradually moving the time window according to the movement step size to obtain multiple audio segments.

[0045] Preprocessing is crucial for improving the accuracy of audio analysis. Noise reduction removes background noise, making the sound of the ball hitting the ground clearer, while normalization ensures that audio segments of different volumes receive equal weight during analysis, avoiding misjudgments due to volume differences. The setting of the time window and the selection of the movement step size allow audio analysis to be performed on a continuous time series, ensuring both comprehensiveness and avoiding excessive redundancy.

[0046] Specifically, a sliding window can be used to divide the audio data stream into overlapping windows of fixed window length. Typically, the window length of the time window is set to the millisecond level to avoid each audio segment containing two or more hit events. For example, in this embodiment, a window can be set every 500 milliseconds with a movement step of 300 milliseconds.

[0047] After dividing the audio into multiple segments, each window (audio segment) is classified to determine whether it contains the sound of hitting the ball. The specific steps are as follows.

[0048] In some embodiments of this application, determining the classification result corresponding to an audio segment includes the following steps: obtaining the audio features corresponding to the audio segment, wherein the audio features include at least one of the following: Mel frequency cepstral coefficient features, spectrogram features; using a sound classification model to analyze the audio features and obtain the classification result corresponding to the audio features, wherein the sound classification model is trained based on a target training set, the target training set includes: multiple audio segment samples, and the classification result label corresponding to each audio segment sample, the classification result includes: near-field ball hitting sound, far-field ball hitting sound, other field ball hitting sound, and others.

[0049] Specifically, the audio features of each audio segment can be extracted, and then the trained sound classification model can be used to classify the audio segments based on the audio features. In this embodiment, the sound classification model can learn the feature patterns of different categories of ball-hitting sounds from a large number of training samples through deep learning technology. It can distinguish the classification results such as "near ball-hitting sound", "far ball-hitting sound", "ball-hitting sound from other venues" and "other", thereby accurately distinguishing the ball-hitting sounds from near, far and from other venues, and identifying sounds unrelated to ball-hitting, which significantly improves the accuracy and robustness of classification.

[0050] Record the time periods corresponding to all audio segments containing the sound of hitting the ball, as candidate hitting time periods. Since the sound may come from other venues or non-target players, visual information needs to be introduced for further verification to filter out real hitting events. First, obtain the video frames corresponding to the candidate hitting time periods in the target video, and then perform feature detection and analysis on the video frames. The specific steps are as follows.

[0051] In some embodiments of this application, after obtaining the video frame corresponding to the candidate hitting time period in the target video, the method further includes the following steps: using a target detection model to extract the first image features of the video frame, and determining the position of the key target object in the video frame based on the first image features, wherein the key target object includes: the hitter and the racket; using a ball tracking model to extract the second image features of the video frame, and determining the position and trajectory of the ball in the video frame based on the second image features, wherein the model structure and computational complexity of the target detection model are both smaller than those of the ball tracking model.

[0052] Specifically, on the one hand, an object detection model can be used to detect video frames during candidate hitting periods in real time and identify key target objects, including: people (the hitter) and rackets. Since the task of detecting people and rackets is relatively simple, models with low model structure and computational complexity can be used, such as the YOLO series of object detection models. On the other hand, a ball tracking model can be used to track and predict the position of a moving ball. Since the YOLO model generally has poor detection performance for small targets and cannot perceive the ball's motion information due to the input being a single-frame image, its ball tracking performance is not good. Therefore, in this embodiment, TrackNet is used as the ball tracking model to track moving balls.

[0053] In this embodiment, the target detection model is responsible for locating the striker and racket, while the ball tracking model focuses on tracking the ball's dynamics. Their collaborative work enables the system to accurately capture key information at the moment of impact. The target detection model is designed to be more lightweight, meaning it can run on devices with limited computing resources, such as smartphones, without affecting overall performance. Simultaneously, the high precision of the ball tracking model ensures accurate tracking of the ball's position and trajectory even in fast-moving scenarios. This is crucial for identifying whether the strike occurred within the target area, effectively avoiding misjudgments. This design not only lowers the system's operational threshold but also improves real-time processing capabilities, facilitating instant editing of sports videos.

[0054] After analyzing and processing the video frames to obtain their visual features, it is necessary to further verify whether these visual features conform to the spatiotemporal rules (preset hitting rules) corresponding to the hitting event. That is, to determine whether the visual features of the video frames match the preset hitting rules, which are as follows.

[0055] In some embodiments of this application, the visual features of the video frame include: the position of the key target object in the video frame, the position and trajectory of the ball in the video frame; the preset hitting rules include at least one of the following: in the video frame, the hitter and the racket appear simultaneously, and the hitter's limb movements conform to the characteristics of a racket swing; in the video frame, the ball is detected, and the position of the ball is within the hitting range of the racket; the key target object is located within the area corresponding to the target field.

[0056] The design of the preset hitting rules in this application aims to further verify the hitting events detected by audio through visual feature matching. For example, when the hitter and racket appear in the frame simultaneously, and the hitter's movements conform to the standard swing posture, the occurrence of the hitting event can be confirmed by combining the position and trajectory analysis of the ball. This rule system, through dual verification of spatial location and temporal sequence, greatly improves the accuracy of hitting event recognition, especially in complex scenarios of multi-court parallel matches, effectively eliminating interference and focusing on hitting events on the target court. Furthermore, by adjusting the parameters of the preset rules, such as the size of the hitting range and the boundary definition of the target court, the system can adapt to different types of sports, demonstrating good versatility and flexibility.

[0057] Furthermore, after filtering candidate hitting times using preset hitting rules, the process is further validated by combining timestamps. The specific steps are as follows.

[0058] In some embodiments of this application, determining whether a candidate hitting time period is a real hitting time period further includes the following steps: determining a first time point corresponding to the hitting sound in the audio segment corresponding to the candidate hitting time period; determining a second time point corresponding to the video frame where the hitting event occurs, wherein the video frame where the hitting event occurs is a video frame containing the moment when the hitter swings the racket and hits the ball; if the first time point and the second time point are the same, and the visual features of the video frame meet the preset hitting rules, then the candidate hitting time period is determined to be a real hitting time period.

[0059] Specifically, by comparing the timing of the ball-hitting sound detected in the audio with the timing of the ball-hitting action in the video, the results of the audio analysis can be verified, ensuring that the edited video clips contain the actual ball-hitting rallies. This is especially important when the speed of sound propagation and the video frame rate are mismatched; timing calibration improves the synchronization and accuracy of the editing. Simultaneously, this process demonstrates the complementarity of audio and visual information; working together, the system can more accurately identify ball-hitting events, providing users with a high-quality editing experience.

[0060] Once the actual hitting time has been determined, a complete video clip of the hitting round can be constructed. The specific steps are as follows.

[0061] In some embodiments of this application, editing the target video according to editing time points to obtain a video segment of a hitting round includes the following steps: determining the first real hitting time period corresponding to the first hitting event of the hitting round and the second real hitting time period corresponding to the last hitting event of the hitting round; determining the first hitting time point within the first real hitting time period and the second hitting time point within the second real hitting time period; determining the time point located before the first hitting time point by a preset duration as the starting editing time point and the time point located after the second hitting time point by a preset duration as the ending editing time point; and editing the target video according to the starting editing time point and the ending editing time point to obtain a video segment of the hitting round.

[0062] Specifically, the boundaries of the rounds to be edited can be defined first. For example, the start of a round is the first shot after the previous score ends (corresponding to the first shot event in the round), and the end of a round is the moment when one side makes a mistake or scores (corresponding to the last shot in the round). Then, a certain preset duration (e.g., 1 second before and after) can be added before and after the shot time (the first shot time and the second shot time) to create a complete round clip. In addition, the trajectory of the ball (through a target tracking algorithm) can be used to help determine the start and end of the round. Finally, an independent video file of the round video clip is generated, and metadata (such as the number of shots, player ID, etc.) can be attached to the video file.

[0063] This application embodiment achieves precise generation of video clips for a hitting rally by intelligently determining the editing time points. Determining the time points of the first and last hits not only captures the complete process of the hitting rally but also preserves key actions before and after the hit, such as the ready posture and the reaction after the hit, through preset duration extensions, making the edited video clips more visually appealing and complete.

[0064] This application significantly improves the accuracy and efficiency of ball-hitting rally recognition by integrating audio and visual information, opening up new avenues for automated editing of sports videos. In practical applications, this technology can be widely used for live and replay editing of various sports events, including but not limited to football, basketball, and volleyball, providing viewers with instant access to exciting moments and valuable training materials for coaches and athletes.

[0065] Compared to single-modal detection methods in related technologies, the false detection rate of this application is significantly reduced. Furthermore, by introducing spatiotemporal correlation analysis, interference from non-target field or environmental noise can be effectively eliminated, ensuring accurate positioning of the ball-hitting event. While maintaining high accuracy, it also enables rapid editing, greatly shortening processing time and providing strong technical support for real-time editing and post-production of sports events. In addition, the lightweight model structure of this application allows the entire system to run on resource-constrained devices, such as mobile devices, greatly expanding its application scope and providing a convenient video editing tool for sports enthusiasts and professionals.

[0066] According to an embodiment of this application, an embodiment of a video editing device is also provided. Figure 4 This is a schematic diagram of the structure of a video editing device according to an embodiment of this application. Figure 4 As shown, the device includes:

[0067] The audio segmentation module 40 is used to acquire the audio data stream of the target video and segment the audio data stream into multiple audio segments;

[0068] The audio classification module 42 is used to determine the classification result corresponding to the audio segment, and when the classification result is that the audio segment contains the sound of hitting the ball, the time period corresponding to the audio segment is determined as the candidate hitting time period;

[0069] The visual assistance module 44 is used to acquire video frames in the target video that correspond to the candidate hitting time period, and determine the candidate hitting time period as the real hitting time period when the visual features of the video frame match the preset hitting rules. The preset hitting rules are used to characterize the spatial positional relationship that the visual features should satisfy when performing the hitting action.

[0070] The round editing module 46 is used to determine the editing time point based on the actual hitting time, and to edit the target video according to the editing time point to obtain the hitting round video segment.

[0071] Optionally, dividing the audio data stream into multiple audio segments includes: performing preprocessing operations on the audio data stream, wherein the preprocessing operations include at least one of the following: noise reduction processing and normalization processing; determining the window length of the time window and the corresponding movement step size of the time window, wherein the movement step size is not greater than the window length; starting from the beginning position of the audio data stream, dividing the audio data stream into multiple audio segments by gradually moving the time window according to the movement step size.

[0072] Optionally, determining the classification result corresponding to the audio segment includes: obtaining the audio features corresponding to the audio segment, wherein the audio features include at least one of the following: Mel frequency cepstral coefficient features, spectrogram features; using a sound classification model to analyze the audio features and obtain the classification result corresponding to the audio features, wherein the sound classification model is trained based on a target training set, the target training set includes: multiple audio segment samples, and the classification result label corresponding to each audio segment sample, the classification result including: near ball hitting sound, far ball hitting sound, ball hitting sound from other venues, and others.

[0073] Optionally, after acquiring the video frames corresponding to the candidate hitting time periods in the target video, the method further includes: using a target detection model to extract the first image features of the video frames, and determining the position of key target objects in the video frames based on the first image features, wherein the key target objects include: the hitter and the racket; using a ball tracking model to extract the second image features of the video frames, and determining the position and trajectory of the ball in the video frames based on the second image features, wherein the model structure and computational complexity of the target detection model are both lower than those of the ball tracking model.

[0074] Optionally, the visual features of the video frame include: the position of the key target object in the video frame, the position and trajectory of the ball in the video frame; the preset hitting rules include at least one of the following: in the video frame, the hitter and the racket appear simultaneously, and the hitter's body movements conform to the characteristics of a swing action; in the video frame, the ball is detected, and the position of the ball is within the hitting range of the racket; the key target object is located within the area corresponding to the target field.

[0075] Optionally, determining whether a candidate hitting time period is a real hitting time period further includes: determining a first time point corresponding to the hitting sound in the audio segment corresponding to the candidate hitting time period; determining a second time point corresponding to the video frame where the hitting event occurs, wherein the video frame where the hitting event occurs is a video frame containing the moment when the hitter swings the racket and hits the ball; if the first time point and the second time point are the same, and the visual features of the video frame meet the preset hitting rules, then the candidate hitting time period is determined to be a real hitting time period.

[0076] Optionally, the target video is edited according to the editing time points to obtain the video clip of the hitting round, including: determining the first real hitting time period corresponding to the first hitting event of the hitting round and the second real hitting time period corresponding to the last hitting event of the hitting round; determining the first hitting time point within the first real hitting time period and the second hitting time point within the second real hitting time period; determining the time point with a preset duration before the first hitting time point as the start editing time point and the time point with a preset duration after the second hitting time point as the end editing time point; and editing the target video according to the start editing time point and the end editing time point to obtain the video clip of the hitting round.

[0077] It should be noted that each module in the above video editing device can be a program module (e.g., a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0078] It should be noted that the video editing device provided in this embodiment can be used to perform... Figure 2 The video editing method shown above is also applicable to the embodiments of this application, and will not be repeated here.

[0079] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following video editing method by running the computer program: acquiring an audio data stream of a target video and dividing the audio data stream into multiple audio segments; determining the classification result corresponding to each audio segment, and if the classification result indicates that the audio segment contains a hitting sound, determining the time period corresponding to the audio segment as a candidate hitting time period; acquiring video frames in the target video corresponding to the candidate hitting time periods, and determining that the candidate hitting time period is a real hitting time period if the visual features of the video frames match preset hitting rules, wherein the preset hitting rules are used to characterize the spatial positional relationship that the visual features should satisfy when performing a hitting action; determining the editing time point based on the real hitting time period, and editing the target video according to the editing time point to obtain a video segment of the hitting round.

[0080] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video editing method described in various embodiments of this application: acquiring an audio data stream of a target video and dividing the audio data stream into multiple audio segments; determining the classification result corresponding to the audio segments, and if the classification result indicates that the audio segment contains a hitting sound, determining the time period corresponding to the audio segment as a candidate hitting time period; acquiring video frames in the target video corresponding to the candidate hitting time periods, and if the visual features of the video frames match preset hitting rules, determining that the candidate hitting time period is a real hitting time period, wherein the preset hitting rules are used to characterize the spatial positional relationship that the visual features should satisfy when performing a hitting action; determining the editing time point based on the real hitting time period, and editing the target video according to the editing time point to obtain a video segment of the hitting round.

[0081] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0082] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0084] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0085] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0086] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0087] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A video editing method, characterized in that, include: Acquire the audio data stream of the target video and divide the audio data stream into multiple audio segments; The classification result corresponding to the audio segment is determined, and if the classification result indicates that the audio segment contains the sound of hitting the ball, the time period corresponding to the audio segment is determined as a candidate hitting time period; The video frame corresponding to the candidate hitting time period in the target video is obtained, and if the visual features of the video frame match the preset hitting rules, the candidate hitting time period is determined to be the actual hitting time period. The preset hitting rules are used to characterize the spatial positional relationship that the visual features should satisfy when performing the hitting action. Based on the actual hitting time, the editing time point is determined, and the target video is edited according to the editing time point to obtain a video segment of the hitting round.

2. The video editing method according to claim 1, characterized in that, Dividing the audio data stream into multiple audio segments includes: The audio data stream is preprocessed, wherein the preprocessing operation includes at least one of the following: noise reduction processing and normalization processing; Determine the window length of the time window and the corresponding movement step size, wherein the movement step size is not greater than the window length; Starting from the beginning of the audio data stream, the audio data stream is segmented by gradually moving the time window according to the moving step size to obtain multiple audio segments.

3. The video editing method according to claim 1, characterized in that, The classification result corresponding to the audio segment includes: Obtain the audio features corresponding to the audio segment, wherein the audio features include at least one of the following: Mel frequency cepstral coefficient features, spectrogram features; A sound classification model is used to analyze the audio features and obtain the classification results corresponding to the audio features. The sound classification model is trained based on a target training set, which includes multiple audio segment samples and classification result labels corresponding to each audio segment sample. The classification results include: near ball hitting sound, far ball hitting sound, ball hitting sound from other venues, and others.

4. The video editing method according to claim 1, characterized in that, After acquiring the video frame corresponding to the candidate shot time period in the target video, the method further includes: A target detection model is used to extract the first image features of the video frame, and the position of the key target object in the video frame is determined based on the first image features. The key target object includes: the person hitting the ball and the racket. A ball tracking model is used to extract the second image features of the video frame, and the position and trajectory of the ball in the video frame are determined based on the second image features. The target detection model has a smaller model structure and lower computational complexity than the ball tracking model.

5. The video editing method according to claim 4, characterized in that, The visual features of the video frame include: the position of the key target object in the video frame, the position and trajectory of the ball in the video frame; the preset hitting rules include at least one of the following: In the video frame, the batter and racket appeared simultaneously, and the batter's body movements matched the characteristics of a racket swing. In the video frame, a ball is detected, and the position of the ball is within the hitting range of the racket; The key target object is located within the area corresponding to the target site.

6. The video editing method according to claim 4, characterized in that, Determining whether the candidate hitting time period is the actual hitting time period also includes: Determine the first time point corresponding to the sound of the ball hitting in the audio segment corresponding to the candidate ball hitting time period; Determine the second time point corresponding to the video frame where the ball-hitting event occurs, wherein the video frame where the ball-hitting event occurs is a video frame containing the moment when the hitter swings the racket and hits the ball; If the first time point and the second time point are the same, and the visual features of the video frame meet the preset hitting rules, the candidate hitting time period is determined to be the actual hitting time period.

7. The video editing method according to claim 1, characterized in that, Editing the target video according to the specified editing time points yields video clips of the hitting round, including: Determine the first true hitting period corresponding to the first hitting event in the hitting round and the second true hitting period corresponding to the last hitting event in the hitting round; The first hitting time point within the first real hitting time period and the second hitting time point within the second real hitting time period are determined. The time point that is a preset duration before the first hitting time point is determined as the start editing time point, and the time point that is a preset duration after the second hitting time point is determined as the end editing time point; Based on the start and end editing times, the target video is edited to obtain the video segment of the ball-hitting round.

8. A video editing device, characterized in that, include: An audio segmentation module is used to acquire the audio data stream of the target video and segment the audio data stream into multiple audio segments; An audio classification module is used to determine the classification result corresponding to the audio segment, and when the classification result indicates that the audio segment contains the sound of hitting the ball, the time period corresponding to the audio segment is determined as a candidate hitting time period; A visual assistance module is used to acquire video frames in the target video corresponding to the candidate hitting time period, and determine the candidate hitting time period as the actual hitting time period when the visual features of the video frame match the preset hitting rules. The preset hitting rules are used to characterize the spatial positional relationship that the visual features should satisfy when performing the hitting action. The round editing module is used to determine the editing time point based on the actual hitting time period, and to edit the target video according to the editing time point to obtain a hitting round video segment.

9. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the video editing method according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the video editing method according to any one of claims 1 to 7 by running the computer program.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the video editing method according to any one of claims 1 to 7.

Images