A method of automatically editing a video of interest, an editing device and a computing device
By synchronously recording potentially interesting frames during video recording and combining this with cloud-based verification, the problems of low recognition accuracy and privacy protection in automatic video editing are solved, achieving efficient and privacy-secure video editing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU DEEPSIGHT TECH CO LTD
- Filing Date
- 2026-04-30
- Publication Date
- 2026-07-10
AI Technical Summary
Existing technologies for automatically editing videos suffer from low recognition accuracy, high computational resource consumption, and difficulty in protecting user privacy.
During the recording process, potentially interesting frames are recorded synchronously using a video recording device, forming a set of potentially interesting frames. These frames are then initially screened locally and reviewed using a large cloud model to generate videos of interest.
It improved recognition accuracy, reduced data processing volume, lowered computing resource consumption, and protected user privacy.
Smart Images
Figure CN122372811A_ABST
Abstract
Description
Technical Field
[0001] Embodiments of this application relate to the field of video analysis and editing, and more particularly to a method, editing apparatus, and computing device for automatically editing videos of interest. Background Technology
[0002] With the development of video analytics and artificial intelligence algorithms, videos recorded in scenarios such as sports events, performances, and security monitoring are increasingly relying on intelligent automatic editing technology to improve video processing efficiency. How to improve the accuracy of identifying the desired segments during automatic editing, ensuring that the final edited video fully contains the required segments while reducing misidentified segments, improving processing timeliness, and protecting user privacy and security are common technical challenges currently faced by the industry.
[0003] Existing video clip editing solutions are either entirely based on offline models performed locally, including using offline models to analyze and identify clips without further verification and proofreading using large cloud-based models. Compared to large cloud-based models, local offline models have weaker computing power and slower knowledge updates, resulting in a higher probability of misidentification and lower recognition accuracy and output precision.
[0004] Furthermore, existing technologies may involve uploading the entire video to the cloud for processing, analyzing it using large-scale cloud models, identifying and editing key segments. This method requires processing massive amounts of data, consuming enormous amounts of computing power and is extremely time-consuming, resulting in low video processing efficiency. Additionally, transmitting video data over the network and relying on external services makes it difficult to protect user privacy. Summary of the Invention
[0005] In view of this, this application provides a solution for automatically editing videos of interest, which aims to improve the recognition accuracy of intelligent editing, reduce the amount of data processing in the system, and simultaneously achieve user personal information protection and privacy compliance.
[0006] In a first aspect of this application, a method for automatically editing a video of interest is provided, the method comprising: Acquire a set of multiple potentially interesting frames recorded synchronously by the video recording device while recording the original video; Based on the multiple sets of potentially interesting frame records, multiple potentially interesting video segments are obtained; Each of the potentially interesting video segments is reviewed to identify one or more video segments of interest. The one or more video clips of interest are spliced together to form a video of interest.
[0007] In this process, the video recording device simultaneously judges each frame recorded in real time while recording the original video, identifies each potential frame of interest, and aggregates all potential frames of interest belonging to the same event of interest to form the potential frame of interest record set.
[0008] The process of splicing one or more video segments of interest into a video of interest includes: Based on a preset video template, one or more video clips of interest are spliced together to form a video of interest.
[0009] The review of each potentially interesting video segment includes: The potentially interesting video clips are uploaded to a large cloud model for review.
[0010] The review of each potentially interesting video segment includes: The frames in the potentially interesting frame record set are uploaded to the cloud-based large model for verification.
[0011] Among them, a potential frame of interest is a video frame in which a preset target exists within the region of interest.
[0012] Among these steps, while simultaneously judging each frame of the real-time recording, the current state is determined, including a marked state and an unmarked state. The aggregation of all potentially interesting frames belonging to the same event of interest to form the potentially interesting frame record set includes: When continuously adding records of the frames of interest to the set, if an unmarked state is detected, the set is closed to obtain a set of records of potential frames of interest belonging to the same event of interest.
[0013] The potentially interesting frames (PIFFs) record set records the timestamps or frame numbers of the PIFFs. Based on each PIFF record set, multiple potentially interesting video segments are obtained, including: Based on the timestamps or frame numbers of the first and last frames in each set of potential interest frames, corresponding video segments are extracted from the original video as the potential interest video segments corresponding to the set of potential interest frames.
[0014] In a second aspect of this application, an editing apparatus is provided, comprising a trimming module, a judging module, and a splicing module, wherein: The interception module is adapted to acquire a set of multiple potentially interesting frames recorded synchronously by the video recording device when recording the original video, and to obtain multiple potentially interesting video segments based on each set of potentially interesting frames. The judgment module is adapted to review each of the potentially interesting video segments and determine one or more video segments of interest; The splicing module is adapted to splice one or more video segments of interest into a video of interest.
[0015] In a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, characterized in that the processor executes the computer program to implement the method of the first aspect of this application.
[0016] The method for automatically editing videos of interest provided in this application can obtain potentially interesting video segments based on a set of potentially interesting frames recorded during video recording, and verify them based on relevant data of the potentially interesting video segments, generating videos of interest based on the verification results. In this method, since the set of potentially interesting frames is obtained by the system based on the interested frame judgment mechanism, it can filter out non-interesting frames to a certain extent. Therefore, the total number of frame records in the entire potentially interesting frame record set is relatively small, and the number of potentially interesting video segments obtained from it will also be correspondingly reduced, achieving simplification of subsequent verification data. Furthermore, in some embodiments of this application, the verification results of the potentially interesting video segments are returned through verification, thereby filtering the potentially interesting video segments. Compared to the prior art that directly outputs video segments, the output data volume of this verification result is also greatly reduced. In summary, this method can improve recognition accuracy and ensure output quality while reducing the system's data processing volume, reducing computing power consumption, and improving video editing efficiency. Moreover, the verification mechanism of this system allows for privacy processing of the verification data, maximizing the protection of user personal information and privacy compliance, resulting in significant technical effects. Attached Figure Description
[0017] Figure 1 A flowchart illustrating a method for automatically editing a video of interest according to an embodiment of this application; Figure 2 A flowchart illustrating a method for obtaining a set of potentially interesting frame records according to an embodiment of this application; Figure 3 A flowchart of another method for obtaining a set of potentially interesting frame records according to an embodiment of this application; Figure 4 A block diagram of an editing device according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a computer device suitable for implementing the embodiments of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] In the field of computer vision analysis, the origin of the coordinate system is usually located at the upper left corner of the screen by default. In all embodiments of this application, unless otherwise stated, the upper left corner is used as the origin of the screen coordinate system. Those skilled in the art should know that such a coordinate system setting is not absolutely fixed. When the origin of the coordinate system is set at any position inside or outside the screen, the corresponding technical solutions that can be obtained by simple adjustments to this solution without creative effort are all within the protection scope of this application.
[0020] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.
[0021] The method for automatically editing videos of interest provided in this application can run on various computing devices, including laptops, smartphones, tablets, or other smart devices with data acquisition capabilities, such as smart cameras and wearable devices.
[0022] According to one embodiment of this application, a method for automatically editing videos of interest is provided. The purpose is to simultaneously filter and classify video frames during video recording to obtain corresponding potentially interesting video segments, and then splice these segments together to obtain the video of interest. In this application, the video of interest is formed by splicing video segments from the original video that meet preset rules, and is a specific video edited based on actual business scenario requirements.
[0023] Figure 1 A flowchart illustrating a method for automatically editing a video of interest according to an embodiment of this application is shown, such as... Figure 1 As shown: S1: Obtain a set of multiple potentially interesting frames recorded synchronously by the video recording device while recording the original video.
[0024] In one possible implementation, the present invention first uses a video recording device to simultaneously record a set of multiple potentially interesting frames while recording the original video. The video recording device is a device for acquiring and storing video data, and can be a camera, smartphone, computer, or other device with data acquisition capabilities.
[0025] In one possible implementation, the object detection algorithm is used to identify and locate the position information of objects from real-time video frames. This application does not limit the specific choice of object detection algorithm: the YOLOv8 (You Only Look Once version 8) model can be used, as well as recognition algorithms such as CNN (Convolutional Neural Network) and ViT (Vision Transformer). For each detected object, the object detection algorithm provides a bounding box. Typically, the bounding box is represented by four parameters (x, y, width, height), where x and y represent the horizontal and vertical coordinates of the center point of the bounding box, respectively; width represents the width of the bounding box; and height represents the height of the bounding box. The object detection algorithm detects each real-time video frame and outputs a log of all bounding box parameters in the current real-time video frame. Each real-time video frame has a corresponding bounding box detection log. An example of a bounding box detection log is as follows: [{"time": 1750161291.6964421, "currentYawAngle": 0.0, "currentPitchAngle": 12.0, "yolo": [{"label": 0, "confidence": 0.96, "x":286.0, "y": 869.0, "width": 78.0, "height": 190.0}, {"label": 0, "confidence": 0.96, "x": 1824.0, "y": 836.0, "width": 72.0, "height": 176.0}, {"label": 0, "confidence": 0.95, "x": 1358.0, "y": 736.0, "width": 60.0, "height": 143.0}, {"label": 0, "confidence": 0.94, "x": 1402.0, "y": 670.0, "width": 50.0, "height": 101.0}, {"label": 0, "confidence": 0.87, "x": 668.0, "y": 681.0, "width": 42.0, "height": 108.0}, {"label": 0, "confidence": 0.92,"x": 1210.0, "y": 662.0, "width": 28.0, "height": 92.0}, {"label": 0, "confidence": 0.91, "x": 106.0, "y": 711.0, "width": 26.0, "height": 84.0}]} In the above example, the "time" field represents the timestamp of the video frame corresponding to this log segment; the "currentYawAngle" field represents the pitch angle of the gimbal in the current frame; and the "currentPitchAngle" field represents the yaw angle of the gimbal in the current frame. Following this is the detailed information for the seven bounding boxes. The "label" field is the bounding box label, typically used to indicate the target type, such as 0 for a human target, 1 for a sphere target, and 2 for a scoring region target; the "confidence" field represents the confidence level, i.e., the degree of confidence the model has in the detection results; and the four parameters "x", "y", "width", and "height" define the position and size of the bounding boxes.
[0026] Preferably, when recording the original video, the video recording device simultaneously judges each frame recorded in real time and identifies each potentially interesting frame.
[0027] Specifically, the video recording device simultaneously captures each frame in real-time while recording the original video. Within each frame recorded in real-time, the potentially interesting frame (LOP) refers to a real-time video frame that meets preset rules; these are specific frames extracted based on the needs of the actual business scenario. Using relatively simple preset rules enables simultaneous filtering of real-time video frames during recording, saving computational resources and improving efficiency.
[0028] Based on different preset rules, there are different ways to identify latent interest frames (LOFFes). In one possible implementation, the video recording device can accept manual marking by the user to record LFFs. When the user considers the current scene to be of interest during recording, they can manually mark the point, and the corresponding video frame will be recorded as a LFF. Alternatively, it can automatically identify the target using a locally preset target recognition algorithm, performing real-time detection on each frame to determine if a preset target of interest exists. If it does, it is automatically recorded as a LFF, completing synchronous marking without manual user intervention. Another approach is to use the multimodal acquisition unit of the video recording device, such as the recording unit, to collect and analyze the sound of the recording site. After marking LFFs, all consecutively marked LFFs belonging to the same interest event within the same time period are aggregated into a LFF record set, with each set corresponding to an independent LFF event.
[0029] In one possible implementation, a potentially interesting frame (PIF) is a video frame containing a predefined target. Specifically, the meaning of the predefined target varies depending on the recording scenario and requirements: in a ball game scenario, for the purpose of editing highlights, the predefined target could be a reference object such as the ball or goal; for the purpose of tactical analysis, it could be a specific player; in a large-scale cultural and sports event scenario, for the purpose of security monitoring, the predefined target could be a human body in an abnormal posture; in a road monitoring scenario, for the purpose of traffic safety monitoring, the predefined target could be a road obstacle. To determine whether a real-time video frame is a PIF, one possible implementation involves first using the target detection algorithm described above to identify and locate the position information of all targets in the real-time video frame, i.e., obtaining a target bounding box detection log for all targets in the real-time video frame based on the target detection algorithm. Secondly, based on the meaning of the predefined target, the real-time video frame is filtered by referring to the target bounding box labels in the target bounding box log that indicate the target type: if there is an entry in the target detection log that matches the predefined target type with the target bounding box label, then the current real-time video frame is a PIF; otherwise, it is not.
[0030] In another possible implementation, a potential frame of interest is a video frame in which a predetermined target exists within a region of interest.
[0031] Specifically, in a ball game scenario, the region of interest (ROI) is the scoring area and a certain area extending outwards, with the ball as the preset target. In a classroom recording scenario, the ROI is the blackboard and a certain area extending outwards, with students as the preset target. First, the aforementioned target detection algorithm is used to identify and locate the position information of all targets in the real-time video frames, resulting in a target bounding box detection log. Second, based on the target bounding box labels in the target bounding box detection log entries, target bounding boxes with labels matching the preset target and the ROI target are selected. For example, if the target bounding box labels for the preset target and the ROI are 1 and 2 respectively, entries with labels 1 and 2 are selected from the target bounding box detection log, and the corresponding target bounding box position parameters (x1, y1, width1, height1) and (x2, y2, width2, height2) are obtained. The target bounding box corresponding to (x1, y1, width1, height1) is the preset target. Third, preset scaling factors k1 and k2 are applied to the width2 and height2 parameters of the ROI target bounding box, such as k1=1.5 and k2=2, to obtain the expanded target bounding box (x2, y2, k1...). The target bounding box (x1, y1, width1, height1) is the region of interest. The scaling factors k1 and k2 can be dynamically adjusted according to different application scenarios and user needs. Finally, the target bounding box (x1, y1, width1, height1) of the preset target and the target bounding box (x2, y2, k1, height2) of the region of interest are compared. If there is an overlapping region between width2, k2, and height2, the current real-time video frame is considered a potential frame of interest (PGI); otherwise, it is not. The advantages of this method for selecting PPIs are twofold: firstly, it only requires a simple algorithm to determine the overlap of specific target boxes recorded in the target box detection log, significantly reducing computational cost; secondly, it can obtain PPIs with greater semantic relevance to the scene based on the spatial relationship between the region of interest and the preset target.
[0032] Figure 2 A flowchart illustrating a method for obtaining a set of potentially interesting frame records according to an embodiment of this application is shown, such as... Figure 2As shown, the formation process of the potential interest frame record set includes a state determination mechanism. This mechanism determines the current state as records of the interest frames are continuously added to the set. The state includes a marked state and an unmarked state. If an unmarked state is detected, the set is closed, resulting in a single potential interest frame record set belonging to the same interest event.
[0033] Specifically, in one possible implementation, based on the determination of potential frames of interest in the aforementioned embodiments of this application, if the real-time video frame is a potential frame of interest, the current state of the state machine is obtained. The state includes a marked state and an unmarked state. A marked state refers to the state in which the state machine is recording potential frames of interest, and in this state, there exists a set of recorded potential frames of interest. An unmarked state refers to the state in which the state machine is not recording potential frames of interest, and in this state, all sets of recorded potential frames of interest are closed, and there is no set of recorded potential frames of interest.
[0034] If the state is unmarked, the current state of the state machine is switched to the marked state, and the real-time video frame is recorded. Since there is no valid set of potential frames of interest (OPIs) records, a new set of OPIs is created, and the frame is saved to the newly created set of OPIs. If the state is marked, the real-time video frame is recorded. Since there is a valid set of OPIs, the frame is saved to the set of OPIs. At the same time, the distance between the current real-time video frame and the previously recorded real-time video frame is calculated. If the distance exceeds a first threshold, the state is switched to the unmarked state.
[0035] Recording real-time video frames refers to recording potentially interesting frames (PIs), including recording their timestamps, frame numbers, storage offsets, and other specific identifiers of a particular video frame. The term "set" is a broad concept; a set of PI records refers to any data carrier capable of carrying, storing, or associating multiple PI records, including but not limited to record pools, record clusters, record sequences, record queues, and linked lists. A valid set of PI records refers to a set of PI records in a recording state, which can have new PI records added to it, excluding sets that are closed and cannot have new records added to. "Saving" refers to any operation that can establish an association between a PI record and a set of PI records, including but not limited to writing, storing, moving, copying, mapping, and indexing.
[0036] The method determines whether to switch the marking state based on the distance between the currently recorded real-time video frame and the previously recorded real-time video frame. The significance of this method is that if the distance difference between the currently recorded and previously recorded real-time video frames is too large, there is a high probability that the event of interest, consisting of the previously recorded real-time video frame and all previous real-time video frames that meet the distance condition, has already ended. For example, after a shot is taken and the ball is kicked off from the center circle, as the target ball leaves the area of interest (goal and surrounding area), the time span or frame interval of the frames of interest may gradually increase until it exceeds a preset distance threshold. When a frame exceeds this distance threshold, the current event of interest ends, and the marking state switches to the unmarked state. Setting the distance threshold based on the distance between potential frames of interest allows for the analysis of real-time video frames and the implementation of state switching with a relatively simple algorithm, significantly reducing computational power consumption.
[0037] Specifically, the distance can be a timestamp difference or a frame sequence number difference. In one possible implementation, the timestamp metadata T of the current real-time video frame is obtained. n The timestamp metadata T of the last recorded real-time video frame n-1 Calculate the difference ΔT between the two values. When ΔT is greater than a first threshold, such as 2 seconds, switch the state to an unmarked state. In another possible implementation, obtain the frame sequence number metadata I of the current real-time video frame. n Frame sequence number metadata I compared to the last recorded real-time video frame n-1 The difference ΔI between the two is calculated. When ΔI is greater than a first threshold, the state is switched to an unmarked state. For example, if the video frame rate during recording is 30fps and the first threshold is set to 60 frames, when ΔI > 60 frames, the state is switched to an unmarked state. Those skilled in the art should understand that the specific value of the first threshold in this application can be flexibly adjusted according to the different requirements for target capture sensitivity and recording redundancy in the actual application scenario. For example, if stronger fault tolerance, higher semantic integrity, and higher recording redundancy are desired, a larger first threshold can be selected, such as 3s; in the case of a video frame rate of 30fps, this is 90 frames.
[0038] If an unmarked state is detected, the set is closed to obtain a set of potential interest frame records belonging to the same interest event.
[0039] Specifically, if the detected state is unmarked, based on the aforementioned implementation methods in this application, it indicates that the distance between the potential frame of interest corresponding to the current potential frame of interest record and the potential frame of interest corresponding to the previous potential frame of interest record is greater than a first threshold, and the current interest event ends. Closing the current set of potential frame of interest records refers to the operation of confirming the complete state of the current set of potential frame of interest records, including but not limited to terminating recording, demarcating boundaries, marking the current frame record, and terminating record association. Therefore, by performing the operation of closing the current set of potential frame of interest records, a set of potential frame of interest records belonging to the same interest event can be obtained.
[0040] After performing the operation of saving a potentially interesting frame record to a set of potentially interesting frame records, or performing a closing operation on a set of potentially interesting frame records based on its state, if the recording of the real-time video stream has not ended, the aforementioned saving operation for the frame record and the creation and closing operation for the set of potentially interesting frame records are repeated for the frame record corresponding to the next potentially interesting frame until the recording of the real-time video stream ends, thus obtaining multiple sets of potentially interesting frame records. These sets store frame records of multiple events of interest and are divided based on the inter-frame distance, which greatly saves computing power and enables simultaneous analysis and processing of video frames during video recording.
[0041] Figure 3 A flowchart of another method for obtaining a set of potentially interesting frame records according to an embodiment of this application is shown. In this embodiment, if an unmarked state is detected, the set is closed to obtain a set of potentially interesting frame records belonging to the same interest event. The method further includes calculating the number of frame records in the set of potentially interesting frame records; if the number of frame records is less than a second threshold, the set of potentially interesting frame records is deleted.
[0042] At typical recording frame rates, such as 30fps or higher, the number of video frames constituting an event of interest is relatively high, resulting in a large number of potential interest frame records. If the number of frame records in a potential interest frame record set is less than a preset second threshold, it indicates that the number of video frames constituting the event is low. The events corresponding to the frames recorded in this set are not events of interest or events of interest with high completeness, and are very likely false alarms or noise from the system. Since the set does not record the potential interest frames required by this application and has low utilization value, it is deleted from all potential interest frame record sets. Deletion refers to the operation of terminating the existence of the set, including but not limited to deleting records, unassociating, releasing memory, and marking the set as unusable. For example, in a football match scenario, a football appears near the goal from outside the field and is then cleared. The system identifies the frames within this time period as potential interest frames and records them, obtaining a potential interest frame record set. After statistics, the number of frame records in this set is F. count The frame rate is 85 frames. The second threshold is set to 90 frames, because F... count If the value is less than 90, delete the set of potentially interesting frame records.
[0043] Those skilled in the art should understand that the specific value of the second threshold in this application can be flexibly adjusted according to the different requirements of recording redundancy and recording frame rate in the actual application scenario. For example, if the recording redundancy is high or the recording frame rate is high, a larger second threshold, such as 105 frames, can be selected; if the recording redundancy is low or the recording frame rate is low, a smaller second threshold, such as 72 frames, can be selected.
[0044] In another possible embodiment, the process of aggregating all potential frames of interest belonging to the same event of interest to form a potential frame of interest record set is as follows: First, for each potential frame of interest in the sequence, the distance between it and the previous potential frame of interest is calculated, and this distance is used as the logical basis for judging the continuity between frames. Then, differentiated processing logic is executed according to the quantization relationship between the distance and a preset third threshold: When the distance is less than the third threshold, if a valid potential frame of interest record set has not yet been established, the system performs an initialization operation, creating a new potential frame of interest record set and storing the previous potential frame of interest record and the current potential frame of interest record together in the set to complete the initial construction of the set. If a valid potential frame of interest record set already exists, the current potential frame of interest record is directly appended and saved to the set to achieve continuous aggregation. Conversely, if the distance is greater than or equal to the third threshold, it indicates that there is a break in the current frame and the preceding sequence in terms of spatiotemporal or logical characteristics. In this case, if there is no valid potential frame of interest record set, the potential frame of interest record is determined to be isolated noise or invalid data. If a valid set of potential frames of interest (OPIs) already exists, after saving the current OPIs as the last frame to the set, a closing operation is immediately performed on the OPIs set to complete the encapsulation of the OPIs set.
[0045] In another possible embodiment, the action of aggregating all potential frames of interest belonging to the same event of interest to form a potential frame of interest record set can be performed after recording is completed. After recording is completed, a clustering algorithm is run based on all the obtained potential frames of interest to cluster spatiotemporally adjacent potential frames of interest, aggregating potential frames of interest that meet the clustering conditions into the same potential frame of interest record set. This method does not require maintaining a state machine during recording, consumes fewer resources, and is suitable for offline scenarios.
[0046] S2: Based on the set of potentially interesting frame records, obtain multiple potentially interesting video segments.
[0047] Preferably, the latent interest frame record set records either a timestamp or a frame number for the latent interest frame. The timestamp is a specific identifier used to characterize the temporal information of a particular interest frame on the recording timeline, and the frame number is a specific identifier used to characterize the physical arrangement or logical position of a particular interest frame in the recorded video sequence. Both are ordered and unique identification methods for video frames, allowing for a unique correspondence between specific latent interest frames based on this record.
[0048] Specifically, structured metadata is generated synchronously during video recording. This structured metadata includes the timestamp or frame number of the real-time video frame. When a real-time video frame is identified as a potentially interesting frame, its timestamp or frame number is recorded as a retrieval index or unique identifier. For example, in a football match scenario, during the recording of the match, the corresponding timestamp or frame number metadata is generated synchronously for each real-time video frame. When a real-time video frame is identified as a potentially interesting frame by the system, such as when a ball is near the goal, the system synchronously extracts the currently generated timestamp metadata or frame number metadata for that frame, such as 20260417_153025 or frame number 15200, and records it. Using uniquely identified timestamp metadata or frame number metadata to identify a specific video frame establishes a precise correspondence between metadata and video frames, facilitating subsequent operations on the video frame based on the metadata and improving the efficiency of frame-level operations.
[0049] In one possible implementation, the process of obtaining multiple potentially interesting video segments based on the multiple sets of potentially interesting frame records includes: Based on the timestamps or frame numbers of the first and last frames in each set of potential frames of interest (OPIs), corresponding video segments are extracted from the original video as the potential video segments corresponding to the set of OPIs. Since all frames within the same set of OPIs belong to the same event of interest, with the first frame corresponding to the start time of the event and the last frame corresponding to the end time, directly extracting the original video segment between the first and last frames completely preserves the entire video content of the event of interest. This simplifies the segment extraction process and reduces resource consumption during processing, as it eliminates the need to extract each frame individually.
[0050] Considering the potential for false alarms when recording potentially interesting frames (PIs), in another possible implementation, preferably, a clustering algorithm is performed on each set of PIs to determine whether it could potentially consist of two or more segments of interest. If cluster centers for two or more segments of interest do exist, the original set of PIs is further divided into multiple new sets of PIs based on these cluster centers. Then, the corresponding potentially interesting video segments are extracted based on the timestamps or frame numbers of the first and last frames of each of the split sets, further improving the accuracy of segment division and avoiding the erroneous merging of multiple consecutive events of interest into a single segment.
[0051] S3: Review each of the potential video segments of interest to identify one or more video segments of interest.
[0052] Verification refers to the operation of performing secondary verification on potentially interesting segments and outputting the verification results. Through secondary verification, verification can significantly reduce the frequency of non-interest-of events in the interesting video segments, improving output accuracy. Simultaneously, the verification output refers to any identifying data that can distinguish between frames of interest and non-interest-of frames, or interesting video segments and non-interest-of segments, including but not limited to binary bits, Boolean values, and preset identification tags. The significance of this method is that, compared to directly outputting the verified interesting video frames or segments, outputting identifying data greatly reduces the amount of output data for verification results, saving computational resources and improving verification processing efficiency.
[0053] In one possible embodiment, the review can be conducted manually to identify one or more video segments of interest. Specifically, when the reviewer selects a potentially interesting video segment, the video player automatically jumps to that segment and intelligently plays it backward from the start frame and forward from the end frame by a certain number of frames, ensuring that the reviewer can see the complete context, such as a run-up before a goal or a celebration after a goal. Based on this, the reviewer can use drag-and-drop operations on the timeline to fine-tune the start and end boundaries of the interesting video segments by adjusting them frame by frame or at 10-frame intervals. The reviewer can also confirm the interesting video segments based on the content of the video and delete falsely reported non-interesting video segments, thus realizing a review process from initial algorithmic screening to refined manual confirmation.
[0054] In another possible implementation, the verification can be performed on each potentially interesting video segment using background noise recognition technology. Specifically, interesting video segments are often accompanied by audience cheers, applause, or sudden changes in the volume and speed of the commentator. Therefore, abnormal peaks in these acoustic features can be captured using audio energy detection, spectral analysis, or speech emotion recognition. For each potentially interesting video segment, when a significant energy boost or concentrated burst of a specific frequency band is detected in the audio signal within a short period of time, it usually means that an interesting event has occurred that elicits a strong reaction from the audience, and the potentially interesting video segment can then be identified as an interesting video segment.
[0055] In another possible embodiment, the review process includes uploading the potentially interesting video clips or frames from the potentially interesting frame record set to a cloud-based large model for review.
[0056] In this implementation, frames from the set of potentially interesting frames (PIFFs) are uploaded to a cloud-based large-scale model for verification. In this method, the frames in the PIFF set refer to the frames corresponding to each PIFF record. Since the records in the PIFF set are not the PIFFs themselves, but rather identification identifiers such as frame numbers and timestamps, these identifiers cannot be verified by the cloud-based large-scale model. Therefore, it is necessary to match the corresponding PIFFs based on the records in the PIFF set and upload these video frames to the cloud-based large-scale model for verification. Furthermore, compared to uploading the entire video segment, uploading only the frames from the PIFF set to the cloud-based large-scale model for verification saves computational resources and improves the processing efficiency of the large-scale model.
[0057] Given that current cloud-based large-scale model-based multimodal processing technology has matured, in the process of uploading potentially interesting video clips or potentially interesting video frames to the cloud-based large-scale model for analysis and processing in this application, the large-scale model is regarded as a mature black box, and this technology is not a technical feature to be protected in this application. Therefore, only its calling logic is described here, and its internal algorithm is not described in detail.
[0058] S4: Combine one or more video segments of interest into a video of interest.
[0059] Preferably, the splicing process includes splicing one or more video segments of interest into a single video of interest based on a preset video template. The preset video template can standardize the arrangement order, transition effects, and segment duration of the spliced video. For example, by default, the video segments of interest are arranged in the original recording time order, with fade-in and fade-out transitions added between segments. It can also be adjusted to other arrangement logics and transition effects as needed to adapt to different usage scenarios. After splicing, a completed video containing only the content of interest is obtained, eliminating the need for manual frame-by-frame analysis of the original long video. This significantly reduces the time and labor costs of editing videos of interest, achieving fully automated processing throughout the entire editing process.
[0060] Preferably, transition effects can be dynamically added based on distance or video rhythm. For example, in a football match goal highlight reel, if the time interval between multiple celebration frames triggered by the same goal is relatively short, they can be smoothly spliced together. However, if the time interval between two interesting video clips corresponding to different goals is far and the content rhythm difference is large, a fast-paced transition effect can be used to better match the rhythm of the goal highlight reel and improve the viewing experience. If the content rhythm difference between adjacent exciting clips within the same segment is small, a fade-in / fade-out gradual blending transition can be used to maintain the smoothness of the narrative. If the content rhythm difference between exciting clips in different thematic segments is large, a fade-in transition effect can be used to divide the content layers, making the logic of the edited product clearer.
[0061] In this solution, only the verification step must be performed in the cloud; all other steps can be done locally. Compared to full video processing solutions, this approach maximizes the protection of user personal information and privacy compliance. Traditional full video processing solutions transmit the entire video to a large model, which then provides a processed, edited video. With this approach, if facial information is blurred before uploading to meet privacy compliance requirements, the facial information in the edited video returned by the large model will also be blurred, resulting in extremely poor usability. In this solution, by blurring all facial information in the video or video frames uploaded to the cloud, all cloud-uploaded data is anonymized. Then, the cloud provides verification results for each potentially interesting video segment, and the locally processed videos are stitched together. Therefore, this solution can generate unanonymized edited videos based on the user's local raw data, provided that anonymized data is uploaded.
[0062] Based on the same inventive concept described above, this application also provides an editing device 100. Figure 4 This is a block diagram of an editing device 100 according to an embodiment of this application. Figure 4 As shown, the editing device 100 includes a cropping module 101, a judgment module 102, and a splicing module 103. This editing device 100 can be used to implement the method for automatically editing videos of interest as described in the foregoing embodiments.
[0063] The extraction module 101 is used to obtain multiple potentially interesting video segments based on the set of potentially interesting frame records; the judgment module 102 is used to review each potentially interesting video segment to determine one or more interesting video segments; the splicing module 103 is used to splice the determined one or more interesting video segments into the final interesting video. These modules work together to complete most of the processing locally, requiring only necessary content to be anonymized and uploaded to the cloud for review. This ensures editing accuracy while protecting privacy and reducing overall computing power consumption.
[0064] Figure 5 A schematic diagram of a computer device suitable for implementing embodiments of this application is shown. The computer device can be implemented as a terminal device or a server.
[0065] like Figure 5As shown, the terminal device or server includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage section 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the terminal device or server. The CPU 501, ROM 502, and RAM 503 are interconnected via bus 504. An input / output (I / O) interface 505 is also connected to bus 504.
[0066] The following components are connected to I / O interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to I / O interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 510 as needed so that computer programs read from it can be installed into storage section 508 as needed.
[0067] Specifically, according to embodiments of this application, the above method flow steps can be implemented as a computer software program. For example, embodiments of this application include a computer program product comprising a computer program carried on a machine-readable medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by central processing unit (CPU) 501, it performs the functions defined in the system of this application.
[0068] It should be noted that the computer-readable medium shown in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, or any suitable combination thereof.
[0069] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0070] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not, in some cases, constitute a limitation on the unit or module itself.
[0071] In another aspect, this application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable storage medium stores one or more programs that are used by one or more processors to execute the methods described in this application.
[0072] In another aspect, embodiments of this application also provide a computer program product that, when executed by a processor, implements the methods of any of the above embodiments.
[0073] The above description is merely a preferred embodiment of this application and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this application is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing application concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions claimed in this application.
Claims
1. A method for automatically editing videos of interest, characterized in that, include: Acquire a set of multiple potentially interesting frames recorded synchronously by the video recording device while recording the original video; Based on the multiple sets of potentially interesting frames, multiple potentially interesting video segments are obtained; Each of the potentially interesting video segments is reviewed to identify one or more video segments of interest. The one or more video clips of interest are spliced together to form a video of interest.
2. The method as described in claim 1, characterized in that, When recording the original video, the video recording device simultaneously judges each frame recorded in real time, identifies each potential frame of interest, and aggregates all potential frames of interest belonging to the same event of interest to form the potential frame of interest record set.
3. The method as described in claim 1, characterized in that, The step of splicing the one or more video segments of interest into a video of interest includes: Based on a preset video template, one or more video clips of interest are spliced together to form a video of interest.
4. The method as described in claim 1, characterized in that, The review of each potentially interesting video segment includes: The potentially interesting video clips are uploaded to a large cloud model for review.
5. The method as described in claim 1, characterized in that, The review of each potentially interesting video segment includes: The frames in the potentially interesting frame record set are uploaded to the cloud-based large model for verification.
6. The method as described in claim 2, characterized in that, The potential frame of interest is a video frame in which a preset target exists within the region of interest.
7. The method as described in claim 2 or 6, characterized in that, While synchronously judging each frame of the real-time recording, the current state is determined, including a marked state and an unmarked state. The aggregation of all potentially interesting frames belonging to the same event of interest to form the potentially interesting frame record set includes: When continuously adding records of the frames of interest to the set, if an unmarked state is detected, the set is closed to obtain a set of records of potential frames of interest belonging to the same event of interest.
8. The method according to any one of claims 1-6, characterized in that, The potentially interesting frames (PIFFs) record set records the timestamps or frame numbers of the PIFFs. Based on each PIFF record set, multiple potentially interesting video segments are obtained, including: Based on the timestamps or frame numbers of the first and last frames in each set of potential interest frames, corresponding video segments are extracted from the original video as the potential interest video segments corresponding to the set of potential interest frames.
9. An editing device, characterized in that, It includes a truncation module, a judgment module, and a splicing module, among which: The interception module is adapted to acquire a set of multiple potentially interesting frames recorded synchronously by the video recording device when recording the original video, and to obtain multiple potentially interesting video segments based on each set of potentially interesting frames. The judgment module is adapted to review each of the potentially interesting video segments and determine one or more video segments of interest; The splicing module is adapted to splice one or more video segments of interest into a video of interest.
10. A computing device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-8.