AI camera and method, apparatus and medium for automatically generating motion highlight clip thereof

Abstract

The application provides an AI camera and a motion highlight segment automatic generation method and device thereof and a medium. The method comprises: collecting a first video stream and a second video stream through a first imaging unit and a second imaging unit respectively, and cyclically writing a long-focus video frame sequence corresponding to the second video stream into a cache unit; identifying and locking a target motion object based on a pose estimation model at the first imaging unit end; performing time sequence matching on a preset highlight action semantic library or a user-defined dynamic template library based on a spatiotemporal skeleton feature extracted by the pose estimation model, and generating a corresponding action trigger timestamp when a matching condition is met; and generating a highlight segment based on a long-focus video frame sequence corresponding to the second video stream within a first preset time window containing the action trigger timestamp. The application adopts a wide-angle monitoring and long-focus close-up binocular structure integrated with an AI algorithm, thereby improving the accuracy and efficiency of highlight segment generation.

Description

Technical Field

[0001] This application belongs to the field of video motion data processing technology, and relates to an AI camera and its automatic generation method, device and medium for motion highlight segments. Background Technology

[0002] Most current mainstream motion capture devices are limited by monocular imaging solutions. This solution has an inherent contradiction between the field of view coverage and spatial resolution at the optical level: wide-angle configuration can meet the needs of continuous tracking across the entire field, but it results in insufficient pixel density of the subject being imaged, which seriously weakens the visual performance; telephoto configuration can improve the resolution of the subject, but it is limited by the narrow field of view. When dealing with high-speed and highly random motion trajectories, it is very easy for the target to leave the field of view, resulting in tracking failure.

[0003] Furthermore, traditional action recognition architectures mostly rely on cloud-based inference, which suffers from significant end-to-end latency bottlenecks due to data transmission limitations. Existing highlight generation mechanisms largely depend on non-real-time manual post-production editing and lack shallow depth-of-field blurring capabilities at the computational photography level, making it difficult to directly produce cinematic visual content. Meanwhile, existing recognition models are mostly based on static pre-training strategies, lacking adaptive iterative capabilities for highly personalized long-tail scenarios such as yoga and skateboarding, thus limiting their generalization performance in complex application scenarios.

[0004] Therefore, improving the accuracy and efficiency of generating highlight fragments in motion has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides an AI camera and its automatic generation method, device and medium for motion highlight clips, which enables the camera to directly generate motion highlight clips without human intervention. This solves the problem that in the prior art, the acquired video data needs to be manually edited by video processing tools, which is time-consuming and the processing effect is inconsistent, failing to meet the real-time and efficient requirements of information dissemination.

[0006] In a first aspect, this application provides a method for automatically generating motion highlight clips based on an AI camera. The method includes: acquiring a first video stream under a first field of view in real time through a first imaging unit; acquiring a second video stream under a second field of view in real time through a second imaging unit, and cyclically writing the telephoto video frame sequence corresponding to the second video stream into a cache unit; performing kinematic feature analysis on the first video stream based on a pose estimation model to identify and lock a target moving object; extracting the spatiotemporal skeleton features of the target moving object based on the pose estimation model, performing temporal matching of the spatiotemporal skeleton features with a preset highlight action semantic library or a user-defined dynamic template library, and generating a corresponding action trigger timestamp when the matching conditions are met; and generating a highlight clip based on the telephoto video frame sequence corresponding to the first preset time window containing the action trigger timestamp.

[0007] In this application, the AI ​​camera includes a first imaging unit with an AI algorithm and a second imaging unit with an AI algorithm. It can directly process high-definition video streams at the local edge, eliminating the high bandwidth consumption and network transmission latency of hundreds of milliseconds or even seconds caused by uploading video data to the cloud. Through the coordinated operation of the first and second imaging units, the technical challenge of "large field of view tracking" and "high-quality close-up" can be perfectly solved, and the device is greatly endowed with highly generalized scene adaptability, which greatly improves the accuracy and efficiency of highlight fragment generation. By performing temporal matching between spatiotemporal skeleton features and a preset highlight action semantic library or a user-defined dynamic template library, it can meet the highly personalized action capture needs of users in different scenarios.

[0008] In one implementation of the first aspect, kinematic feature analysis of the first video stream based on a pose estimation model is performed to identify and lock the target moving object. This includes: using an edge computing processing unit to run the pose estimation model to perform kinematic analysis on the first video stream, extracting global and local features of the moving object in the first video stream; establishing a recognition feature library corresponding to the global and local features based on a pedestrian re-identification algorithm; and performing similarity matching on the global and local features of multiple moving or stationary objects based on the recognition feature library to identify and lock the target moving object.

[0009] In one implementation of the first aspect, generating a highlight segment based on a telephoto video frame sequence corresponding to a first preset time window containing the action trigger timestamp includes: extracting the telephoto video frame sequence corresponding to the second video stream within the first preset time window containing the action trigger timestamp; generating an initial highlight segment based on the telephoto video frame sequence; performing post-processing rendering on the initial highlight segment, wherein the post-processing rendering includes masking the target moving object in the initial highlight segment and performing digital blurring processing on its background area, and applying nonlinear time axis rescaling processing to a specific time sub-interval in the initial highlight segment to generate a multi-rate variable speed highlight segment.

[0010] In one implementation of the first aspect, applying nonlinear time-axis rescaling to a specific time sub-interval in the initial highlight segment to generate a multi-rate variable-speed highlight segment includes: dividing the telephoto video frame sequence in the initial highlight segment into an action preparation period, an action firing period, and an action ending period based on the action phase, wherein the specific time sub-interval includes the time sub-intervals corresponding to the action preparation period, the action firing period, and the action ending period; mapping the action preparation period and the action ending period in the initial highlight segment to a first playback rate; mapping the action firing period in the initial highlight segment containing the action trigger timestamp to a second playback rate lower than the first playback rate; and automatically generating a visually impactful multi-rate variable-speed highlight segment based on the first playback rate and the second playback rate.

[0011] In one implementation of the first aspect, the method further includes: automatically matching the first playback rate based on the intensity of the action during the action preparation period and the action end period; and automatically matching the second playback rate based on the intensity of the action during the action firing period.

[0012] In one implementation of the first aspect, the construction steps of the user-defined dynamic template library include: using the pose estimation model to monitor the motion of the target moving object in real time; when the target moving object is identified to make a preset interaction command gesture, automatically capturing the spatiotemporal skeletal trajectory sequence of the target moving object within a second preset time window before the interaction command gesture occurs; adding the spatiotemporal skeletal trajectory sequence as a target action template feature to the dynamic template library, so that the AI ​​camera has the ability to calibrate custom highlight actions under single sample or small sample conditions.

[0013] In one implementation of the first aspect, the method further includes: determining the positional deviation of the target moving object in the first video stream; generating a control signal in real time based on the positional deviation to adjust the horizontal and / or pitch axes of the gimbal motor, so that the target moving object remains within the effective shooting range of the binocular acquisition module composed of the first imaging unit and the second imaging unit, wherein the binocular acquisition module is rigidly fixedly mounted on the rotating platform of the gimbal motor as a load.

[0014] In one implementation of the first aspect, the method further includes: controlling the gimbal motor to rotate so that the target moving object is located in the central region of the second video stream, including: acquiring a motion coordinate sequence of the target moving object within a preset historical duration; using a trajectory prediction algorithm to calculate the predicted position of the target moving object in the next sampling period based on the motion coordinate sequence; calculating the angular velocity compensation vector of the gimbal motor rotation based on the current real-time position of the target moving object and the predicted position; matching the corresponding dynamic velocity response curve according to the angular velocity compensation vector; and driving the gimbal motor to perform predictive interpolation rotation according to the dynamic velocity response curve to achieve smooth tracking of the target moving object.

[0015] In one implementation of the first aspect, the number of AI cameras is at least one, and the multiple AI cameras trigger timestamps synchronously via a network.

[0016] In one implementation of the first aspect, the method further includes: acquiring an audio stream that is concurrent with the second video stream; performing acoustic feature analysis on the audio stream to determine the audio trigger timestamp of the highlight segment in the audio stream; determining the audio frame sequence corresponding to the audio stream within a first preset time window containing the audio trigger timestamp; and generating a highlight segment based on the telephoto video frame sequence corresponding to the second video stream within the first preset time window containing the action trigger timestamp and the audio frame sequence.

[0017] In one implementation of the first aspect, the method further includes: using an edge computing processing unit to identify non-target moving objects in the telephoto video frame sequence other than the target moving object; extracting the face region or the identifying feature region of the non-target moving object; and performing real-time anonymization masking processing on the face region or the identifying feature region during the generation of the highlight segment.

[0018] Secondly, this application provides an automatic motion highlight fragment generation device based on an AI camera. The device includes: a first video stream acquisition module, used to acquire a first video stream under a first field of view in real time through a first imaging unit; a second video stream acquisition module, used to acquire a second video stream under a second field of view in real time through a second imaging unit, and cyclically write the telephoto video frame sequence corresponding to the second video stream into a cache unit; a target moving object recognition module, used to perform kinematic feature analysis on the first video stream based on a pose estimation model, and identify and lock the target moving object; an action trigger timestamp generation module, used to extract the spatiotemporal skeleton features of the target moving object based on the pose estimation model, and perform temporal matching of the spatiotemporal skeleton features with a preset highlight action semantic library or a user-defined dynamic template library, and generate a corresponding action trigger timestamp when the matching condition is met; and a highlight fragment generation module, used to generate highlight fragments based on the telephoto video frame sequence corresponding to a first preset time window containing the action trigger timestamp.

[0019] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the automatic generation method for motion highlight fragments based on an AI camera as described in any of the first aspects of embodiments of this application.

[0020] Fourthly, embodiments of this application provide an AI camera, the AI ​​camera comprising: a gimbal motor; a binocular acquisition module, the binocular acquisition module comprising a first imaging unit and a second imaging unit; wherein, the first imaging unit is used to capture a first video stream containing a complete motion field, and the second imaging unit is used to capture a second video stream of a target moving object; the binocular acquisition module is rigidly fixedly mounted on the rotating platform of the gimbal motor as a load; an edge computing processing unit or an AI processing chip, used to implement the automatic generation method of motion highlight segments based on the AI ​​camera according to any one of the first aspects.

[0021] As described above, the AI ​​camera and its automatic generation method, apparatus, and medium for motion highlight segments described in this application have the following beneficial effects:

[0022] 1) In this application, the AI ​​camera includes a first imaging unit with an AI algorithm and a second imaging unit with an AI algorithm. It can directly process high-definition video streams at the local edge, eliminating the high bandwidth consumption and network transmission latency of hundreds of milliseconds or even seconds caused by uploading video data to the cloud. Through the coordinated cooperation of the first imaging unit and the second imaging unit, the technical problem of "large field of view tracking" and "high-quality close-up" cannot be achieved at the same time. It also greatly endows the device with highly generalized scene adaptability and greatly improves the accuracy and efficiency of highlight fragment generation. By performing temporal matching between spatiotemporal skeleton features and a preset highlight action semantic library or a user-defined dynamic template library, it can meet the highly personalized action capture needs of users in different scenarios.

[0023] 2) The pose estimation model is run using the edge computing processing unit to analyze the first video stream. Compared to traditional cloud processing solutions, this application processes high-definition video streams directly at the local edge, eliminating the high bandwidth consumption and network transmission latency of hundreds of milliseconds or even seconds caused by uploading video data to the cloud. This is crucial for high-speed sports scenarios such as badminton and tennis, ensuring extremely short end-to-end time from detection to locking, enabling telephoto lenses to follow the target in real time, avoiding the target "running out" of the field of view due to processing delays, significantly reducing system latency, and achieving millisecond-level real-time response.

[0024] 3) This application extracts global and local features and performs similarity matching based on a pedestrian re-identification algorithm. In multi-person sports scenarios (such as doubles matches or team lessons), athletes often wear uniforms or sportswear of similar colors. Relying solely on global features (overall outline, color distribution) can easily lead to misidentification. This application introduces local features and combines them with global features for multi-granularity fusion matching. Even when multiple people have highly similar appearances, the system can accurately distinguish between the "target user" and the "interference object (teammate or opponent)" through detailed features, effectively preventing the "following the wrong person" phenomenon. During sports, it is common for the body to be partially obscured by the net, teammates, or equipment. The pedestrian re-identification algorithm has a natural advantage in handling partial occlusion: when global features are damaged due to occlusion, the system can still use unoccluded local features for matching. At the same time, the establishment of the recognition feature database allows the system to quickly re-identify and re-lock the target after a brief loss (such as running out of the frame and returning), achieving highly stable tracking that is "recoverable when lost and not lost when occluded".

[0025] 4) This application simulates the shallow depth-of-field effect of professional photography equipment through masking and digital background blurring, effectively eliminating cluttered background interference and forcing the viewer's attention to the target moving object, significantly enhancing the visual impact and cinematic artistic beauty of the video clips; it adopts non-linear timeline rescaling processing (i.e., intelligent speed adjustment), which can accurately slow down close-ups of key moments of action (such as hitting the ball or jumping), while accelerating the transition process, thereby optimizing the narrative rhythm within a limited time, preserving the exciting action details completely, and avoiding being long and boring, greatly enhancing the viewing experience and immersion of the video; the entire process from clip generation to special effects rendering is automated, eliminating the need for users to manually edit, cut out, and color grade tedious professional operations, realizing the efficient experience of "shooting and getting the finished product", allowing ordinary users to easily obtain professional-level sports highlight clips.

[0026] 5) In this application, the abstract concept of "target deviation" is transformed into a precise numerical indicator of positional deviation. This quantitative feedback mechanism provides the gimbal motor with a precise basis for error correction, avoiding lag or overshoot caused by fuzzy control and ensuring the accuracy of tracking actions. Real-time control signals generated based on the positional deviation drive the gimbal to perform horizontal / tilt adjustments, forming a closed-loop control system. This mechanism can compensate for the displacement caused by target movement in real time, ensuring that the moving target object always remains within the effective shooting range of the binocular acquisition module (wide-angle and telephoto), effectively solving the problems of "out of frame" or "losing track" caused by rapid target movement or the narrow field of view of the telephoto lens. This method also ensures spatial synchronization between wide-angle monitoring and telephoto close-ups. By keeping the moving target object within the effective range, the second imaging unit can continuously acquire stable and clear close-up images of the target, avoiding voids or out-of-focus telephoto images caused by target deviation. This provides a stable video source for subsequent highlight capture and feature extraction, significantly improving the reliability of automated tracking and providing an accurate video stream for subsequent highlight clip generation.

[0027] 6) In this application, by verifying both video stream and audio stream, it is possible to effectively filter out “empty waving” that only has movement but no sound, and “external interference” that only has sound but no movement, which greatly improves the accuracy of identifying video stream data and audio stream data in a specific time period and can generate accurate key segments, thus solving the technical problems of low single-modal recognition rate and high false alarm rate in the prior art. Attached Figure Description

[0028] Figure 1 The flowchart shown is a method for automatically generating motion highlight fragments based on an AI camera, as provided in an embodiment of this application.

[0029] Figure 2The flowchart shown is a process for identifying and locking a moving target object using an attitude estimation model based on the first imaging unit provided in an embodiment of this application.

[0030] Figure 3 The flowchart shown is a process for generating highlight segments based on a telephoto video frame sequence corresponding to a second video stream, as provided in an embodiment of this application.

[0031] Figure 4 The flowchart shown is provided in an embodiment of this application, illustrating the application of nonlinear time axis rescaling to a specific time sub-interval in an initial highlight clip to generate a multi-rate variable-speed highlight clip.

[0032] Figure 5 The flowchart shown is a position correction flowchart of the binocular acquisition module provided in the embodiment of this application.

[0033] Figure 6 The flowchart shown is a process for automatically generating motion highlight clips based on an AI camera and incorporating audio streams, as provided in an embodiment of this application.

[0034] Figure 7 The diagram shown is a structural diagram of an AI camera-based automatic motion highlight fragment generation device provided in an embodiment of this application.

[0035] Figure 8 The diagram shown is a structural diagram of the AI ​​camera provided in an embodiment of this application.

[0036] Component designation explanation

[0037] S11~S15 step 72 Second video stream acquisition module S21~S23 step 73 Target moving object recognition module S31~S33 step 74 Action trigger timestamp generation module S41~S44 step 75 Highlight fragment generation module S51~S52 step 80 AI camera S61~S64 step 81 gimbal motor 70 Automatic generation device for motion highlight segments based on AI camera 82 First imaging unit 71 First video stream acquisition module 83 Second imaging unit 84 Edge computing processing unit Detailed Implementation

[0038] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0039] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0040] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0041] like Figure 1 As shown, this application embodiment provides a flowchart of a method for automatically generating motion highlight segments based on an AI camera. The AI ​​camera includes a first imaging unit with an AI algorithm and a second imaging unit with an AI algorithm, such as... Figure 1 As shown, the automatic generation method for motion highlight segments based on AI cameras provided in this application includes the following steps S11 to S15.

[0042] S11, the first video stream under the first field of view is acquired in real time through the first imaging unit.

[0043] For example, the first imaging unit includes: a fixed-focus wide-angle lens, a zoom wide-angle lens, a low-distortion wide-angle lens, a regular wide-angle lens, etc.

[0044] It should be noted that the specific types of the first imaging units listed in the above examples are merely illustrative. In practical applications, any suitable first imaging unit can be selected to acquire the first video stream under the first field of view in real time based on specific application requirements. This application does not impose any restrictions on this.

[0045] In some embodiments, the number of AI cameras is at least one, and multiple AI cameras trigger timestamps synchronously via a network.

[0046] Multiple AI cameras trigger timestamps synchronously via the network, which forces multiple dispersed camera devices to maintain a high degree of consistency in the time dimension, eliminates time reference deviations caused by independent operation of devices, and ensures precise alignment of multiple video streams on the timeline. The unified timestamp provides a reliable index for AI algorithms to process multiple videos, enabling the system to quickly lock image segments of the same event from different perspectives, realize automated and seamless multi-view switching and splicing, greatly improve the timeliness of content output and enhance the efficiency and quality of multi-camera linkage editing.

[0047] S12, the second video stream under the second field of view is acquired in real time through the second imaging unit, and the telephoto video frame sequence corresponding to the second video stream is written to the buffer unit in a loop.

[0048] For example, the second imaging unit includes a catadioptric telephoto lens, a catadioptric telephoto lens, etc.

[0049] It should be noted that the specific types of the second imaging units listed in the above examples are only for illustrative purposes. In actual applications, any suitable second imaging unit can be selected to acquire the second video stream under the second field of view in real time based on specific application requirements. This application does not impose any restrictions on this.

[0050] For example, the cache unit includes any one of a cache unit, a linear buffer, and a double buffer.

[0051] In a preferred embodiment, the buffer unit adopts a circular buffer structure. This is necessary because the attitude estimation model incurs computational time for identifying highlight actions. Through continuous overwriting of the circular buffer, the system can retain a sequence of telephoto video frames of a preset duration prior to the current trigger moment in real time. When the matching conditions are met to generate an action trigger timestamp, the system can backtrack from this buffer unit to extract a complete sequence of telephoto video frames containing the action preparation period, action firing period, and action ending period. This solves the problem of AI triggering lagging behind physical actions and ensures the integrity of highlight segment acquisition.

[0052] S13, perform kinematic feature analysis on the first video stream based on the pose estimation model to identify and lock the target moving object.

[0053] For example, training the pose estimation model includes: determining human skeletal keypoints, which include: head, nose, left eye, right eye, left ear, right ear, upper body / arm, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, lower body / leg, left hip, left knee, right knee, left ankle, and right ankle. The data corresponding to these human skeletal keypoints is raw data and requires extensive manual or AI-assisted annotation. The purpose of annotation is to sequentially mark the coordinates of each human skeletal keypoint in the first video stream and save them to a label document. The label document can be in txt format or other text formats. After annotating the data in the first video stream, human pose data can be obtained. Inputting this human pose data into a GPU training server allows for the training of the pose estimation model.

[0054] For example, global and local features of the target moving object in the first video stream are extracted by a pose estimation model, and the target moving object is identified and locked based on the global and local features of the target moving object in the first video stream.

[0055] S14. Based on the posture estimation model, extract the spatiotemporal skeleton features of the target moving object, and perform temporal matching between the spatiotemporal skeleton features and a preset specular action semantic library or a user-defined dynamic template library. When the matching conditions are met, generate the corresponding action trigger timestamp.

[0056] For example, the preset highlight action semantic library includes: the moment of hitting a badminton shuttlecock, inverted balance movements in yoga (such as handstand), deep backbend movements (such as wheel pose), and extreme stretching movements (such as dancer pose).

[0057] It should be noted that the highlight actions in the preset highlight action semantic library listed in the above examples are only for illustrative purposes. In actual applications, other suitable highlight actions can be added to the preset highlight action semantic library based on specific application requirements. This application does not impose any restrictions on this.

[0058] In some embodiments, the steps of constructing the user-defined dynamic template library include: using the pose estimation model to monitor the motion of the target moving object in real time; when the target moving object is identified to make a preset interaction command gesture, automatically capturing the spatiotemporal skeletal trajectory sequence of the target moving object within a second preset time window before the interaction command gesture occurs; adding the spatiotemporal skeletal trajectory sequence as a target action template feature to the dynamic template library, so that the AI ​​camera has the ability to calibrate custom highlight actions under single sample or small sample conditions.

[0059] For example, step S1: Obtain a continuous sequence of skeletal key points (spatiotemporal skeletal feature extraction).

[0060] Suppose that in the first video stream acquired by the first imaging unit, the target moving object (user A) is playing badminton.

[0061] Single-frame extraction: The pose estimation model processes each frame of the video stream. For example, from frame t to frame...

[0062] Within a time window of t+n frames, the model identifies key points of user A's human skeleton, including: right shoulder (coordinates...). ), right elbow (coordinates) ), right wrist (coordinates) ), right hip (coordinates) ), right knee (coordinates) )wait.

[0063] Feature vectorization: The system vectorizes the two-dimensional coordinates (x, y) of these key points and the geometric features (such as joint angles and bone length ratios) calculated based on the bone points.

[0064] For example: calculate the elbow joint angle formed by "right shoulder - right elbow - right wrist" and the trunk twist angle formed by "right hip - right shoulder - right elbow".

[0065] Spatiotemporal feature construction: The above single-frame features are concatenated along the time axis to construct a spatiotemporal skeleton feature sequence.

[0066] This spatiotemporal skeletal feature sequence contains not only static posture information, but also the motion velocity and acceleration information of key points (such as the rate of change of the right wrist in the vertical direction).

[0067] Step S2: Call the preset specular action semantic library

[0068] The system retrieves the motion template for "the moment the badminton shuttlecock is hit" from storage.

[0069] Template Definition: In the highlight action semantic library, the "badminton smash" action is defined as a standard spatiotemporal feature sequence. This template feature includes the following key semantic criteria:

[0070] Preparatory phase: Lean back, raise the right elbow, with the elbow joint angle greater than 90° and less than 120°, and the torso twist angle reaches its maximum value.

[0071] The striking phase (core characteristics): The right elbow is quickly extended, and the elbow joint angle is rapidly increased to nearly 180°; the right wrist movement speed reaches its peak (exceeding the preset threshold); the body is airborne (the vertical coordinates of the key points of both ankles reach their minimum values ​​simultaneously).

[0072] Follow-through phase: Swing your arm to the left side of your body and lower your center of gravity.

[0073] Step S3: Timing Matching Calculation

[0074] The system uses a dynamic time warping algorithm or a sliding window similarity algorithm to compare the real-time extracted input sequence with the template sequence.

[0075] Similarity calculation: Calculate the Euclidean distance or cosine similarity between the real-time feature vector and the template feature vector.

[0076] Process simulation: At time t, the user is in the "raise and observe" state, with low feature similarity (e.g., 0.3), and the condition is not triggered. At time t+5, the user jumps and leans back, and the feature curve begins to coincide with the template's "preparatory stage," with the similarity rising to 0.6.

[0077] At time t+8, the user swings the racket downwards, causing a surge in the speed of the right wrist and a sudden widening of the elbow angle. At this moment, the real-time feature vector closely matches the "hitting phase" of the template, resulting in a similarity score of 0.92 (assuming a matching threshold of 0.85).

[0078] Step S4: Generate action trigger timestamp

[0079] Since the similarity score (0.92) exceeded the preset matching threshold (0.85), and the state was maintained for a sufficient number of frames (excluding instantaneous jitter interference), the matching condition was met, and the match was successful.

[0080] Locking time: The system locks the frame corresponding to the similarity peak (i.e., the interval near frame t+8) as the core point of the action.

[0081] Output: The system generates an action trigger timestamp (e.g., video frame 00:01:12:05) and tags it "badminton - smash".

[0082] It should be noted that the time values ​​and similarity scores listed in the above examples are merely illustrative. In actual applications, other suitable values ​​can be selected based on specific application requirements, and this application does not impose any restrictions on this.

[0083] S15, generate a highlight segment based on the telephoto video frame sequence corresponding to the first preset time window containing the action trigger timestamp.

[0084] In some embodiments, the method further includes: using an edge computing processing unit to identify non-target moving objects in the telephoto video frame sequence other than the target moving object; extracting the face region or the identifying feature region of the non-target moving object; and performing real-time anonymization masking processing on the face region or the identifying feature region during the generation of the highlight segment.

[0085] This application provides a method for automatically generating motion highlight clips based on an AI camera. The AI ​​camera includes a first imaging unit with an AI algorithm and a second imaging unit with an AI algorithm. It can directly process high-definition video streams at the local edge, eliminating the high bandwidth consumption and network transmission latency of hundreds of milliseconds or even seconds caused by uploading video data to the cloud. Through the coordinated operation of the first and second imaging units, the technical challenge of simultaneously achieving "wide field of view tracking" and "high-quality close-ups" is perfectly solved, greatly endowing the device with highly generalized scene adaptability and significantly improving the accuracy and efficiency of highlight clip generation. By temporally matching spatiotemporal skeleton features with a preset highlight motion semantic library or a user-defined dynamic template library, it can meet the highly personalized motion capture needs of users in different scenarios.

[0086] like Figure 2 As shown, this application embodiment provides a flowchart for identifying and locking a moving target object based on a pose estimation model at the first imaging unit end, as follows: Figure 2 As shown, the method for identifying and locking a moving target object based on the attitude estimation model of the first imaging unit provided in this application includes the following steps S21 to S23.

[0087] S21, the edge computing processing unit runs the pose estimation model to perform kinematic analysis on the first video stream, and extracts the global and local features of the target moving object in the first video stream.

[0088] Global features include clothing color and texture, and body proportions, while local features include head and shoulder features, and sports accessories features.

[0089] It should be noted that the specific features of the global and local features listed in the above examples are only for illustrative purposes. In actual applications, other suitable global and / or local features can be selected based on specific application requirements, and this application does not impose any restrictions on this.

[0090] Before extracting the global and local features of the target moving object in the first video stream, the background blurring process of the telephoto video frame sequence includes: using the AI ​​computing power of the edge computing processing unit, without the need for a depth sensor, based on the skeletal key points and contour features of the target moving object, calculating the boundary between the foreground target moving object and the background frame by frame in the telephoto video frame sequence in real time, and applying Gaussian blur or lens optical bokeh simulation rendering to the surrounding environment while keeping the target in sharp focus, thereby extracting accurate global and local features of the target moving object in the first video stream.

[0091] S22, establish the recognition feature library corresponding to the global features and the local features.

[0092] Specifically, a recognition feature library is established by extracting global and local features of the target moving object.

[0093] For example, after the AI ​​camera is powered on and enters working mode, the target moving object stands within the field of view of the first imaging unit and performs a predefined trigger gesture (such as waving or making a heart shape). After the pose estimation model recognizes the gesture, the system performs a dual registration operation: 1) Action template registration: Extract the spatiotemporal skeletal features of the gesture and store it as a user-defined dynamic trigger template in the specular action semantic library for subsequent recognition of specific specular actions. 2) Identity feature extraction: At the moment the gesture is detected, the system simultaneously captures an image frame sequence containing the target user's entire body, extracts global and local features, generates a user-specific feature vector, and stores it in the recognition feature library.

[0094] After identity registration is complete, the system enters the formal video capture mode. The gimbal motion control module sets this unique feature vector as the sole tracking priority identifier. In subsequent complex motion scenarios lasting several hours (such as competitions or training), even if multiple similar-looking interfering objects appear in the frame, the gimbal's horizontal / tilt motion control commands remain strictly bound to this unique feature vector. By calculating the similarity between each object in the video stream and the registered feature vector in real time, the system ensures that the gimbal only responds to the movement displacement of the target moving object, achieving precise and continuous locking onto the moving object and effectively avoiding mistracking or losing track of it.

[0095] It should be noted that the purpose of establishing the recognition feature library corresponding to the global features and the local features is to ensure that when the target moving object briefly runs out of the wide-angle frame or is obscured by others, it can still be instantly re-locked through similarity matching when it reappears, thus ensuring that the gimbal motor does not "follow the wrong person".

[0096] S23, Based on the pedestrian re-identification algorithm, the global and local features of multiple moving objects are matched with the global and local features in the identification feature library to identify and lock the target moving object.

[0097] For example, the steps of global feature matching are as follows: calculate the cosine distance or Euclidean distance between the global feature and the global features in the recognition feature library to obtain the global similarity score. The purpose of global feature matching is to quickly filter out candidate objects with similar overall appearance and eliminate significantly different interference items.

[0098] Local Feature Matching: For candidate objects with high global similarity, the similarity of each local feature is further calculated. Real-time local features (such as upper body features) are aligned and matched with corresponding local features in the database, and a local similarity score is calculated. The role of local feature matching: In multi-person sports scenarios, when two people are wearing the same color team uniform, they can be accurately distinguished by local details such as "shoe color", "leg protective gear" or "facial features".

[0099] Weighted fusion and determination:

[0100] Fusion calculation: A weighted fusion strategy is used to calculate the total similarity score. :

[0101] in, and Preset weighting coefficients (e.g.) =0.4, =0.6), focusing on the ability to distinguish local details. Represents global features. Indicates local features.

[0102] Threshold determination: The calculated threshold value will be used to determine the threshold value. Matching threshold with preset Compare. If ≥ If so, then the currently moving object is determined to be the target moving object. < If so, the currently moving object is determined to be an interference object.

[0103] This application provides a method for identifying and locking a moving target based on a pose estimation model at the first imaging unit. In this method, the pose estimation model is run using an edge computing processing unit to analyze a first video stream. Compared to traditional cloud processing solutions, this application directly processes high-definition video streams at the local edge, eliminating the high bandwidth consumption and network transmission delays of hundreds of milliseconds or even seconds caused by uploading video data to the cloud. This is crucial for high-speed sports scenarios such as badminton and tennis, ensuring extremely short end-to-end time from detection to locking, enabling telephoto lenses to follow the target in real time, avoiding the target "running out" of the field of view due to processing delays, significantly reducing system latency, and achieving millisecond-level real-time response. The method also extracts global and local features and performs similarity matching based on a pedestrian re-identification algorithm. In multi-person sports scenarios (such as doubles matches or team lessons), athletes often wear uniforms or similar colored sportswear, making misidentification easily possible using only global features (overall outline, color distribution). This application introduces local features and combines them with global features for multi-granularity fusion matching. Even when multiple people have highly similar appearances, the system can accurately distinguish between the "target user" and the "interference object (teammate or opponent)" through detailed features, effectively preventing the phenomenon of "following the wrong person." During sports activities, the body is often partially obscured by the net, teammates, or equipment. Pedestrian re-identification algorithms have a natural advantage in handling partial occlusion: when global features are damaged due to occlusion, the system can still use unoccluded local features for matching. At the same time, the establishment of the recognition feature database allows the system to quickly re-identify and re-lock on the target after it is briefly lost (such as running out of the frame and then returning), achieving highly stable tracking that is "recoverable when lost and not lost when occluded."

[0104] like Figure 3 As shown, this application provides a flowchart for generating highlight segments based on a telephoto video frame sequence corresponding to a second video stream, as follows: Figure 3 As shown, the method for generating highlight segments based on the telephoto video frame sequence corresponding to the second video stream provided in this application embodiment includes the following steps S31 to S33.

[0105] S31, extract the telephoto video frame sequence corresponding to the second video stream within the first preset time window containing the action trigger timestamp.

[0106] For example, the action trigger timestamp is used This indicates that the first preset time window can be... , , wait.

[0107] It should be noted that the above-mentioned first preset time window is only used as an example. In actual applications, the first preset time window of any size can be determined based on specific application requirements, and this application does not impose any restrictions on this.

[0108] S32, Generate an initial highlight segment based on the telephoto video frame sequence.

[0109] Specifically, the second imaging unit includes an AI algorithm that can automatically generate a telephoto video frame sequence to create an initial highlight segment.

[0110] S33, perform post-processing rendering on the initial highlight fragment. The post-processing rendering includes masking the target moving object in the initial highlight fragment and performing digital blurring on its background area, and applying non-linear time axis rescaling processing to a specific time sub-interval in the initial highlight fragment to generate a multi-fold speed-variable highlight fragment.

[0111] This application provides a method for generating highlight clips based on a telephoto video frame sequence corresponding to a second video stream. In this method, masking and digital background blurring simulate the shallow depth-of-field effect of professional photography equipment with a large aperture, effectively eliminating cluttered background interference and forcing the viewer's attention to the moving target, significantly enhancing the visual impact and cinematic aesthetics of the video clips. The use of non-linear timeline rescaling (i.e., intelligent speed adjustment) enables precise slow-motion close-ups of key moments in the action (such as hitting a ball or jumping), while accelerating the transition process, thereby optimizing the narrative rhythm within a limited timeframe. This preserves the details of exciting actions while avoiding lengthy and tedious content, greatly enhancing the video's watchability and immersive experience. The entire process from clip generation to special effects rendering is automated, eliminating the need for users to manually edit, cut out images, and adjust colors, achieving a highly efficient "shoot and be filmed" experience, allowing ordinary users to easily obtain professional-quality motion highlight clips.

[0112] like Figure 4 As shown, this application provides a flowchart for applying nonlinear time axis rescaling to a specific time sub-interval in an initial highlight clip to generate a multi-rate variable-speed highlight clip, as illustrated below. Figure 4 As shown in the embodiments of this application, the method for applying nonlinear time axis rescaling to a specific time sub-interval in an initial highlight segment to generate a multi-rate variable speed highlight segment includes the following steps S41 to S44.

[0113] S41, the long-focus video frame sequence in the initial highlight segment is divided into an action preparation period, an action firing period, and an action ending period based on the action phase.

[0114] The specific time sub-intervals include the time sub-intervals corresponding to the action preparation period, the action firing period, and the action end period.

[0115] Specifically, AI algorithms can be used to automatically divide the long-focus video frame sequence in the initial highlight segment into an action preparation period, an action firing period, and an action ending period based on the action phase.

[0116] For example, the definition of the action preparation period: the system traces back the timeline of the first video stream to detect changes in the kinematic state of the target moving object, and the end point is the instant when the highlight action occurs.

[0117] Starting point: The moment when the target moving object transitions from a "relatively static state" or "normal motion state" to a "power-accumulating state" is taken as the starting point. For example, in a badminton smash, the system detects the moment when the athlete begins to jump, draws back the racket, and leans back, marking this as the start of the preparation period.

[0118] Definition of the end of the action: The system tracks the second video stream backward to detect the recovery state after the highlight action is completed.

[0119] Start point: The point at which the core phase of the movement ends. End point: The point at which the target object completes the movement buffer, returns to a standing posture, or transitions to the next routine movement. For example, the moment after a smash lands and the center of gravity stabilizes and the racket returns to its original position is marked as the end of the end phase.

[0120] S42, map the action preparation period and the action end period in the initial highlight segment to a first playback rate.

[0121] It should be noted that the specific value of the first playback rate can be determined based on the specific application scenario, and this application does not impose any restrictions on it.

[0122] S43, the action firing period containing the action trigger timestamp in the initial highlight segment is mapped to a second playback rate lower than the first playback rate.

[0123] In some embodiments, the method further includes: automatically matching the first playback rate based on the intensity of the action during the action preparation period and the action end period; and automatically matching the second playback rate based on the intensity of the action during the action firing period.

[0124] For example, the first playback rate and the second playback rate can be automatically matched based on an adaptive playback rate mapping algorithm.

[0125] Specifically, the working principle of the adaptive playback rate mapping algorithm includes: feature quantization (calculating the intensity), determination of specific time sub-intervals, and playback rate mapping.

[0126] Calculating the intensity index of movement: The intensity index of movement is calculated by weighting the velocity and acceleration at key points, and the formula is expressed as:

[0127]

[0128] in, Indicator of the intensity of the action , Let represent the weighting coefficients, and < (Emphasizing the sensitivity of acceleration to "hitting the ball"), N represents the total number of key points. Indicates the first Key points (such as wrist, racket) at The velocity vector at time t, Indicates the first Key points (such as wrist, racket) at The acceleration vector at time t.

[0129] Action phase judgment logic: Set judgment threshold , and .

[0130] like > and duration < This is determined to be the firing phase of the action.

[0131] like < < If so, it is determined to be the preparation period or the end period of the action.

[0132] Playback rate mapping function:

[0133] Calculate the playback rate based on the judgment results and the severity of the situation. .

[0134]

[0135] in, Indicates the first playback rate. This represents the second playback rate, C represents a constant, and K represents the mapping scaling factor.

[0136] For example, during the running phase of the preparation and end phases of a badminton action, if the intensity of the running is relatively gentle, the first playback rate corresponding to the running phase will be automatically switched to 1.5x or 2.0x fast forward based on the adaptive playback rate mapping algorithm; while at the moment of hitting the shuttlecock during the hitting phase of a badminton action, if the intensity of the hitting is relatively strong, the second playback rate corresponding to the hitting moment will be automatically switched to 8x or 10x slow motion based on the adaptive playback rate mapping algorithm.

[0137] It should be noted that the specific value of the second playback rate can be determined based on the specific application scenario, and this application does not impose any restrictions on it.

[0138] S44, automatically generate visually impactful high-speed highlights based on the first playback rate and the second playback rate.

[0139] This application provides a method for applying nonlinear timeline rescaling to specific time sub-intervals in an initial highlight clip to generate highlight clips with multiple speed ratios. In this method, by mapping the action preparation period and action ending period to a faster first playback rate, non-critical redundant time is effectively compressed, bland transition processes are eliminated, and video dragging is avoided, making the highlight clip content more concise and compact, which conforms to the fast-paced viewing habits of short videos. The action firing period containing key moments is mapped to a slower second playback rate, and the time dimension of the core action is extended by using slow motion effects, allowing users to clearly capture instantaneous details that are difficult to distinguish with the naked eye, greatly enhancing the visual impact and expressiveness of the picture.

[0140] The non-linear speed-changing processing, combining fast and slow motion, simulates the editing logic of professional sports broadcasts, creating a well-paced and layered visual rhythm. This rhythmic variation not only guides the user's visual focus but also endows the video with a cinematic artistic quality, significantly enhancing the user's viewing immersion and satisfaction.

[0141] like Figure 5 As shown in the figure, this application embodiment provides a flowchart of the position correction of a binocular acquisition module, as follows: Figure 5 As shown, the method for position correction of the binocular acquisition module provided in this application embodiment includes the following steps S51 to S52.

[0142] S51, determine the positional deviation of the target moving object in the first video stream.

[0143] Specifically, by extracting the minimum bounding rectangle of the target moving object, the coordinates of the geometric center point of the minimum bounding rectangle are calculated to represent the real-time position of the target moving object in the current frame. A preset reference position of the target moving object is determined, where the reference position can represent the geometric center of the first video stream. The position deviation is determined using Euclidean distance based on the real-time position and the reference position.

[0144] S52, based on the position deviation, a control signal is generated in real time to adjust the horizontal and / or pitch axis of the gimbal motor, so that the target moving object remains within the effective shooting range of the binocular acquisition module composed of the first imaging unit and the second imaging unit.

[0145] The binocular acquisition module serves as a load and is rigidly fixed to the rotating platform of the gimbal motor.

[0146] In some embodiments, the method further includes: controlling the gimbal motor to rotate so that the target moving object is located in the central region of the second video stream, including: acquiring a motion coordinate sequence of the target moving object within a preset historical duration; using a trajectory prediction algorithm to calculate the predicted position of the target moving object in the next sampling period based on the motion coordinate sequence; calculating an angular velocity compensation vector for the rotation of the gimbal motor based on the current real-time position of the target moving object and the predicted position; matching a corresponding dynamic velocity response curve according to the angular velocity compensation vector; and driving the gimbal motor to perform predictive interpolation rotation according to the dynamic velocity response curve to achieve smooth tracking of the target moving object.

[0147] This application provides a position correction method for a binocular acquisition module. In this method, the abstract concept of "target deviation" is transformed into a precise numerical indicator of position deviation. This quantitative feedback mechanism provides the gimbal motor with accurate error correction, avoiding lag or overshoot caused by fuzzy control and ensuring the accuracy of tracking actions. Based on the position deviation, a control signal is generated in real time to drive the gimbal for horizontal / tilt adjustment, forming a closed-loop control system. This mechanism can compensate for the displacement caused by target movement in real time, ensuring that the moving target object always remains within the effective shooting range of the binocular acquisition module (wide-angle and telephoto), effectively solving the problems of "out of frame" or "lost tracking" caused by rapid target movement or the narrow field of view of the telephoto lens. This method also ensures spatial synchronization between wide-angle monitoring and telephoto close-ups. By keeping the moving target object within the effective range, the second imaging unit can continuously acquire stable and clear target close-up images, avoiding voids or out-of-focus telephoto images caused by target deviation. This provides a stable video source for subsequent highlight capture and feature extraction, significantly improving the reliability of automated tracking and providing an accurate video stream for subsequent highlight clip generation.

[0148] In an optimized implementation, the control of the gimbal motor no longer relies solely on real-time feedback of the target's deviation from the center, but instead introduces a kinematic prediction mechanism.

[0149] Specifically, the edge computing processing unit maintains a sliding window in real time, recording the spatial coordinates of the target moving object over the past N frames. Through prediction algorithms such as linear regression or Kalman filtering, the system can predict the target's motion trend within the next 10-50 milliseconds.

[0150] When driving the gimbal motor, the system does not rotate at a constant speed. Instead, it matches a dynamic speed response curve (such as an S-curve or a Bezier curve) based on the target's acceleration and deceleration characteristics. This "predictive tweening" can offset the gimbal motor's start-up delay and mechanical inertia, ensuring that the image captured by the telephoto lens maintains a smooth and stable visual effect even when the target changes direction rapidly (such as a badminton smash or a soccer ball change of direction), avoiding the image oscillation or lag commonly found in traditional PID feedback control.

[0151] While existing intelligent vision terminals integrate basic motion detection modules, their algorithmic logic primarily relies on pixel-level differential or single-dimensional visual human detection. In complex application scenarios, this type of technology is highly susceptible to interference from dynamic elements in the background environment (such as swaying branches and leaves, or the passing of irrelevant personnel), inducing numerous false alarms and reducing the monitoring signal-to-noise ratio. Furthermore, for small, high-speed moving targets (such as badminton shuttlecocks or table tennis balls), relying solely on visual tracking algorithms often results in loss of tracking or decreased motion recognition accuracy due to the target's small size or blurred motion. Although some technical solutions introduce acoustic decibel thresholds as triggering mechanisms, in complex acoustic environments with high reverberation and strong background noise, such as stadiums, non-target sound sources like audience shouts and shoe friction noises can easily exceed the preset threshold, failing to meet the application requirements for low false triggering.

[0152] To address the aforementioned technical issues, this application provides a method for automatically generating motion highlight clips based on an AI camera and incorporating audio streams. For example... Figure 6 As shown, this application embodiment provides a flowchart for the automatic generation of motion highlight clips based on an AI camera, combining audio streams, as follows: Figure 6 As shown, the automatic generation method for motion highlight segments based on AI cameras combined with audio streams provided in this application embodiment includes the following steps S61 to S64.

[0153] S61, Acquire the audio stream that is concurrent with the second video stream.

[0154] S62, perform acoustic feature analysis on the audio stream to determine the audio trigger timestamp of the highlight segment in the audio stream.

[0155] Specifically, the audio stream data is processed to extract the acoustic feature vector corresponding to the audio stream data; the acoustic feature vector is input into an audio classification model to obtain the preset voiceprint features recognized by the audio classification model; the time point when the preset voiceprint features with energy greater than a preset noise energy occur is determined as the audio trigger timestamp.

[0156] The acoustic feature vector includes MFCC, Fbank, LPC coefficients, or embedding features extracted from deep neural networks.

[0157] S63, determine the audio frame sequence corresponding to the audio stream within the first preset time window of the audio trigger timestamp.

[0158] S64, Generate a highlight segment based on the telephoto video frame sequence and the audio frame sequence corresponding to the second video stream within a first preset time window containing the action trigger timestamp.

[0159] For example, calculation If the difference is less than a preset time window threshold, it is considered a valid highlight moment. ;in, .

[0160] based on Backtrack from the cache unit and extract a preset time interval. The audio and video data are used to generate highlight clips.

[0161] In this application, by verifying both video and audio streams, it is possible to effectively filter out "empty waving" that only has movement but no sound, and "external interference" that only has sound but no movement. This greatly improves the accuracy of identifying video and audio stream data for specific time periods and can generate accurate key segments, solving the technical problems of low single-modal recognition rate and high false alarm rate in the prior art.

[0162] In an optimized implementation, to ensure the accuracy of multimodal data analysis and prevent audio-visual asynchrony caused by differences in different sensor links, this application employs a globally unified clock synchronization mechanism. Specifically, the system establishes a hardware-level global reference clock and assigns a globally unique timestamp to the audio acquisition stream, the first video stream, and the second video stream stored in the buffer unit.

[0163] When the edge computing processing unit identifies the acoustic features corresponding to a highlight action (such as the popping sound of a badminton shuttlecock being hit), the system not only obtains the audio trigger time but also performs millisecond-level spatiotemporal alignment between the audio trigger point and the corresponding telephoto video frame sequence in the buffer unit using a global reference clock. This mechanism effectively compensates for the asymmetric delay offsets caused by the large data volume encoding and decoding of the video stream, buffer unit read and write operations, and audio stream processing, ensuring that the final synthesized highlight clip achieves precise audio-visual synchronization in physical timing and eliminating the image-sound misalignment caused by technical links.

[0164] The scope of protection of the automatic generation method for motion highlight segments based on AI cameras described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0165] The method provided in this application is not limited to generating clips locally, but can be further extended to a cloud-based collaborative ecosystem. In an extended embodiment, the method further includes: using the camera's communication module (including but not limited to Wi-Fi, 5G, or Bluetooth Low Energy) to automatically and asynchronously upload the generated highlight clips containing motion metadata to an associated mobile terminal or cloud server.

[0166] The mobile terminal is equipped with a dedicated sports social application, which automatically synchronizes and publishes clips via a communication link with a cloud-based sports community platform. In this scenario, the system can automatically match challenge topics or leaderboards within the community based on the semantic tags of highlight actions (such as 'spiking' or 'game-winning shot'), enabling real-time flow and interaction of highlight moments within the social matrix. This closed-loop architecture of "local collection - cloud synchronization - community sharing" not only enhances the user's immediate sense of accomplishment at the sports event but also provides a standardized data interface for subsequent cloud-based big data-driven sports performance analysis and community-based operations.

[0167] This application also provides an automatic motion highlight fragment generation device based on an AI camera. The automatic motion highlight fragment generation device based on an AI camera can implement the automatic motion highlight fragment generation method based on an AI camera described in this application. However, the implementation device of the automatic motion highlight fragment generation method based on an AI camera described in this application includes, but is not limited to, the structure of the automatic motion highlight fragment generation device based on an AI camera listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0168] like Figure 7 As shown, in one embodiment, the motion highlight segment automatic generation device 70 based on AI camera of this application includes a first video stream acquisition module 71, a second video stream acquisition module 72, a target moving object recognition module 73, an action trigger timestamp generation module 74, and a highlight segment generation module 75.

[0169] The first video stream acquisition module 71 is used to acquire the first video stream under the first field of view in real time through the first imaging unit.

[0170] The second video stream acquisition module 72 is used to acquire the second video stream under the second field of view in real time through the second imaging unit, and to cyclically write the telephoto video frame sequence corresponding to the second video stream into the buffer unit.

[0171] The target moving object recognition module 73 is used to perform kinematic feature analysis on the first video stream based on the pose estimation model, and to identify and lock the target moving object.

[0172] The action trigger timestamp generation module 74 is used to extract the spatiotemporal skeleton features of the target moving object based on the posture estimation model, perform temporal matching of the spatiotemporal skeleton features with a preset specular action semantic library or a user-defined dynamic template library, and generate the corresponding action trigger timestamp when the matching conditions are met.

[0173] The highlight clip generation module 75 is used to generate highlight clips based on the corresponding telephoto video frame sequence within a first preset time window containing the action trigger timestamp.

[0174] The structure and principle of the first video stream acquisition module 71, the second video stream acquisition module 72, the target moving object recognition module 73, the action trigger timestamp generation module 74, and the highlight fragment generation module 75 correspond one-to-one with the steps in the above-mentioned automatic generation method of motion highlight fragments based on AI cameras, so they will not be described again here.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed apparatus or method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0176] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0177] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0178] This application also provides an AI camera. Figure 8 The diagram shown is a structural schematic of an AI camera 80 in one embodiment of this application. The automatic generation method for motion highlight fragments based on an AI camera provided in this embodiment can be applied to… Figure 8 The AI ​​camera shown is 80, but it is not limited to this. For example... Figure 8 As shown, the AI ​​camera 80 includes a gimbal motor 81; a binocular acquisition module, which includes a first imaging unit 82 and a second imaging unit 83; wherein the first imaging unit is used to capture a first video stream containing the complete motion field, and the second imaging unit is used to capture a second video stream of the target moving object; the binocular acquisition module is rigidly fixedly mounted on the rotating platform of the gimbal motor as a load; and an edge computing processing unit 84 or an AI processing chip, used to implement the automatic generation method of motion highlight segments based on the AI ​​camera according to any one of the first aspects.

[0179] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0180] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0181] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0182] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0183] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for automatically generating motion highlight segments based on an AI camera, characterized in that, The AI ​​camera includes a first imaging unit with an AI algorithm and a second imaging unit with an AI algorithm, and the method includes: The first video stream under the first field of view is acquired in real time through the first imaging unit; The second video stream under the second field of view is acquired in real time by the second imaging unit, and the telephoto video frame sequence corresponding to the second video stream is written to the cache unit in a loop. Based on the attitude estimation model, kinematic feature analysis is performed on the first video stream to identify and lock the target moving object; Based on the pose estimation model, the spatiotemporal skeleton features of the target moving object are extracted from the second video stream. The spatiotemporal skeleton features are then matched temporally with a preset specular motion semantic library or a user-defined dynamic template library. When the matching conditions are met, a corresponding action trigger timestamp is generated. A highlight clip is generated based on the corresponding telephoto video frame sequence within a first preset time window containing the action trigger timestamp.

2. The method according to claim 1, characterized in that, Based on the pose estimation model, kinematic feature analysis is performed on the first video stream to identify and lock onto the target moving object, including: The edge computing processing unit runs the pose estimation model to perform kinematic analysis on the first video stream, and extracts the global and local features of the target moving object in the first video stream. Establish a recognition feature library corresponding to the global features and the local features; Based on the pedestrian re-identification algorithm, the global and local features of multiple moving objects are matched with the global and local features in the identification feature library to identify and lock the target moving object.

3. The method according to claim 1, characterized in that, The generation of highlight clips based on the corresponding telephoto video frame sequence within a first preset time window containing the action trigger timestamp includes: Extract the telephoto video frame sequence corresponding to the second video stream within a first preset time window containing the action trigger timestamp; An initial highlight segment is generated based on the telephoto video frame sequence; The initial specular fragment is post-processed and rendered. The post-processing rendering includes masking the target moving object in the initial specular fragment and performing digital blurring on its background area, as well as applying non-linear time axis rescaling to a specific time sub-interval in the initial specular fragment to generate specular fragments with multiple speed ratios.

4. The method according to claim 3, characterized in that, Applying nonlinear time-axis rescaling to specific time sub-intervals within the initial highlight clip to generate multi-rate variable-speed highlight clips includes: The long-focus video frame sequence in the initial highlight segment is divided into an action preparation period, an action firing period, and an action end period based on the action phase. The specific time sub-interval includes the time sub-intervals corresponding to the action preparation period, the action firing period, and the action end period. Map the action preparation period and the action end period in the initial highlight segment to a first playback rate; The action firing period, which contains the action trigger timestamp in the initial highlight clip, is mapped to a second playback rate that is lower than the first playback rate. Based on the first playback rate and the second playback rate, visually impactful high-speed highlights are automatically generated using multiple speed ratios.

5. The method according to claim 4, characterized in that, The method further includes: The first playback rate is automatically matched based on the intensity of the action during the preparation period and the end period of the action. The second playback rate is automatically matched based on the intensity of the action during the firing phase.

6. The method according to claim 1, characterized in that, The steps for building the user-defined dynamic template library include: The motion of the target moving object is monitored in real time using the posture estimation model. When the target moving object makes a preset interactive command gesture, the spatiotemporal skeletal trajectory sequence of the target moving object within a second preset time window before the interactive command gesture occurs is automatically captured. The spatiotemporal skeletal trajectory sequence is added to the dynamic template library as a target action template feature, so that the AI ​​camera has the ability to calibrate custom highlight actions under single sample or small sample conditions.

7. The method according to claim 1, characterized in that, The method further includes: Determine the positional deviation of the target moving object in the first video stream; Based on the position deviation, a control signal is generated in real time to adjust the horizontal and / or pitch axis of the gimbal motor, so that the target moving object remains within the effective shooting range of the binocular acquisition module composed of the first imaging unit and the second imaging unit, wherein the binocular acquisition module is rigidly fixed on the rotating platform of the gimbal motor as a load.

8. The method according to claim 7, characterized in that, The method further includes: Controlling the rotation of the gimbal motor to position the target moving object in the center region of the second video stream includes: Obtain the motion coordinate sequence of the target moving object within a preset historical time period; The predicted position of the target moving object in the next sampling period is calculated based on the motion coordinate sequence using a trajectory prediction algorithm; Based on the current real-time position of the target moving object and the predicted position, calculate the angular velocity compensation vector of the gimbal motor rotation; The dynamic velocity response curve is matched according to the angular velocity compensation vector; The gimbal motor is driven to perform predictive interpolation rotation according to the dynamic speed response curve in order to achieve smooth tracking of the target moving object.

9. The method according to claim 1, characterized in that, The number of AI cameras is at least one, and multiple AI cameras trigger timestamps synchronously via a network.

10. The method according to claim 1, characterized in that, The method further includes: Acquire the audio stream that is concurrent with the second video stream; Perform acoustic feature analysis on the audio stream to determine the audio trigger timestamps of the highlight segments in the audio stream; Determine the audio frame sequence corresponding to the audio stream within the first preset time window of the audio trigger timestamp; A highlight clip is generated based on the telephoto video frame sequence and the audio frame sequence corresponding to the second video stream within a first preset time window containing the action trigger timestamp.

11. The method according to claim 1, characterized in that, The method further includes: The edge computing processing unit is used to identify non-target moving objects in the telephoto video frame sequence, excluding the target moving object. Extract the facial region or identifying feature region of the non-target moving object; During the generation of the highlight fragment, real-time anonymization masking is performed on the face region or the distinctive feature region.

12. An automatic motion highlight fragment generation device based on an AI camera, characterized in that, The device includes: The first video stream acquisition module is used to acquire the first video stream under the first field of view in real time through the first imaging unit; The second video stream acquisition module is used to acquire the second video stream under the second field of view in real time through the second imaging unit, and to write the telephoto video frame sequence corresponding to the second video stream into the cache unit in a loop. The target moving object recognition module is used to perform kinematic feature analysis on the first video stream based on the pose estimation model, and to identify and lock the target moving object. The action trigger timestamp generation module is used to extract the spatiotemporal skeleton features of the target moving object based on the pose estimation model, and perform temporal matching of the spatiotemporal skeleton features with a preset specular action semantic library or a user-defined dynamic template library. When the matching conditions are met, the corresponding action trigger timestamp is generated. The highlight clip generation module is used to generate highlight clips based on the corresponding telephoto video frame sequence within a first preset time window containing the action trigger timestamp.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 11.

14. An AI camera, characterized in that, The AI ​​camera includes: Gimbal motor; A binocular acquisition module, comprising a first imaging unit and a second imaging unit; wherein the first imaging unit is used to capture a first video stream containing the complete motion field, and the second imaging unit is used to capture a second video stream of a target moving object; the binocular acquisition module is rigidly fixedly mounted on the rotating platform of the gimbal motor as a load; An edge computing processing unit or AI processing chip is used to implement the automatic generation method of motion highlight fragments based on an AI camera according to any one of claims 1 to 11.

Images