Multi-modal data fusion video generation system and method

Abstract

The invention relates to a video generation system, which comprises an image acquisition device and a background server, and is characterized in that the image acquisition device comprises an event camera module (10) used for capturing a brightness change event in a scene and generating original event stream data; the motion sensor module (11) is used for acquiring motion posture data of the image acquisition equipment; the processing module (12) is used for carrying out real-time privacy protection processing on the original event stream data and carrying out data fusion on the motion posture data and the event stream data subjected to privacy protection processing to generate 4D event trajectory data; the communication module (13) is used for uploading the 4D event trajectory data processed by the processing module (12) to a background server; and the power supply module (14) is used for supplying power to the image acquisition equipment, and the background server (2) generates a video from the received 4D event trajectory data by using a video generation AI algorithm. The invention also discloses a method for generating a video by using the system.

Description

TECHNICAL FIELD

[0001] The present application relates to a video generation system and method, in particular to a multi-modal data fusion video generation system and method based on an event camera, and more particularly to a video generation system and video generation method that can be applied to a wearable device. BACKGROUND

[0002] Event cameras are a new type of vision sensor that captures only changes in brightness in a scene based on asynchronous pixel responses, rather than full-frame images. This technology has the advantages of high temporal resolution, low latency and low power consumption, and is widely used in autonomous driving, robot navigation and surveillance. However, existing event cameras have the following limitations.

[0003] Insufficient privacy protection: Event cameras may capture sensitive information (such as faces, license plates) when shooting, but existing technologies generally lack real-time privacy processing capabilities, resulting in data leakage risks.

[0004] Lack of 4D data fusion: Traditional event cameras only provide 2D event streams, without deep fusion with motion sensors (such as IMU) and timestamps, and cannot generate complete 4D (three-dimensional space + time) scene representations.

[0005] Limited power optimization: Although event cameras themselves have low power consumption, system-level design (such as data transmission and processing) is often not optimized, resulting in high overall energy consumption.

[0006] Low video generation efficiency: Existing methods rely heavily on continuous event stream reconstruction of video, but this approach is computationally complex and does not fully utilize timed shooting and background AI algorithms, resulting in poor video quality and high latency.

[0007] For example, patent US20200007851A1 discloses an event camera system, but does not address privacy protection; patent CN110290327A describes a dynamic scene reconstruction based on an event camera, but does not integrate motion sensors for 4D data fusion. Therefore, there is a need for an innovative solution to address the above problems.

[0008] In addition, traditional pet wearable devices mainly use GPS positioning and health monitoring (such as steps, heart rate), and cannot intuitively understand the specific behavior of the pet. For example, GPS can only show where the pet is located, but the user cannot know what the pet is doing at that location (playing, fighting, or being trapped); the present solution restores pet behavior (such as "digging", "jumping", "chasing") through event stream + motion data, generates visualized video, and allows the user to intuitively understand the pet's dynamics. SUMMARY

[0009] The technical problem to be solved by the present application is to provide a multi-modal data fusion video generation system capable of realizing efficient and safe scene capture and video generation by using an event camera.

[0010] To solve the above technical problem, the present application provides a video generation system, comprising: an image acquisition device and a background server, wherein the image acquisition device comprises: an event camera module for capturing luminance change events in a scene and generating raw event stream data; a motion sensor module for acquiring motion posture data of the image acquisition device; a processing module for performing real-time privacy protection processing on the raw event stream data and performing data fusion on the motion posture data and the event stream data after privacy protection processing to generate 4D event trajectory data; a communication module for uploading the 4D event trajectory data processed by the processing module to the background server; and a power module for supplying power to the image acquisition device, and the background server uses a video generation AI algorithm to generate a video from the received 4D event trajectory data.

[0011] Preferably, the processing module comprises an event stream preprocessing module, which implements the privacy protection processing, and the privacy protection processing comprises the following steps: a1: converting the raw event stream into a spatio-temporal voxel grid V(x, y, t); a2: simultaneously detecting multiple sensitive targets using a multi-task convolutional neural network (MT-CNN) and generating a sensitive region probability map Mk(x, y, t); a3: applying different degrees of Gaussian blur to the multiple sensitive targets according to a preset sensitive level; and a4: converting the privacy-processed event stream into a spatio-temporal voxel grid with privacy protection markers and outputting it as structured 2D visual data V'(x, y, t).

[0012] Preferably, the motion sensor module is a six-axis IMU that synchronously captures the motion state of the device and outputs raw motion data, and the processing module (12) comprises a motion data preprocessing module (12b) that preprocesses the raw motion data and converts the raw data into higher-order accurate motion data through integral operation.

[0013] Preferably, in the video generation system, Kalman filtering algorithm is used to remove noise in the motion data before preprocessing the raw motion data.

[0014] Preferably, the processing module of the video generation system comprises a data fusion module for fusing the structured 2D visual data output after privacy protection and the accurate motion data into the 4D event trajectory data.

[0015] Preferably, in the video generation system, the video generation AI algorithm uses a multimodal spatiotemporal fusion network to generate video, specifically including: S1: Extracting image features F from 4D event trajectory data. img (t i ) and encode the 4D trajectory data into temporal features F traj S2: Perform frame interpolation on the result of S1 based on bidirectional optical flow; S3: Train and use the result obtained in step S2 to enhance physical rationality using trajectory consistency loss; S4: Optimize video quality on the result of S3; S5: Optimize the output of the result of S4 using adaptive bitrate control.

[0016] Preferably, in the video generation system, the privacy protection processing adopts an event-driven reasoning mechanism, which triggers detection only when the cumulative number of events exceeds a threshold.

[0017] Preferably, in the video generation system, the threshold is 50 times.

[0018] Preferably, in the video generation system, in step a1, the original event stream is converted into a spatiotemporal voxel grid V(x,y,t) using formula (1): Where δ is the Dirac function, (x i ,y i ,t i Let pi ∈ {-1,+1} be the event coordinates, and let pi ∈ {-1,+1} be the event polarity.

[0019] Preferably, in the video generation system, in step a2, a lightweight multi-task convolutional neural network (MT-CNN) is used to simultaneously detect multiple sensitive targets: Input: Voxel grid V(x,y,t) Output: Probability map of sensitive regions Mk(x,y,t), where k∈{face, license plate, signature,...} The network loss function is a weighted multi-task loss, as shown in formula (2): L tota; =∑λ k ·L k +λ bbox ·L bbox ...Formula (2) Where: -L k The cross-entropy loss is for the k-th type of target; -L bbox Smooth L1 loss for bounding box regression; -λ k ,λ bboxwhere w is the weight coefficient, which is optimized by grid search.

[0020] Preferably, in the video generation system, step a3 employs formula (3) for adaptive blur processing: where: - is element-wise multiplication; -G σ1 ,G σ2 are Gaussian kernels with different standard deviations (σ1=2, σ2=5); -M low ,M high correspond to low sensitivity and high sensitivity regions, respectively.

[0021] Preferably, in the video generation system, the loss function of the video generation AI algorithm is represented by formula (4): L total = λ content · L content + λ temporal · L temporal + λ trajectory · L trajectory + λ adv · L adv Formula (4) where: L content : content loss L temporal : temporal smoothness loss L adv : adversarial loss

[0022] Preferably, in the video generation system, the image acquisition device is a wearable device.

[0023] According to another aspect of the present application, a method for generating a video using the above system is provided, which comprises: generating 4D event trajectory data by the image acquisition device and uploading to the background server; and generating a video from the 4D event trajectory data received by the background server using a video generation AI algorithm.

[0024] Preferably, in the video generation method, the processing module comprises an event stream preprocessing module, the event stream preprocessing module implements the privacy protection processing, and the privacy protection processing comprises the following steps: a1, converting an original event stream into a spatio-temporal voxel grid V(x, y, t); a2, simultaneously detecting multiple sensitive targets using a multi-task convolutional neural network (MT-CNN) and generating a sensitive region probability map Mk(x, y, t); a3, applying different degrees of Gaussian blur to the multiple sensitive targets according to a preset sensitive level; and a4, converting the event stream after privacy protection into a spatio-temporal voxel grid with a privacy protection mark and outputting the spatio-temporal voxel grid as structured 2D visual data V'(x, y, t).

[0025] Preferably, in the video generation method, the motion sensor module is a six-axis IMU for synchronously capturing the motion state of a device and outputs original motion data, wherein the processing module comprises a motion data preprocessing module, the motion data preprocessing module pre-processes the original motion data, and converts the original data into higher-order accurate motion data through integral operation.

[0026] Preferably, in the video generation method, before pre-processing the original motion data, a Kalman filtering algorithm is used to remove noise in the motion data.

[0027] Preferably, in the video generation method, the processing module comprises a data fusion module, the data fusion module is used for fusing the structured 2D visual data output after privacy protection and the accurate motion data into the 4D event trajectory data.

[0028] Preferably, in the video generation method, the video generation AI algorithm generates a video using a multi-modal spatio-temporal fusion network, and specifically comprises the following steps: S1, extracting image features F img (t i ) in the 4D event trajectory data and encoding the 4D trajectory data into time sequence features F traj ; S2, performing frame interpolation on the result of S1 based on bidirectional optical flow; S3, training and using the result obtained in step S2 using trajectory consistency loss to enhance physical rationality; S4, performing video quality optimization on the result of S3; and S5, outputting the result of S4 through adaptive bit rate control optimization.

[0029] Preferably, in the video generation method, the image acquisition device is a wearable device.

[0030] The application has the following advantages:

[0031] 1. When an event camera is shooting, a sensitive area can be detected and blurred in real time through an embedded AI model, thereby providing a privacy protection function.

[0032] 2. Combine event camera, motion sensor and high-precision timestamp, generate 4D event trajectory, then upload 4D event trajectory data containing multiple time-stamped photos to cloud server through wireless communication, and AI algorithm in the background can synthesize high-quality video based on 4D data.

[0033] 3. Extremely low power consumption design: the overall power consumption of the image acquisition device is less than 10mW through hardware optimization and software strategy.

[0034] 4. The present application integrates computer vision, wearable devices and Internet of Things technology, realizes the integration of real-time privacy protection, 4D data fusion and low power consumption, can be used for dynamic scene capture and video generation, and can be widely used in motion recording, security monitoring, industrial detection, medical assistance and other fields. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a structural block diagram of the video generation system of the present application. DETAILED DESCRIPTION

[0036] The preferred embodiments of the present application are described below.

[0037] Figure 1 The block diagram of a video generation system according to the preferred embodiment of the present application is shown, which includes an image acquisition device 1 and a background server 2. The image acquisition device 1 may, for example, be an image acquisition device 1 installed on a wearable device (such as a device worn on a pet or a human body). The image acquisition device 1 includes an event camera 10, a motion sensor 11, a processing module 12, a communication module 13, and a power module 14.

[0038] The image acquisition device 1 captures the brightness change events in the scene using the event camera 10, and outputs the original event stream when the change of a certain pixel exceeds the preset threshold, with the data format being (x, y, t, p): Where x: horizontal coordinate (pixel level); y: vertical coordinate (pixel level); t: timestamp (preferably synchronized with the motion sensor 11); p: brightness polarity (+1 indicates brightness increase, -1 indicates brightness decrease).

[0039] The motion sensor 11 includes an IMU module for collecting motion posture data. Preferably, the IMU module is a six-axis IMU (three-axis accelerometer + three-axis gyroscope) that synchronously captures the motion state of the device, outputs "raw motion data" with the format (ax, ay, az, ωx, ωy, ωz, t): Where: ax / ay / az: linear acceleration of X / Y / Z axis (unit: m / s 2 ); ωx / ωy / ωz: angular velocity of rotation about the X / Y / Z axes (unit: ° / s); t: Timestamp synchronized with the event camera.

[0040] The processing module 12 includes an event stream preprocessing module 12a, a motion data preprocessing module 12b, and a data fusion module 12c.

[0041] The event stream preprocessing module 12a is used for sensitive target detection and blurring for privacy processing. Preferably, a noise filtering step can be added before the privacy processing step to filter isolated noise events (such as single abnormal events caused by sensor interference) and retain continuous and valid event clusters. Every time a set value (e.g., 50) of raw events accumulates (approximately 0.1s), a real-time privacy protection algorithm is triggered to detect whether the event stream contains preset sensitive targets. When a sensitive target is detected, it is blurred.

[0042] The privacy protection algorithm in this embodiment adopts a hierarchical detection-fuzzy architecture and includes the following steps.

[0043] Step a1: Event Stream Preprocessing

[0044] The original asynchronous event stream is converted into a spatiotemporal voxel grid using formula (1). Voxel Grid): Where δ is the Dirac function, (x i ,y i ,t i ) represents the event coordinates, p i ∈{-1,+1} represents the event polarity.

[0045] Step a2: Multi-scale sensitive region detection Simultaneous detection of multiple sensitive targets using a lightweight multi-task convolutional neural network (MT-CNN): Input: Voxel grid V(x,y,t) Output: Probability map of sensitive regions Mk(x,y,t), where k∈{face, license plate, signature,...} The network loss function is a weighted multi-task loss, as shown in formula (2): L total =∑λ k ·L k +λ bbox ·L bbox ...Formula (2) Where: -Lk is the cross-entropy loss of the k-th type of target. - Lbbox is the Smooth L1 loss of bounding box regression - λk, λbbox are weight coefficients, optimized by grid search

[0046] Step a3: Adaptive blur processing: where: - is element-wise multiplication - G σ1 - G σ2 are Gaussian kernels with different standard deviations (σ1=2, σ2=5) - M low - M high correspond to low-sensitive (e.g. human silhouette) and high-sensitive (e.g. human face) regions respectively

[0047] In step a3 above, for the detected sensitive regions, adaptive Gaussian blur is performed according to preset levels: high-sensitive targets (e.g. human face): blurred with Gaussian kernel σ=5, completely eliminating facial features; low-sensitive targets (e.g. human silhouette): blurred with Gaussian kernel σ=2, retaining the general outline but not exposing details; non-sensitive regions (e.g. ground, wall): not processed, retaining the original event features.

[0048] Step a4: Structured conversion Convert the event stream after privacy processing into a spatio-temporal voxel grid V'(x, y, t) (with privacy protection markers), output the structured 2D visual data V'(x, y, t) after privacy protection.

[0049] In the real-time privacy protection algorithm of the above embodiment, the MT-CNN model is deployed using TensorFlow Lite Micro, the model is quantized to 8-bit integers, and the size is <2MB. OTA updates of the sensitive target classifier are supported. The specific real-time optimization uses an event-driven inference mechanism, triggering detection only when the cumulative event number exceeds a threshold, for example 50. After actual measurement, the delay on the embedded device is <15ms, which can meet the real-time requirements.

[0050] Using an event-driven inference mechanism, triggering detection only when the cumulative event number exceeds a threshold N_event, the program can be, for example:

[0051] In addition, a timed shooting mechanism can also be set, and the event camera captures a static photo at a configurable time interval.

[0052] Motion data preprocessing module 12b preprocesses the IMU data. Preferably, a Kalman filter algorithm is used to remove noise (such as device vibration, instantaneous interference caused by pet running and jumping) in the motion data. Then, the preprocessing module converts the raw data into higher-order motion parameters through integral operation:

[0053] Step b1: Acceleration first integral → velocity (v); acceleration second integral → displacement (s).

[0054] Step b2: Angular velocity integral → attitude angle (θ, reflecting device tilt, rotation state).

[0055] Step b3: Finally output accurate motion data (v, s, θ, t), including velocity, displacement, attitude angle and corresponding timestamp.

[0056] Data fusion module 12c fuses the privacy-protected structured 2D visual data and accurate motion data into 4D data, with the following steps:

[0057] Step c1: Time-space alignment binding With timestamp t as the only index, the privacy-protected structured 2D visual data V'(x, y, t) and accurate motion data (v, s, θ) at the same time are bound, ensuring that each set of data corresponds to the state of the device at the same time.

[0058] Step c2: Dimension supplement and coordinate calibration Z-axis depth supplement: use the displacement (s) calculated by the IMU to derive the depth information (z-axis) — for example, when the device moves forward, the z-axis displacement accumulates over time, reflecting the front-back distance between the target and the device.

[0059] Step c3: X / Y coordinate calibration: correct the 2D coordinate deviation of the event camera through the attitude angle (θ) of the IMU. If the device is tilted (such as the pet lowering its head), the x / y coordinates are compensated in the opposite direction according to the rotation angle, ensuring that the coordinates reflect the true position of the target, rather than the device's own attitude change.

[0060] Step c4: Finally generate single-time 3D coordinates (x, y, z, t), realizing the conversion from 2D plane to 3D space.

[0061] Step c5: 4D trajectory splicing In time sequence (t1 < t2 <... < tn), all single-time 3D coordinates are concatenated to form a complete 4D event trajectory data T(t), which has the form [(x1, y1, z1, t1), (x2, y2, z2, t2),..., (x n ,y n ,z n ,t n )].

[0062] The communication module 13 is in communication connection with the processing module 12, and uploads the received 4D event trajectory data to the background server 2 through wireless connection.

[0063] The power supply module 14 supplies power to each module of the image acquisition device 1 with low power consumption.

[0064] In order to realize low power consumption, the system of the application preferably adopts the following hardware architecture.

[0065] Special low-power chip selection

[0066] Event camera chip: event sensor static power consumption based on 28nm FD-SOI process: <1mW Dynamic power consumption: 0.5pJ / event Support event-driven pixel-level power gating

[0067] Processor: visual processing unit (VPU) of RISC-V architecture Multi-voltage domain design: core voltage 0.6V-1.0V adjustable clock gating coverage: >95% Deep sleep mode power consumption: 50μW

[0068] Memory: using STT-MRAM non-volatile memory standby power consumption: 0mW (non-volatile characteristics) Read and write energy consumption: 0.1pJ / bit

[0069] Power management integrated circuit (PMIC) The PMIC functional modules include: Multi-channel adjustable LDO: efficiency >92% Dynamic voltage and frequency scaling (DVFS): 32 voltage-frequency combinations Power gating: 8 independent power domains Real-time power consumption monitoring: accuracy ±2%

[0070] The above hardware architecture is only an example, and any other configuration capable of realizing low power consumption can be adopted by those skilled in the art.

[0071] After the background cloud server 2 receives the 4D event trajectory data sent by the device image acquisition device 1 to the background, a video is generated.

[0072] The background AI video generation algorithm adopts a multi-modal spatio-temporal fusion network, and the specific process is as follows:

[0073] Step S1: feature extraction and alignment Image feature extraction: using ResNet-50 (Residual Network) to extract the deep features of each photo, using the following formula (4) F img (t i ) = CNN θ (P(t i ))......Formula (4) in: P(t i ): at time t i Still photos captured by an event camera θ: Weight parameters of the convolutional neural network F img (t i ): From photo P(t) i Image feature vectors extracted from ) Trajectory feature encoding: Encode the 4D trajectory data T(t) into temporal features. in: T(t i ): Time t i Corresponding 4D event trajectory point (x) i ,y i ,z i ,t i ) Weight parameters of LSTM network F traj : Temporal feature vector after encoding the entire trajectory sequence

[0074] Step S2: Spatiotemporal interpolation model

[0075] Optical flow estimation based on bidirectional optical flow: F t→t+1 =FlowNet(P t ,P t+1 )...Formula (6) F t+1→t =FlowNet(P t+2 ,P3)...Formula (7)

[0076] Intermediate frame generation: P τ =α τ ·warp(P t ,F t→τ )+(1-α τ )·warp(P t+1 ,F t+1→τ )+β·CNN refine (P t ,P t+1 ,τ) ...Formula (8) wherein: α τ : time weight coefficient, based on relative position of τ between t and t+1 β: weight coefficient of refinement network CNN refine : convolutional neural network for refining quality of intermediate frames warp() is an optical flow based image warping operation.

[0077] Step S3: Video quality optimization

[0078] The total loss function is defined by equation (9): L total = λ content · L content + λ temporal · L temporal + λ trajectory · L trajectory + λ adv · L adv … equation (9) wherein: - L content : content loss (perceptual similarity) - L temporal : temporal smoothness loss - L adv : adversarial loss (using discriminator to boost realism)

[0079] Step S4: Adaptive bitrate control

[0080] The output video quality is dynamically adjusted according to network conditions, using equation (10): bitrate = f(network_bandwidth, user_preference, content_complexity) … equation (10)

[0081] In addition, during the execution of steps S1, S2 and S3, motion consistency enhancement is simultaneously performed, which is a training stage constraint, boosting physical plausibility.

[0082] Specifically, step S5: Introducing trajectory consistency loss to ensure physical plausibility:

[0083] Based on 4D event trajectory data, the background AI algorithm can generate high-quality videos, and through optical flow interpolation and trajectory consistency optimization, discrete event streams are converted into smooth video (such as 30fps) output. Users can generate videos in real time or review them.

[0084] In addition, based on the output video, the background AI algorithm can also perform behavior recognition according to different application scenarios, analyze trajectory features, and identify target behaviors, such as running, jumping, and struggling of pets, abnormal movement of mechanical arms, or others.

[0085] In addition, based on the output video, the background AI algorithm can also perform scene restoration and early warning: restore a 3D scene, or trigger real-time early warning when a dangerous trajectory is detected, such as a pet entering a road.

[0086] The preferred embodiments of the present application are described in detail above. It should be understood that those of ordinary skill in the art can make many modifications and changes without creative labor based on the concept of the present application. Therefore, any technical solutions obtained by logical analysis, reasoning or limited experiments based on the existing technology within the scope of the concept of the present application should be within the protection scope determined by the claims.

Claims

1. A video generation system, characterized in that, include: Image acquisition device (1) and backend server (2), The image acquisition device includes: an event camera module (10) for capturing brightness change events in the scene and generating raw event stream data; a motion sensor module (11) for acquiring motion posture data of the image acquisition device; a processing module (12) for performing real-time privacy protection processing on the raw event stream data and fusing the motion posture data with the privacy-protected event stream data to generate 4D event trajectory data; a communication module (13) for uploading the 4D event trajectory data processed by the processing module (12) to the backend server; and a power module (14) for supplying power to the image acquisition device, and The backend server (2) uses a video generation AI algorithm to generate a video from the received 4D event trajectory data.

2. The video generation system as described in claim 1, characterized in that, The processing module (12) includes an event stream preprocessing module (12a), which implements the privacy protection process, which includes the following steps: a1: Convert the original event stream into a spatiotemporal voxel grid V(x,y,t); a2: Use a multi-task convolutional neural network (MT-CNN) to detect multiple sensitive targets simultaneously and generate a probability map of sensitive regions Mk(x,y,t); a3: Apply different degrees of Gaussian blurring to the various sensitive targets according to the preset sensitivity level; a4: Convert the privacy-processed event stream into a spatiotemporal voxel grid with privacy-preserving tags and output it as structured 2D visual data V'(x,y,t).

3. The video generation system as described in claim 1 or 2, characterized in that, The motion sensor module (11) is a six-axis IMU that synchronously captures the motion state of the device and outputs raw motion data. The processing module (12) includes a motion data preprocessing module (12b), which preprocesses the raw motion data and converts the raw data into higher-order, more accurate motion data through integration.

4. The video generation system as described in claim 3, characterized in that, Before preprocessing the raw motion data, noise in the motion data is removed using the Kalman filter algorithm.

5. The video generation system as described in claim 3, characterized in that, The processing module (12) includes a data fusion module (12c), which is used to fuse the structured 2D visual data output after privacy protection with the precise motion data into the 4D event trajectory data.

6. The video generation system as described in claim 5, characterized in that, The video generation AI algorithm uses a multimodal spatiotemporal fusion network to generate videos, specifically including: S1: Extract image features F from 4D event trajectory data img (t i ) and encode the 4D trajectory data into temporal features F traj ; S2: Frame interpolation is performed on the result of S1 based on bidirectional optical flow; S3: Use the trajectory consistency loss to enhance physical plausibility based on the results obtained in step S2 for training and application; S4: Optimize the video quality of the results from S3; S5: Optimize the output of the result of S4 through adaptive bit rate control.

7. The video generation system as described in claim 2, characterized in that, The privacy protection process employs an event-driven reasoning mechanism, triggering detection only when the cumulative number of events exceeds a threshold.

8. The video generation system as described in claim 7, characterized in that, The threshold is 50 times.

9. The video generation system as described in claim 2, characterized in that, In step a1, the original event stream is converted into a spatiotemporal voxel grid V(x,y,t) using formula (1): Where δ is the Dirac function, (xi,yi,ti) are the event coordinates, and pi∈{-1,+1} is the event polarity.

10. The video generation system as described in claim 9, characterized in that, In step a2, a lightweight multi-task convolutional neural network (MT-CNN) is used to detect multiple sensitive targets simultaneously: Input: Voxel grid V(x,y,t) Output: Probability map of sensitive regions Mk(x,y,t), where k∈{face, license plate, signature,...} The network loss function is a weighted multi-task loss, as shown in formula (2): L total =∑λ k ·L k +λ bbox ·L bbox ...Formula (2) Where: -L k The cross-entropy loss is for the k-th type of target; -L bbox Smooth L1 loss for bounding box regression; -λ k ,λ bbox The weighting coefficients are optimized using a grid search.

11. The video generation system as described in claim 10, characterized in that, Step a3 uses formula (3) for adaptive fuzzy processing: Where: -⊙ represents element-wise multiplication; -G σ1 G σ2 Gaussian kernels with different standard deviations (σ1=2, σ2=5); -M low M high These correspond to the low-sensitivity and high-sensitivity regions, respectively.

12. The video generation system as described in claim 6, characterized in that, The loss function of the video generation AI algorithm is expressed by formula (4): L total =λ content ·L content +λ temporal ·L temporal +λ trajectory ·L trajectory +λ adv ·L adv ...Formula (4) in: -L content Content loss -L temporal Temporal smoothness loss -L adv : To combat losses.

13. The video generation system as described in claim 1, characterized in that, The image acquisition device (1) is a wearable device.

14. A method for generating video using the system of claim 1, characterized in that, include: The image acquisition device (1) generates 4D event trajectory data and uploads it to the backend server (2); as well as The backend server (2) uses a video generation AI algorithm to generate a video from the received 4D event trajectory data.

15. The method for generating video according to claim 14, characterized in that, The processing module (12) includes an event stream preprocessing module (12a), which implements the privacy protection process, which includes the following steps: a1: Convert the original event stream into a spatiotemporal voxel grid V(x,y,t); a2: Use a multi-task convolutional neural network (MT-CNN) to detect multiple sensitive targets simultaneously and generate a probability map of sensitive regions Mk(x,y,t); a3: Apply different degrees of Gaussian blurring to the various sensitive targets according to the preset sensitivity level; a4: Convert the privacy-processed event stream into a spatiotemporal voxel grid with privacy-preserving tags and output it as structured 2D visual data V'(x,y,t).

16. The method for generating video as described in claim 14 or 15, characterized in that, The motion sensor module (11) is a six-axis IMU that synchronously captures the motion state of the device and outputs raw motion data. The processing module (12) includes a motion data preprocessing module (12b). The motion data preprocessing module (12b) preprocesses the raw motion data and converts the raw data into higher-order, more accurate motion data through integral calculation.

17. The method for generating video as described in claim 16, characterized in that, Before preprocessing the raw motion data, noise in the motion data is removed using the Kalman filter algorithm.

18. The method for generating video as described in claim 16, characterized in that, The processing module (12) includes a data fusion module (12c), which is used to fuse the structured 2D visual data output after privacy protection with the precise motion data into the 4D event trajectory data.

19. The method for generating video as described in claim 14, characterized in that, The video generation AI algorithm uses a multimodal spatiotemporal fusion network to generate videos, specifically including the following steps: S1: Extract image features F from 4D event trajectory data img (t i ) and encode the 4D trajectory data into temporal features F traj ; S2: Frame interpolation is performed on the result of S1 based on bidirectional optical flow; S3: Use the trajectory consistency loss to enhance physical plausibility based on the results obtained in step S2 for training and application; S4: Optimize the video quality of the results from S3; S5: Optimize the output of the result of S4 through adaptive bit rate control.

20. The method for generating video as described in claim 14, characterized in that, The image acquisition device (1) is a wearable device.

Images