A large model-oriented unmanned aerial vehicle video real-time analysis method and system

Abstract

The application discloses a large model-oriented unmanned aerial vehicle video real-time analysis method and system. The method comprises the following steps: deploying a lightweight convolutional neural network backbone model on the unmanned aerial vehicle side, collecting original video sequences in real time and extracting high-dimensional feature vectors; calculating the semantic entropy of the current video frame based on the high-dimensional feature vector, and representing the semantic offset degree by quantifying the distance measurement of the current frame and the dynamically updated reference key frame in the feature space; comparing the semantic entropy with the dynamic judgment threshold, and triggering the video slice interception logic and encoding upload to the cloud when the threshold is exceeded; the multi-modal large language model deployed on the cloud performs deep semantic analysis on the video slice, generates feedback instructions according to the analysis result, and adjusts the data acquisition behavior on the side through the downlink. Through the semantic entropy-driven adaptive triggering mechanism and the end-cloud collaborative closed-loop control, the application reduces the redundant data transmission, and improves the real-time performance and accuracy of the unmanned aerial vehicle video analysis in complex scenes.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, specifically relating to a method and system for real-time analysis of drone video for large models. Background Technology

[0002] With the deep integration of drone technology and artificial intelligence, real-time video analytics has become a key means to enhance perception capabilities in fields such as traffic inspection, emergency rescue, and environmental monitoring. Against the backdrop of the evolution of large-scale model technology, visual data acquired through drone payloads is no longer limited to basic image recording, but provides core support for automated decision-making through deep semantic analysis. Especially in dynamic and complex inspection tasks, the system's processing efficiency of video streams and the depth of scene understanding directly affect the real-time performance and accuracy of task execution, which sets higher standards for the synergy between video transmission links and intelligent analysis algorithms.

[0003] Among them, the video real-time analysis technology for large models focuses on combining lightweight perception on the UAV edge with deep semantic reasoning in the cloud, aiming to build an efficient data transmission and closed-loop response mechanism. This technical direction requires the system to perform accurate feature recognition and information filtering on the raw video stream while ensuring real-time communication. By performing preliminary feature extraction on the edge, it ensures that the data transmitted back to the large model in the backend has high semantic relevance and task value, thereby achieving a deep understanding of complex scenes.

[0004] However, traditional UAV video processing solutions often employ a fixed frame rate full-data transmission strategy, which imposes a huge bandwidth load on wireless communication links in high-resolution scenarios and is highly susceptible to severe analysis delays due to fluctuations in link quality. Simultaneously, edge motion detection technologies based on pixel differences struggle to adapt to background motion and aircraft shake generated during UAV flight, leading to frequent false triggers and excessive redundant data consuming computational resources. Furthermore, existing preprocessing methods lack dynamic quantitative evaluation of the semantic information contained within video frames, failing to accurately capture key state changes in complex environments. This results in data noise interference and inference feedback lag in the backend large models, making it difficult to meet the demands of high-precision real-time tasks.

[0005] Therefore, a real-time analysis method for UAV videos targeting large models is desired. Summary of the Invention

[0006] The purpose of this invention is to provide a method and system for real-time analysis of UAV video for large models, which can effectively solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: Firstly, a real-time video analysis method for large-scale drone models includes the following steps: Step S1: The lightweight convolutional neural network backbone model deployed on the UAV end acquires the original video sequence in real time and extracts a high-dimensional feature vector for each frame of the image. The high-dimensional feature vector is used to characterize the global deep semantic features of the current video frame. Step S2: Calculate the semantic entropy of the current video frame based on the high-dimensional feature vector. The semantic entropy is used to characterize the semantic offset of the current video frame relative to the historical keyframe by quantifying the distance between the current video frame and the dynamically updated reference keyframe in the feature space. Step S3: Compare the semantic entropy with the dynamic judgment threshold. When the semantic entropy exceeds the dynamic judgment threshold, generate a trigger signal, start the video slice capture logic, and use the encoder to encode the captured video slices and upload them to the cloud server. Step S3 further includes a step of adaptively adjusting the dynamic judgment threshold according to the real-time status of the wireless communication link, and a step of preheating the encoder in advance based on the prediction results of the semantic entropy change trend. In step S4, the multimodal large language model deployed on the cloud server performs deep semantic analysis on the uploaded video slices and generates feedback instructions based on the analysis results. The feedback instructions are sent to the drone end via the downlink to adjust the data collection behavior of the end.

[0008] Preferably, step S1 specifically includes: Before inputting the raw video sequences acquired by the airborne vision sensor into the lightweight convolutional neural network backbone model, image preprocessing is performed, which includes size scaling and normalization. The lightweight convolutional neural network backbone model adopts a structure based on depthwise separable convolution and outputs high-dimensional feature vectors through global average pooling layers. Step S1 also includes integrating noise reduction measures during the extraction of high-dimensional feature vectors. These measures include performing brightness histogram equalization or Gamma correction on the image to reduce interference from illumination changes, and performing high-pass filtering or exponentially weighted moving average filtering on the initial feature vectors to eliminate high-frequency feature noise caused by camera shake.

[0009] Preferably, in step S2, semantic entropy is obtained by calculating the Euclidean distance or cosine similarity between the high-dimensional feature vector of the current frame and the high-dimensional feature vector of the reference keyframe. The reference keyframe is initialized as the feature vector of the first frame image at the beginning of system startup, and during subsequent operation, whenever step S3 determines that the video slice upload is triggered, the high-dimensional feature vector of the current frame is updated to the new reference keyframe. Step S2 also includes a step of normalizing the calculated semantic entropy to map it to a preset numerical range.

[0010] Preferably, in step S3, adaptively adjusting the dynamic judgment threshold based on the real-time status of the wireless communication link specifically includes: Set a preset initial value for the dynamic determination threshold; Real-time monitoring of bandwidth utilization, signal-to-noise ratio, or data packet loss rate of wireless communication links; When link congestion or signal attenuation is detected, the dynamic judgment threshold is automatically increased by a preset adjustment ratio to reduce unnecessary data uploads when the link quality is poor.

[0011] Preferably, preheating the encoder based on the prediction results of semantic entropy change trends specifically includes: Continuously record semantic entropy values ​​over multiple periods to form a semantic entropy change trajectory; Based on the semantic entropy change trajectory, the semantic shift trend of the next period is predicted using linear regression or Kalman filtering algorithms. If the prediction results show that the semantic entropy continues to increase, the encoder will be warmed up in advance. The warm-up operation includes pre-allocating the buffer resources required for encoding, pre-loading the encoder firmware parameters, and switching the encoder from a low-power standby state to a working ready state.

[0012] Preferably, in step S3, the video slicing logic is implemented based on a circular buffer mechanism. When a trigger signal is generated, the historical frames before the trigger time and the subsequent frames after the trigger time are read from the circular buffer at the same time. The total length of the video slice is dynamically adjusted based on the target's moving speed detected in real time by the speed sensor on the drone. When the target's moving speed is detected to be greater than a preset speed threshold, the slice length is automatically extended from the first preset length to the second preset length.

[0013] Preferably, in step S3, when encoding the captured video slice, the current multidimensional state information of the UAV is embedded in the encapsulation header or supplementary enhancement information field. The multidimensional state information includes GPS coordinates, flight altitude, flight speed, gimbal angle and UAV attitude angle, and is transmitted to the cloud along with the video slice.

[0014] Preferably, in step S4, after the multimodal large language model performs deep semantic analysis on the uploaded video slices, it also executes a continuous attention value assessment logic. The value assessment logic is as follows: If the probability of a multimodal large language model detecting a task-related target in a video slice is greater than a preset probability threshold, then the scene is determined to have high sustained attention value. The feedback instruction includes a correction coefficient for the dynamic judgment threshold. When a high sustained attention value is determined, the correction coefficient is used to proportionally lower the dynamic judgment threshold on the UAV side, enabling the system to transmit subsequent video sequences with higher sensitivity.

[0015] Preferably, in step S4, after completing the video slice analysis, the multimodal large language model also generates adjustment suggestions for the parameters of the airborne vision sensor based on the image quality analysis. The adjustment suggestions are encapsulated as feedback instructions and sent to the drone via the downlink. The adjustment suggestions include increasing exposure compensation, increasing sensor gain, enabling wide dynamic range mode, or adjusting contrast parameters. After receiving feedback instructions, the drone automatically adjusts the corresponding sensor parameters.

[0016] Secondly, a real-time video analysis system for large-scale drones, used to perform the above methods, includes: On the drone's edge, a lightweight convolutional neural network backbone model is deployed to acquire raw video sequences in real time and extract high-dimensional feature vectors for each frame of image; The semantic entropy discrimination module is used to calculate the semantic entropy of the current video frame based on the high-dimensional feature vector. The semantic entropy is a quantification of the distance between the current video frame and the dynamically updated reference keyframe in the feature space. The video slice extraction module compares the semantic entropy with a dynamic judgment threshold. When the semantic entropy exceeds the dynamic judgment threshold, the video slice extraction logic is triggered, and the extracted video slice is encoded. The threshold adaptive control module is used to adaptively adjust the dynamic judgment threshold according to the real-time status of the wireless communication link. The preheating module is used to preheat the encoder in advance based on the prediction results of the semantic entropy change trend; The cloud server is equipped with a multimodal large language model to perform deep semantic analysis on the uploaded video slices and generate feedback instructions based on the analysis results. The feedback instructions are sent to the drone end via the downlink to adjust the data collection behavior of the end. The wireless communication module is used to upload the encoded video slices to the cloud server and receive feedback instructions from the cloud.

[0017] In summary, this application includes at least one of the following beneficial technical effects: 1. This application introduces semantic entropy as a quantitative indicator of video content changes, and measures the distance between the high-dimensional feature vector extracted from the terminal and the dynamically updated reference keyframe. Video slices are only uploaded when the semantic entropy exceeds the dynamic judgment threshold. Compared with the traditional fixed frame rate full transmission scheme, this can effectively avoid redundant data transmission caused by link quality fluctuations and reduce the bandwidth utilization of wireless communication links.

[0018] 2. This application establishes a dynamically updated semantic benchmark and state prediction algorithm. When the system determines that a video slice upload is triggered, the current frame is updated to a new reference key frame. Based on linear regression or Kalman filtering, the semantic entropy change trend is predicted, and the encoder is preheated in advance. This enables the system to accurately capture key semantic changes while shortening the response latency and achieving millisecond-level event capture capability.

[0019] 3. This application utilizes an intelligent closed-loop control mechanism that integrates edge and cloud technologies. After the cloud-based multimodal large language model performs deep semantic analysis on the uploaded video slices, it generates feedback instructions containing dynamic judgment threshold correction coefficients based on the analysis results. This lowers the threshold on the edge side to improve sampling sensitivity. Simultaneously, it remotely adjusts the parameters of the airborne vision sensor based on image quality analysis, forming a global optimization closed loop from image acquisition to feature extraction to cloud analysis. This enhances the system's adaptability and analysis accuracy in complex task scenarios. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical solution architecture of the real-time video analysis method for large-scale UAVs of the present invention; Figure 2 This is a schematic diagram of the core principle framework of the present invention for semantic entropy discrimination and triggering based on semantic offset degree quantification; Figure 3 This is a flowchart illustrating the logical process of end-side feature extraction and circular buffer video slice generation in this invention. Figure 4 This is a flowchart illustrating the dynamic threshold correction logic of the present invention based on cloud-based multimodal large model value assessment. Detailed Implementation

[0021] The following will be combined with the appendix Figures 1 to 4 The technical solution of the present invention is clearly and completely described below. Obviously, the following embodiments are merely preferred embodiments of the present invention and are not intended to limit the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0022] The real-time UAV video analysis method for large models provided in this application achieves semantic-level compression and intelligent analysis of high-dimensional video data through an edge-cloud collaborative architecture. This method is built upon a bidirectional data interaction link between the UAV edge and a cloud server. The UAV edge is a computing platform deployed on the UAV, used to perform edge-side processing, specifically including edge-side feature extraction, semantic entropy discrimination, triggering and encoding, and cloud-based inference and feedback. The following sections will elaborate on each step.

[0023] First, the edge feature extraction process in step S1 is explained, which transforms the raw video sequence acquired by the airborne vision sensor into a high-dimensional feature representation that can be used for subsequent semantic discrimination. Specifically, it includes the following steps: Step S101: Acquire the original video sequence. The UAV terminal receives the original video sequence acquired by the airborne vision sensor in real time. The airborne vision sensor can be an industrial-grade CMOS image sensor or an infrared thermal imaging sensor.

[0024] To ensure a balance between image quality and data processing efficiency, the acquisition resolution of the original video sequence can be set to typical values ​​such as 1920×1080, 2560×1440, or 3840×2160. The acquisition frame rate can be set to 25fps, 30fps, or 60fps depending on the specific task requirements, and the image bit depth can be set to 8bit, 10bit, or 12bit.

[0025] Step S102: Perform image preprocessing. Before inputting the original image into the lightweight convolutional neural network backbone model, the system calls the image preprocessing operator to scale the original image to a target size such as 224×224, 299×299, or 320×320.

[0026] At the same time, the system will normalize the image matrix according to the preset mean vector and standard deviation vector, mapping the pixel values ​​to the range of [-1, 1] or [0, 1].

[0027] Step S103: Deploy a lightweight convolutional neural network backbone model. This model is deployed in the hardware acceleration unit on the UAV side. The hardware acceleration unit can be a neural network processing unit (NPU), a tensor processing unit (TPU), or an integrated graphics processing unit (GPU).

[0028] The lightweight convolutional neural network backbone model specifically adopts a neural network based on a depthwise separable convolutional structure, such as MobileNetV3, ShuffleNetV2, or GhostNet. This lightweight convolutional neural network backbone model removes the traditional fully connected layers and instead uses a global average pooling layer to reduce the spatial dimension of the last layer feature map, ultimately outputting a feature vector with dimensions of 1024, 1280, or 512. This feature vector has high sparsity and representativeness in the feature space and is used to characterize the global deep semantic features of the current video frame, i.e., a high-dimensional feature vector.

[0029] To meet real-time requirements, the feature extraction time for a single frame image is controlled within 10ms, 15ms, or 20ms, and the computing resource utilization of the edge processor is kept below 60%, 70%, or 80%.

[0030] Step S104 involves noise suppression and feature filtering. To improve the stability of feature extraction and reduce the interference of environmental factors on semantic judgment, this method integrates multiple anti-noise measures in the feature extraction layer.

[0031] To address visual noise caused by changes in lighting, cloud cover, or the angle of the sun, the system performs brightness histogram equalization preprocessing or introduces a Gamma correction mechanism before feature extraction, thereby reducing feature fluctuations caused by non-semantic factors.

[0032] In addition, after obtaining the initial feature vector, the system will also perform an environmental noise filtering step. By performing high-pass filtering or exponentially weighted moving average filtering on the feature vector, high-frequency noise in the feature space caused by the slight shaking of the drone body or the thermal noise of the sensor is eliminated, ensuring that the semantic entropy fluctuations calculated subsequently can truly reflect the substantial changes in the semantic content of the video, rather than being subject to external interference.

[0033] In summary, the above-described process for extracting edge features involves standardizing the original video frames, using lightweight neural network inference, and employing targeted noise suppression to ultimately generate a global feature vector that stably and accurately reflects the core semantics of the current scene.

[0034] The semantic entropy discrimination process in step S2 will be explained next. This process quantifies the degree of difference between the current video frame and the reference frame at the semantic level, providing a basis for decision-making on whether to trigger video upload.

[0035] Step S201: Define semantic entropy. The system quantifies the semantic offset of the current image relative to historical images by calculating the distance between the feature vector extracted at the current moment and the feature vector of the preset reference keyframe in the feature space.

[0036] It should be noted that the degree of semantic offset is defined as semantic entropy in this method. The larger the value of semantic entropy, the more significant the change in semantic content of the current frame compared to the reference frame.

[0037] Step S202: Establish a dynamically updated semantic baseline. The reference keyframe feature vector is initialized as the feature vector of the first frame image at the initial stage of system startup. During subsequent operation, whenever the system determines that a video slice upload action needs to be triggered, the feature vector of the current frame will be synchronously updated to the new reference keyframe feature vector.

[0038] In this way, the system establishes a semantic benchmark that can be dynamically adjusted as events occur, ensuring that subsequent semantic entropy calculations are always based on the latest key images.

[0039] The steps S201 and S202 mentioned above together form the basis of semantic entropy discrimination. Step S201 clarifies the physical meaning of semantic entropy, while step S202 specifies the initialization and update rules of the reference benchmark. The combination of the two enables semantic entropy to dynamically reflect the relative changes of semantic content in the video stream, rather than making an absolute comparison with a fixed initial state.

[0040] Step S203: Select and calculate the distance metric. The system provides two optional metric methods to measure the difference between feature vectors.

[0041] The first method uses Euclidean distance, and its calculation formula is as follows: in, This represents the feature vector extracted at the current time. This represents the feature vector of the reference keyframe. The total dimension of the feature vectors. This represents the value of the i-th dimension component of the feature vector at the current time. This represents the component value of the reference keyframe feature vector in the i-th dimension. It is calculated using this formula. The value is semantic entropy.

[0042] The second method uses cosine similarity, which measures the semantic similarity between the current feature vector and the reference keyframe feature vector by calculating the cosine of the angle in the multidimensional feature space. The formula for calculating cosine similarity is: In this formula, the numerator is the dot product of two vectors, and the denominator is the product of the magnitudes of the two vectors.

[0043] The calculated cosine similarity value range is [-1, 1]. The closer the value is to 1, the more consistent the two vectors are in the direction, that is, the higher the semantic similarity. If cosine similarity is used as a measure of semantic entropy, the system needs to convert the similarity value into a difference measure so that the semantic entropy is positively correlated with the degree of semantic change. The conversion method is that the semantic entropy is equal to 1 minus the cosine similarity.

[0044] Step S204: Perform numerical normalization processing. In order to facilitate the unified comparison with the dynamic judgment threshold in the future, the system performs normalization processing on the calculated semantic entropy value.

[0045] Normalization can be achieved using Min-Max normalization, which maps semantic entropy to a preset numerical range such as [0, 1] or [0, 100]. The formula for calculating Min-Max normalization is as follows: in, This is the original semantic entropy value. and These are the minimum and maximum values ​​of historical semantic entropy, respectively. and These are the lower and upper bounds of the target interval.

[0046] Normalization can also be achieved using Z-score standardization, which converts semantic entropy into a standard normal distribution with a mean of 0 and a standard deviation of 1. The formula for calculating Z-score standardization is: Where μ is the mean of historical semantic entropy, and σ is the standard deviation of historical semantic entropy.

[0047] The normalized semantic entropy value eliminates the influence of dimensions, making the threshold setting more universal and stable.

[0048] Through step S2 above, the abstract semantic difference is quantified into a comparable and threshold-determinable value. This value not only reflects the degree of semantic change of the current image relative to historical keyframes, but also adapts to the subsequent triggering decision logic through normalization processing.

[0049] The triggering and encoding process of step S3 will be explained next. This process dynamically determines whether to trigger the video slice upload based on the semantic entropy comparison results, and performs efficient encoding and information encapsulation on the video data after the trigger.

[0050] Step S301: Determine the trigger condition. The system compares the semantic entropy value calculated in step S2 with the internally set dynamic judgment threshold. If the semantic entropy value exceeds the dynamic judgment threshold, the system determines that a new semantic target has appeared in the current video frame, or the behavior of the original target has changed significantly, or the entire scene has changed abruptly. This determination result serves as the trigger signal to start the subsequent video slicing process.

[0051] Step S302: Implement adaptive adjustment of the dynamic judgment threshold. In order to adapt to changes in the communication link status, the dynamic judgment threshold has the ability to adaptively adjust.

[0052] The system first sets a preset initial value for the threshold. During operation, the system monitors the bandwidth utilization, signal-to-noise ratio (SNR), and data packet loss rate of the current wireless communication link in real time. When link congestion is detected (i.e., bandwidth utilization exceeds 85%), or signal attenuation is detected (i.e., SNR is lower than the preset threshold), the system's control logic will automatically increase the current dynamic judgment threshold according to the preset adjustment ratio.

[0053] The adjustment ratio can be 10%, 20%, or 30%. By raising the trigger threshold, the system can effectively reduce unnecessary data uploads when the link quality is poor, thereby avoiding further exacerbation of link congestion.

[0054] The steps S301 and S302 described above together form the basis for trigger decision and adaptive control. S301 clarifies the basis and result of the trigger determination, while S302 specifies how the threshold is dynamically adjusted according to the external link status. The combination of the two enables the system to adaptively adjust the data transmission frequency according to the network environment while ensuring that no critical events are missed.

[0055] Step S303: Start the video slicing and capturing logic. When the triggering condition is met, the system immediately starts the video slicing and capturing logic. This logic is implemented based on the circular buffer mechanism. The circular buffer is composed of a circular queue in memory, which can store a preset number of original image historical frames before the current time point in real time. The preset number can be 30 frames, 60 frames or 90 frames.

[0056] When a trigger signal is generated, the airborne video encoder performs two reading operations simultaneously: first, it reads the historical frames before the trigger moment from the circular buffer, and second, it captures a preset number of subsequent frames after the trigger moment in real time. This preset number can be 30 frames, 60 frames, or 90 frames.

[0057] This two-way capture method allows the uploaded video clips to fully contain the cause of the event, the process of the event, and the preliminary results of the event.

[0058] Step S304: Dynamically adjust the slice length. To further adapt to the target motion characteristics in different scenarios, the total length of the video slice can be dynamically adjusted according to the target motion speed.

[0059] The drone's onboard sensors detect the target's moving speed in real time. When the detected target's moving speed exceeds a preset speed threshold, the system will automatically extend the slice length from a first preset length to a second preset length. The preset speed threshold can be set to 10m / s or 15m / s, the first preset length can be set to 2 seconds, and the second preset length can be set to 5 seconds.

[0060] The above adjustment mechanism ensures that for fast-moving targets, the slice can completely capture their movement trajectory, avoiding the loss of key process information due to slices that are too short.

[0061] The above step S303 ensures the integrity of the slice through a circular buffer and bidirectional truncation, while step S304 dynamically adjusts the slice duration according to the target speed. The combination of the two enables the system to generate video slices with complete information and appropriate length for scenes with different dynamic characteristics.

[0062] Step S305: Perform high-efficiency video encoding compression. For the captured video slices, the system uses high-efficiency video encoding compression technology for processing. This technology is based on block motion compensation transform coding and supports H.265 / HEVC or H.266 / VVC encoding standards.

[0063] During the compression process, the encoder limits the compression bitrate of video slices to below a preset bitrate upper limit, which can be 1Mbps, 2Mbps or 5Mbps. By controlling the bitrate upper limit, the system can minimize the amount of data transmitted while ensuring usable image quality.

[0064] Step S306: Embed multi-source metadata. During the video slicing and encapsulation process, the system embeds the current multi-dimensional status information of the UAV in the encapsulation header or the SEI field of the supplementary enhancement information.

[0065] The SEI field, or Supplemental Enhancement Information field, is a field in video coding standards used to carry additional information. Multidimensional state information includes GPS coordinates, flight altitude, flight speed, three-axis gimbal angle, UAV attitude angle, and precise timestamps. This metadata is transmitted to the cloud along with video slices, providing crucial auxiliary references for spatial reasoning and target localization in large cloud models.

[0066] The above step S3 achieves a complete closed loop from semantic change triggering to adaptive threshold adjustment, and then to slice extraction and efficient encoding. This ensures that data transmission is only initiated when there is a substantial semantic change in the video content. Furthermore, the transmitted video slices contain complete event context and have undergone efficient compression and metadata enhancement, providing high-quality and efficient data input for the deep semantic analysis of the cloud-based large model in step S4.

[0067] The cloud-based reasoning and feedback process in step S4 will be explained next. This process uses a multimodal large language model deployed in the cloud to perform deep semantic analysis on the uploaded video slices and generates feedback instructions based on the analysis results to achieve intelligent closed-loop control of end-cloud collaboration.

[0068] Step S401: Establish a wireless communication link and upload data.

[0069] The drone sends the video slices encapsulated in step S3 to the cloud server via a wireless communication link. This wireless communication link can use 4G communication technology, 5G communication technology, or satellite communication technology.

[0070] Through the above communication methods, a reliable data transmission channel is established between the terminal and the cloud.

[0071] Step S402: Deploy a cloud-based multimodal large language model. The cloud server is equipped with a high-performance multimodal large language model (VLM), which has the ability to simultaneously process image sequences, video streams, text prompts, and sensor metadata.

[0072] To meet the analysis accuracy requirements in different scenarios, the parameter scale of this model can be set to 7B, 13B, 34B or higher. The specific structure of the model can adopt existing multimodal large language models, such as CLIP, Flamingo or LLaVA. Its input is a video frame sequence and text prompts, and the output is the semantic analysis results and confidence scores of the corresponding prompts.

[0073] Step S403: Set guidance prompt information. The cloud system pre-sets guidance prompt information Prompt according to the current task type to guide the multimodal large language model to focus on specific analysis objectives.

[0074] In power line inspection scenarios, guidance messages can be set as "Please analyze the video clips to see if there are any damaged insulators, hanging objects on the wires, or rusted towers, and assess the risk level"; in traffic monitoring scenarios, guidance messages can be set as "Please identify the length of the vehicle queue and the number of abnormally stopped vehicles in the video to predict traffic congestion trends."

[0075] By setting differentiated prompts for different task types, the inference results of the large model can be made more closely aligned with actual business needs.

[0076] Step S402 above clarifies the core model on which cloud inference relies and its capability boundaries, while step S403 specifies how to guide the model to focus on specific analysis tasks through prompts. The combination of the two ensures the relevance and effectiveness of cloud inference.

[0077] Step S404: Perform deep semantic reasoning and value assessment. After receiving the video slices, the multimodal large language model performs deep semantic scanning in conjunction with the guidance prompts to identify specific categories of targets appearing in the video. These targets include specific models of vehicles, people with abnormal behaviors such as falling or running, fire smoke, and environmental disaster risks.

[0078] The cloud-based inference process not only outputs specific recognition results, but also executes a continuous attention value assessment logic. Specifically, if the probability of the multimodal large language model detecting a task-related target in a video slice is greater than a preset probability threshold, then the scene is determined to have high continuous attention value.

[0079] The probability threshold is preset based on the task's tolerance for false positives and false negatives, and can be set to 0.85, 0.9, or 0.95.

[0080] The probability values ​​output by a multimodal large language model originate from its classification head or confidence estimation module, which is the standard output format of multimodal large language models in the prior art.

[0081] Step S405: Generate and send feedback instructions.

[0082] When step S404 determines that the current scene has high continuous attention value, the cloud server generates a corresponding feedback instruction and sends it to the drone end via the downlink. This feedback instruction contains a correction coefficient for the dynamic judgment threshold.

[0083] If the value is determined to be of high sustained interest, the correction coefficient is a preset proportional coefficient used to proportionally lower the dynamic judgment threshold on the end side. That is, the new threshold is equal to the original threshold multiplied by the coefficient. The coefficient can be set to 0.5, 0.7 or 0.8.

[0084] After receiving the feedback command, the drone terminal completes the threshold update operation in the next frame image processing cycle, enabling the system to transmit subsequent video sequences with higher sensitivity and sampling frequency, thus establishing an intelligent closed-loop sampling mechanism for end-cloud collaboration.

[0085] Step S404 above achieves deep semantic understanding and value judgment of video slices through a large model. Step S405 generates specific control commands based on the judgment results, so that the sampling behavior on the end side can be dynamically adjusted in real time according to the analysis results in the cloud.

[0086] Step S406: Establish an inspection result database. The cloud server establishes an inspection result database to associate and store the structured information generated by each large model inference. The stored information includes structured semantic descriptions, corresponding video slice indexes, geographical location information, and task status.

[0087] When the cumulative number of stored records in the database exceeds the preset storage threshold, the system supports the automatic generation of periodic inspection reports and can perform quantitative comparative analysis of the situation based on historical data. The preset storage threshold can be 10,000, 50,000, or 100,000 records.

[0088] Inspection reports can be generated based on preset templates, such as summarizing statistics by time, region, or defect type.

[0089] Step S407: Ensure downlink communication quality. To ensure that feedback instructions can be delivered to the UAV end in a timely manner, the downlink transmission delay is controlled within a preset time threshold. This time threshold can be 50ms, 100ms, or 200ms. In specific implementation, this can be achieved by selecting low-latency 5G network slices or optimizing the communication protocol stack.

[0090] Meanwhile, to ensure reliable transmission of commands in complex electromagnetic environments, the downlink uses the AES-256 encrypted communication protocol for data encryption and employs forward error correction (FEC) technology to control the command error rate, ensuring that the error rate is kept below a preset threshold, which can be 0.001% or 0.0001%. Forward error correction can use standard schemes such as Reed-Solomon codes or LDPC codes, adding redundant information during encoding to recover from transmission errors.

[0091] The above steps S4 achieve a complete closed loop from video slice uploading, cloud-based deep semantic analysis, to feedback instruction generation and downlink transmission. Specifically, it utilizes the semantic understanding capabilities of a multimodal large language model to achieve high-precision scene analysis, and uses a dynamic threshold adjustment mechanism to achieve intelligent adaptation of end-side sampling behavior, thereby improving the system's data utilization efficiency and real-time response capabilities in complex task scenarios.

[0092] In a specific application implementation logic, the method of the present invention is applied to power line inspection. The drone flies automatically according to the preset tower route, and the end-side feature extraction module processes the video stream collected by the airborne visual sensor in real time.

[0093] During the normal flight phase, the drone flies over repetitive background vegetation or sky areas. At this time, the semantic content between video frames does not change much, and the calculated semantic entropy remains at a low level, below the dynamic judgment threshold set by the system. In this state, the system does not trigger the video slice upload action, but only retains the low-resolution recording as the flight record on the local machine, thereby effectively saving wireless communication bandwidth and cloud computing resources.

[0094] As the drone gradually approaches the tower, and the complex tower structure, insulator strings, and hardware begin to appear in the field of vision, the semantic content of the video frame changes significantly. The semantic entropy calculated in real time by the edge computing platform rapidly increases and exceeds the dynamic judgment threshold. The system then triggers the video slicing and capture logic. The encoder immediately captures a video slice containing the panoramic view of the tower and local details, and uploads it to the cloud server via the wireless communication link.

[0095] After receiving video clips, the cloud-deployed multimodal large language model performs deep semantic analysis based on preset guidance prompts. These prompts are set to "Analyze whether there are insulator damage, wire snagging, or tower corrosion in the video clips, and assess the risk level." After the model analysis is completed, if it determines that there are suspected defects such as cracks, it generates a feedback command and sends it to the drone. This command lowers the dynamic judgment threshold on the drone by a preset ratio, enabling the system to maintain higher sampling sensitivity for the tower area. On the other hand, it controls the gimbal camera to automatically switch to telephoto mode to re-shoot and refine the imaging of the suspected defect location, thereby achieving accurate identification and location of the defect.

[0096] In another specific application implementation logic, the method of the present invention is applied to urban traffic flow monitoring, where drones hover and patrol over key urban intersections, and the edge computing platform performs real-time semantic analysis on the video stream.

[0097] Under normal traffic flow conditions, the traffic flow is stable and the semantic entropy between video frames remains at a low level. When abnormal events such as traffic accidents or illegal parking occur at intersections, the traffic flow changes abruptly, and the semantic entropy value rises rapidly, triggering the upload condition.

[0098] After receiving the video slice, the cloud-based multimodal language model performs deep semantic analysis based on preset guidance prompts. These prompts are set to "Please identify the length of the vehicle queue and the number of abnormally stopped vehicles in the video to predict traffic congestion trends."

[0099] Cloud-based multimodal language models, through multi-frame analysis, can not only identify the specific locations of vehicles involved in accidents but also output deep semantic descriptions of the causes of congestion. For example, a cloud-based multimodal language model can generate the following analysis result: "Due to a minor collision between two cars, which occupied the two lanes on the left, the queue behind exceeded 200 meters in length." This structured semantic information can provide precise decision support for traffic management departments.

[0100] Furthermore, in the UAV power pole bypass mission, this method introduces a dynamic adjustment mechanism for the perception field. This mechanism dynamically adjusts the perception field range of the semantic feature extraction layer based on the distance values ​​measured in real time by the airborne lidar or ultrasonic distance sensor.

[0101] Specifically, when the drone is far from the target tower, the lightweight convolutional neural network backbone model focuses on the global contour features of the target object and extracts the semantic information of the overall structure. When the drone approaches the target object for detailed inspection, such as when the distance is less than 5 meters, the system automatically switches to high-resolution input mode, and the neural network focuses on local details such as bolts, pins, and hardware.

[0102] By dynamically switching the receptive field, the system ensures that the calculation of semantic entropy at different inspection distances can accurately reflect the core semantic content that the current task is concerned with, and avoids feature extraction from becoming out of focus or semantic discrimination from becoming inaccurate due to changes in distance.

[0103] To further improve the system's robustness and response efficiency in complex dynamic environments, this method introduces several optimization mechanisms on top of the basic architecture, as follows: (1) This method introduces a state prediction algorithm in the dynamic threshold adjustment logic. The system continuously records the semantic entropy values ​​of 5 to 10 historical cycles to form a semantic entropy change trajectory. Based on this trajectory, the system uses a linear regression algorithm or a Kalman filter algorithm to predict the semantic shift trend of the next cycle.

[0104] The linear regression algorithm calculates the slope of semantic entropy change by fitting historical data points, thereby determining its upward or downward trend; the Kalman filter algorithm combines historical observations with prediction models to recursively estimate the semantic entropy value at the next moment in the presence of noise interference.

[0105] If the prediction results show that the semantic entropy has a continuous growth trend, meaning that the threshold is very likely to be triggered in the near future, the system will preheat the encoder in advance.

[0106] The warm-up operation specifically includes pre-allocating the buffer resources required for encoding, pre-loading the encoder firmware parameters, and switching the encoder from a low-power standby state to a ready-to-work state.

[0107] Through this predictive warm-up mechanism, the system can effectively shorten the response delay from the fulfillment of trigger conditions to the actual completion of video slice encoding, achieving millisecond-level event capture capability.

[0108] (2) In terms of end-side data management, this method sets an automatic expansion and overwrite mechanism for the circular buffer. The circular buffer adopts a circular queue structure and is configured by default to store raw image frames with a duration of 3 seconds.

[0109] The system monitors the complexity of the current task and the memory resource usage of the edge processor in real time. The criteria for determining the task complexity include the fluctuation frequency of semantic entropy, the frequency of triggering uploads, and the density of target detection.

[0110] When the system detects that the current task is highly complex and memory resources are sufficient, the length of the circular buffer can be temporarily extended from 3 seconds to 10 seconds to ensure that more historical frames and subsequent frames can be fully preserved during the event-frequent phase, and to avoid key frames being overwritten prematurely due to insufficient buffer capacity.

[0111] Meanwhile, the overwrite mechanism of the circular buffer follows the first-in, first-out principle. When the buffer is full, newly written frames will automatically overwrite the oldest stored frames, ensuring that the buffer always stores the latest video data.

[0112] (3) The cloud-based feedback instructions in this method further extend the remote adjustment capability of airborne visual sensor parameters. After completing the video slice analysis, the cloud-based multimodal large language model not only outputs target recognition results and value assessment, but also provides targeted sensor adjustment suggestions based on image quality analysis.

[0113] For example, when the model detects that the overall video image is too dark and details are lost, it can suggest increasing exposure compensation or increasing sensor gain; when it detects that the foreground target is too dark and the background is too bright in a backlit scene, it can suggest enabling wide dynamic range (WDR) mode; when it detects that the image contrast is insufficient, it can suggest adjusting the contrast parameter.

[0114] The cloud server encapsulates these suggestions into feedback instructions and sends them to the drone via the downlink. Upon receiving the feedback instructions, the drone automatically adjusts the corresponding airborne vision sensor parameters.

[0115] Through a remote adjustment mechanism, the system can continuously optimize the quality of the original image acquisition, thereby improving the stability and accuracy of edge feature extraction and forming a global optimization closed loop from image acquisition to feature extraction to cloud analysis.

[0116] The aforementioned optimization mechanisms enhance the core technical solution from three dimensions: response latency, data caching, and image acquisition. The state prediction algorithm enables the encoder to warm up in advance by predicting semantic change trends, effectively compressing the response time of event capture. The adaptive expansion mechanism of the circular buffer allows the system to flexibly allocate storage resources in complex task scenarios, avoiding the loss of key frame data. The cloud-based remote adjustment of sensor parameters constructs a complete feedback chain from image quality analysis to front-end acquisition parameter adjustment, improving the stability of feature extraction from the source.

[0117] These optimization mechanisms work together to enable the entire system to maintain efficient, stable, and adaptive operation when facing dynamically changing environments and complex task requirements.

[0118] This application also provides a real-time video analysis system for large-scale drones, used to perform the steps in the above method embodiments, and the system includes the following components: On the drone's edge, a lightweight convolutional neural network backbone model is deployed to acquire raw video sequences in real time and extract high-dimensional feature vectors for each frame. These high-dimensional feature vectors are used to characterize the global deep semantic features of the current video frame.

[0119] The semantic entropy discrimination module is used to calculate the semantic entropy of the current video frame based on the high-dimensional feature vector. Semantic entropy is used to characterize the semantic offset of the current video frame relative to historical keyframes by quantifying the distance between the current video frame and dynamically updated reference keyframes in the feature space.

[0120] The video slice extraction module compares the semantic entropy with a dynamic judgment threshold. When the semantic entropy exceeds the dynamic judgment threshold, a trigger signal is generated to start the video slice extraction logic and the encoder is used to encode the extracted video slice.

[0121] The threshold adaptive control module is used to adaptively adjust the dynamic judgment threshold according to the real-time status of the wireless communication link, so as to reduce unnecessary data uploads when the link quality deteriorates.

[0122] The warm-up module is used to perform preheating operations on the encoder in advance based on the prediction results of the semantic entropy change trend. The warm-up operations include pre-allocating the buffer resources required for encoding, preloading the encoder firmware parameters, and switching the encoder from a low-power standby state to a working ready state.

[0123] The cloud server is equipped with a multimodal large language model to perform deep semantic analysis on the uploaded video slices and generate feedback instructions based on the analysis results. The feedback instructions are sent to the drone end via the downlink to adjust the data collection behavior of the end.

[0124] The wireless communication module is used to upload the encoded video slices to the cloud server and receive feedback instructions from the cloud.

[0125] The modules mentioned above work together to achieve intelligent closed-loop control with end-to-cloud collaboration, and jointly complete the real-time analysis and adaptive transmission of UAV video.

[0126] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0127] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A real-time video analysis method for large-scale UAV models, characterized in that, Includes the following steps: Step S1: The lightweight convolutional neural network backbone model deployed on the UAV end acquires the original video sequence in real time and extracts a high-dimensional feature vector for each frame of the image. The high-dimensional feature vector is used to characterize the global deep semantic features of the current video frame. Step S2: Calculate the semantic entropy of the current video frame based on the high-dimensional feature vector. The semantic entropy is used to characterize the semantic offset of the current video frame relative to the historical keyframe by quantifying the distance between the current video frame and the dynamically updated reference keyframe in the feature space. Step S3: Compare the semantic entropy with the dynamic judgment threshold. When the semantic entropy exceeds the dynamic judgment threshold, generate a trigger signal, start the video slice capture logic, and use the encoder to encode the captured video slices and upload them to the cloud server. Step S3 further includes a step of adaptively adjusting the dynamic judgment threshold according to the real-time status of the wireless communication link, and a step of preheating the encoder in advance based on the prediction results of the semantic entropy change trend. In step S4, the multimodal large language model deployed on the cloud server performs deep semantic analysis on the uploaded video slices and generates feedback instructions based on the analysis results. The feedback instructions are sent to the drone end via the downlink to adjust the data collection behavior of the end.

2. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, Step S1 specifically includes: Before inputting the raw video sequences acquired by the airborne vision sensor into the lightweight convolutional neural network backbone model, image preprocessing is performed, which includes size scaling and normalization. The lightweight convolutional neural network backbone model adopts a structure based on depthwise separable convolution and outputs high-dimensional feature vectors through global average pooling layers. Step S1 also includes integrating noise reduction measures during the extraction of high-dimensional feature vectors. These measures include performing brightness histogram equalization or Gamma correction on the image to reduce interference from illumination changes, and performing high-pass filtering or exponentially weighted moving average filtering on the initial feature vectors to eliminate high-frequency feature noise caused by camera shake.

3. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, In step S2, semantic entropy is obtained by calculating the Euclidean distance or cosine similarity between the high-dimensional feature vector of the current frame and the high-dimensional feature vector of the reference keyframe. The reference keyframe is initialized as the feature vector of the first frame image at the beginning of system startup, and during subsequent operation, whenever step S3 determines that the video slice upload is triggered, the high-dimensional feature vector of the current frame is updated to the new reference keyframe. Step S2 also includes a step of normalizing the calculated semantic entropy to map it to a preset numerical range.

4. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, In step S3, the adaptive adjustment of the dynamic judgment threshold based on the real-time status of the wireless communication link specifically includes: Set a preset initial value for the dynamic determination threshold; Real-time monitoring of bandwidth utilization, signal-to-noise ratio, or data packet loss rate of wireless communication links; When link congestion or signal attenuation is detected, the dynamic judgment threshold is automatically increased by a preset adjustment ratio to reduce unnecessary data uploads when the link quality is poor.

5. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, Preheating the encoder in advance based on the prediction results of semantic entropy change trends specifically includes: Continuously record semantic entropy values ​​over multiple periods to form a semantic entropy change trajectory; Based on the semantic entropy change trajectory, the semantic shift trend of the next period is predicted using linear regression or Kalman filtering algorithms. If the prediction results show that the semantic entropy continues to increase, the encoder will be warmed up in advance. The warm-up operation includes pre-allocating the buffer resources required for encoding, pre-loading the encoder firmware parameters, and switching the encoder from a low-power standby state to a working ready state.

6. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, In step S3, the video slicing logic is implemented based on a circular buffer mechanism. When a trigger signal is generated, the historical frames before the trigger time and the subsequent frames after the trigger time are read from the circular buffer at the same time. The total length of the video slice is dynamically adjusted based on the target's moving speed detected in real time by the speed sensor on the drone. When the target's moving speed is detected to be greater than a preset speed threshold, the slice length is automatically extended from the first preset length to the second preset length.

7. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, In step S3, when encoding the captured video slices, the current multidimensional state information of the UAV is embedded in the encapsulation header or supplementary enhancement information field. The multidimensional state information includes GPS coordinates, flight altitude, flight speed, gimbal angle, and UAV attitude angle, and is transmitted to the cloud along with the video slices.

8. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, In step S4, after the multimodal large language model performs deep semantic analysis on the uploaded video slices, it also executes a continuous attention value assessment logic. The value assessment logic is as follows: If the probability of a multimodal large language model detecting a task-related target in a video slice is greater than a preset probability threshold, then the scene is determined to have high sustained attention value. The feedback instruction includes a correction coefficient for the dynamic judgment threshold. When a high sustained attention value is determined, the correction coefficient is used to proportionally lower the dynamic judgment threshold on the UAV side, enabling the system to transmit subsequent video sequences with higher sensitivity.

9. The real-time video analysis method for large-scale UAVs according to claim 1, characterized in that, In step S4, after completing the video slice analysis, the multimodal large language model also generates adjustment suggestions for the parameters of the airborne vision sensor based on image quality analysis. The adjustment suggestions are encapsulated as feedback instructions and sent to the drone via the downlink. The adjustment suggestions include increasing exposure compensation, increasing sensor gain, enabling wide dynamic range mode, or adjusting contrast parameters. After receiving feedback instructions, the drone automatically adjusts the corresponding sensor parameters.

10. A real-time video analysis system for large-scale unmanned aerial vehicles (UAVs), used to execute the real-time video analysis method for large-scale UAVs as described in any one of claims 1 to 9, characterized in that, include: On the drone's edge, a lightweight convolutional neural network backbone model is deployed to acquire raw video sequences in real time and extract high-dimensional feature vectors for each frame of image; The semantic entropy discrimination module is used to calculate the semantic entropy of the current video frame based on the high-dimensional feature vector. The semantic entropy is a quantification of the distance between the current video frame and the dynamically updated reference keyframe in the feature space. The video slice extraction module compares the semantic entropy with a dynamic judgment threshold. When the semantic entropy exceeds the dynamic judgment threshold, the video slice extraction logic is triggered, and the extracted video slice is encoded. The threshold adaptive control module is used to adaptively adjust the dynamic judgment threshold according to the real-time status of the wireless communication link. The preheating module is used to preheat the encoder in advance based on the prediction results of the semantic entropy change trend; The cloud server is equipped with a multimodal large language model to perform deep semantic analysis on the uploaded video slices and generate feedback instructions based on the analysis results. The feedback instructions are sent to the drone end via the downlink to adjust the data collection behavior of the end. The wireless communication module is used to upload the encoded video slices to the cloud server and receive feedback instructions from the cloud.

Images