Drone video processing method and system

Abstract

The application belongs to the technical field of unmanned aerial vehicles, and particularly relates to an unmanned aerial vehicle video processing method and system. The unmanned aerial vehicle video processing method comprises the following steps: acquiring a video collected by an unmanned aerial vehicle and associated flight metadata, and generating a memory unit according to the video and the flight metadata; determining a target historical memory unit from a historical memory library based on a current memory unit; determining whether a video segment corresponding to the current memory unit meets an analysis condition according to the current memory unit and the target historical memory unit; in response to the video segment corresponding to the current memory unit meeting the analysis condition, determining a change detection result according to the video segment corresponding to the current memory unit and a historical video segment corresponding to the target historical memory unit; and generating an evidence chain according to the change detection result, wherein the evidence chain is used to review the change detection result.

Description

Technical Field

[0001] This application belongs to the field of unmanned aerial vehicle (UAV) technology, specifically relating to a UAV video processing method and system. Background Technology

[0002] With the development of the low-altitude economy, drone inspections are widely used in scenarios such as park management, river patrols, road inspections, construction site supervision, and energy facility inspections. However, while the same area is periodically filmed repeatedly, the altitude, heading, gimbal angle, and lighting conditions vary with each flight, resulting in lengthy inspection videos.

[0003] In relevant inspection video analysis solutions, image-based change detection methods rely on pixel-level or feature-level alignment of two temporal images. However, they lack stability when there are significant differences in shooting angles, making it difficult to determine the comparability of two flight segments. Large-model-based image recognition methods directly analyze single frames or fixed-frame extractions, failing to consider the high redundancy of UAV videos, resulting in high computational costs and slow response times for full-scale analysis. While long-video summarization methods can compress video length, they do not utilize metadata such as geographic coverage and shooting angle, meaning the selected segments may not contain the key change information required for inspection. Furthermore, the output conclusions of large models lack a structured association with the original image frames, requiring manual retrieval of corresponding images during manual review, which is inefficient and prone to omissions.

[0004] Therefore, there is an urgent need to optimize drone video processing technology. Summary of the Invention

[0005] Therefore, it is necessary to provide a UAV video processing method and system to address the technical problems in existing technologies, such as difficulty in associating inspection videos across flights, high cost of full analysis, and difficulty in verifying inspection results.

[0006] To achieve the above objectives, this specification provides a method for processing UAV video, including...

[0007] Acquire video data collected by the drone and associated flight metadata, and generate a memory unit based on the video data and the flight metadata;

[0008] The target historical memory unit is determined from the historical memory bank based on the current memory unit;

[0009] Based on the current memory unit and the target historical memory unit, determine whether the video segment corresponding to the current memory unit meets the analysis conditions;

[0010] In response to the video segment corresponding to the current memory unit satisfying the analysis conditions, the change detection result is determined based on the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit.

[0011] An evidence chain is generated based on the change detection results, and the evidence chain is used to verify the change detection results.

[0012] On the other hand, this specification provides a drone video processing system, including:

[0013] The access and memory module is used to acquire video and flight metadata collected by the drone and generate memory units;

[0014] The retrieval module is used to determine the target historical memory unit from the historical memory bank based on the current memory unit;

[0015] The judgment module is used to determine whether the video segment corresponding to the current memory unit meets the analysis conditions based on the current memory unit and the target historical memory unit.

[0016] The detection module is used to determine the change detection result based on the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit when the analysis conditions are met.

[0017] An evidence module is used to generate an evidence chain based on the change detection results, and the evidence chain is used to verify the change detection results.

[0018] The UAV video processing method and system of this application have the following beneficial effects:

[0019] In the embodiments of this application, video clips are first fused with flight metadata. Using geographic coverage information and acquisition time from the memory unit, and with a geographic grid as the spatial reference and temporal proximity as the temporal basis, the target historical memory unit is determined from the historical memory bank. Even if the shooting angle and altitude of the drone differ in different flights, as long as the geographic coverage areas overlap, a correlation can be established, thus solving the matching difficulty caused by perspective differences and providing a reliable historical reference for subsequent change detection.

[0020] Secondly, the embodiments of this application calculate the difference measure between the current segment and the historical reference through the visual features in the memory unit, and combine it with the output analysis probability of the lightweight segment selection network to determine whether the current segment is worth entering the subsequent large model analysis process, instead of sending all video segments into the large model. This significantly reduces inference cost and response latency while ensuring that key changes are not lost.

[0021] Furthermore, embodiments of this application input the filtered current video segment and its corresponding historical video segments into a multimodal large model, enabling the model to perform reasoning within the spatiotemporal context of the same geographical location. Compared to schemes that only analyze single-frame or single-temporal images, the determination of change type is more accurate. In addition, this embodiment associates and binds the change detection results with the current evidence frame and historical reference frames to generate a structured evidence chain. This allows the detection conclusions to be traced from textual descriptions back to the original image data, facilitating rapid location and verification by reviewers and reducing review costs. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application or the conventional technology, the drawings used in the description of the embodiments or the conventional technology will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the UAV video processing method provided in the first embodiment;

[0024] Figure 2 This is a schematic diagram of the UAV video processing method provided in the second embodiment;

[0025] Figure 3 This is a schematic diagram of the UAV video processing method provided in the third embodiment;

[0026] Figure 4 This is a schematic diagram of the UAV video processing method provided in the fourth embodiment;

[0027] Figure 5 This is a schematic diagram of the UAV video processing method provided in the fifth embodiment;

[0028] Figure 6 This is a schematic diagram of the UAV video processing method provided in the sixth embodiment;

[0029] Figure 7 This is a schematic diagram of a drone video processing system provided in one embodiment. Detailed Implementation

[0030] To facilitate understanding of this application, a more complete description will be provided below with reference to the accompanying drawings, which illustrate preferred embodiments of the application. However, this application can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to make the disclosure of this application more thorough and complete.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0032] In existing inspection video analysis methods, change detection based on image comparison relies on pixel-level or feature-level alignment of two temporal images. When the UAV's heading deviates or the gimbal's pitch angle changes, the position, scale, and perspective of the same ground object in the image change drastically, making the similarity measurement results unreliable and difficult to determine whether two flight segments are comparable.

[0033] The method of directly using a large model to understand the inspection video involves inputting the complete video or fixed frame images into the large model for analysis. The computational load is linearly related to the video duration. Many segments in the inspection video are repetitive background images, and frame-by-frame or full-volume analysis results in high processing costs and long response times.

[0034] Existing video summarization or keyframe extraction methods select representative frames based on inter-frame differences, motion amplitude, or image clarity, without considering spatial correspondence between voyages and historical change references. In periodic inspection scenarios, they cannot actively focus on areas that differ from historical states, and the selection results are not sufficiently relevant to the change detection task.

[0035] Furthermore, the changes output by the large model are presented in the form of natural language descriptions, without establishing an index association with the original video frames and corresponding historical reference frames. Reviewers need to manually search for the corresponding screen locations in large segments of video, which is inefficient and makes it difficult to quickly verify the correctness of the conclusions.

[0036] Therefore, there are currently technical problems such as difficulty in reliably associating cross-flight segments, low efficiency in inspection video analysis, and difficulty in verifying test results.

[0037] To address the aforementioned technical problems, one or more embodiments of this specification provide a UAV video processing method and system. This method achieves spatiotemporal correlation of cross-flight segments by generating memory units that include geographical coverage, acquisition time, and visual features. It also reduces the inference cost of large models by filtering video segments that meet the analysis conditions based on memory units, and enables traceable verification of detection results by generating evidence chains that bind evidence frames and reference frames.

[0038] In some possible embodiments, please refer to Figure 1 A method for processing drone video is provided, comprising at least the following steps:

[0039] Step S100: Acquire the video and associated flight metadata collected by the UAV, and generate a memory unit based on the video and flight metadata.

[0040] Step S120: Determine the target historical memory unit from the historical memory bank based on the current memory unit.

[0041] Step S140: Based on the current memory unit and the target historical memory unit, determine whether the video segment corresponding to the current memory unit meets the analysis conditions.

[0042] Step S160: In response to the video segment corresponding to the current memory unit satisfying the analysis conditions, determine the change detection result based on the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit.

[0043] Step S180: Generate an evidence chain based on the change detection results. The evidence chain is used to verify the change detection results.

[0044] In step S100, the video stream collected by the UAV during the inspection mission and the accompanying flight metadata are received in real time from the UAV's onboard storage device or via a data link. The flight metadata comes from the onboard GPS receiver, inertial measurement unit, gimbal encoder, and flight controller, and includes at least the UAV's position information, altitude, attitude, gimbal angle, camera intrinsic parameters, and timestamp.

[0045] The video is divided into several candidate segments in chronological order. Geographic coverage information, collection time and visual features are extracted from each candidate segment, and the segments are assembled into memory units according to a predetermined data format.

[0046] Geographic coverage information refers to the location and extent of the ground area captured by the candidate video clip in geographic space. It is generated as follows: based on the location, attitude, gimbal angle, and camera intrinsic parameters in the flight metadata, the image pixel coordinates of keyframes in the candidate video clip are projected onto the ground coordinate system to obtain the coverage polygon of that frame on the ground. This coverage polygon is then mapped onto a preset geographic grid to obtain the set of geographic grids covered by the clip. The acquisition time is taken from the timestamp corresponding to the time of the candidate video clip in the flight metadata. Visual features are feature vectors extracted by inputting keyframes from the candidate video clip into a pre-trained visual encoding model.

[0047] Specifically, when generating geographic overlay information, the image pixels are... According to the camera intrinsic parameter matrix Backproject it as a ray in the camera coordinate system:

[0048]

[0049] Based on the rotation matrix from the UAV system to the world coordinate system Rotation matrix of gimbal-to-device system The direction of the ray in the world coordinate system is obtained:

[0050]

[0051] Let the position of the camera center in the world coordinate system be... The ground intersection is ,but:

[0052]

[0053] in, is the scale factor for the intersection of the ray and the ground. For an approximate plane of the ground... Scene:

[0054]

[0055] in, The height of the camera center. Let be the vertical component of the ray.

[0056] Perform the above back projection on the four corner points or several sampling points of the image to obtain the covering polygon of the segment on the ground. .Will Mapping to a preset geographic grid yields:

[0057]

[0058] in, For a single geographic grid, This represents the area calculation function.

[0059] As an example, the generated memory unit can be stored in a history memory bank. In one embodiment, the memory unit generated for the i-th candidate segment is represented as:

[0060]

[0061] in, This represents the set of geographic grids covered by the segment. This indicates the shooting cone of the segment. Indicates the pose of the drone. Indicates the collection timestamp. Represents visual feature vectors. Indicates image quality score, Indicating evidence index. In one embodiment, capturing a cone of vision. This includes the camera's principal optical axis direction, horizontal field of view, vertical field of view, gimbal pitch angle, gimbal yaw angle, and ground sampling distance. UAV pose. This includes latitude and longitude or projected coordinates, altitude, roll angle, pitch angle, and heading angle.

[0062] In step S120, the current memory unit is any one of the memory units generated in step S100, that is, the memory unit corresponding to a candidate segment segmented from the current voyage video (the memory unit corresponding to a candidate segment currently being processed). At this time, the current memory unit can be used as the retrieval basis to find the historical memory unit in the historical memory bank that is spatially related to the current memory unit and temporally closest to it as the target historical memory unit.

[0063] The determination of target historical memory units includes spatial filtering and temporal filtering. Spatial filtering determines whether two segments captured the same area based on geographic coverage information. In one embodiment, the intersection-union ratio (IUR) of the geographic coverage of the current memory unit and each historical memory unit is calculated, and historical memory units with an IUR greater than a preset threshold are retained. In another embodiment, the spatial distance between the center points of two geographic coverage areas is calculated, and historical memory units with a spatial distance less than a preset threshold are retained. In yet another embodiment, the overlap area of ​​two geographic coverage areas is calculated, and historical memory units with an overlap area greater than a preset threshold are retained. Temporal filtering selects the historical memory unit whose acquisition time is closest to the current time from the spatial filtering results.

[0064] In step S140, a difference metric can be calculated based on the visual features in the current memory unit and the visual features in the target historical memory unit. The difference metric quantifies the degree of change of the current segment relative to a historical reference. In one embodiment, the difference metric is the Euclidean distance between two visual feature vectors. In another embodiment, the difference metric is the cosine distance between two visual feature vectors. In yet another embodiment, the difference metric is the Manhattan distance between two visual feature vectors.

[0065] The difference metric is input into the segment selection network to obtain the analysis probability of the current video segment being selected. The segment selection network outputs a probability value between 0 and 1. When the analysis probability is greater than a preset probability threshold, the current segment is determined to meet the analysis conditions. The analysis conditions can be understood as the comprehensive conditions that a candidate video segment must meet to be deemed worthy of being included in the subsequent multimodal large-scale model for change detection; that is, the segment must meet preset requirements in terms of image quality, change significance, and analytical value to qualify for entry into the large-scale model inference stage.

[0066] In one embodiment, the fragment selection network is a multilayer perceptron (MLP) structure that directly maps difference metrics to analytical probabilities. In another embodiment, the fragment selection network is a Transformer structure with a self-attention mechanism that performs contextual modeling on the difference metrics of multiple fragments within the same voyage and outputs the analytical probabilities of each fragment. In yet another embodiment, the fragment selection network is a lightweight gradient boosting tree (LightGBM) model that takes difference metrics and other auxiliary features as input and outputs analytical probabilities.

[0067] In step S160, change detection can be performed only on video segments that meet the analysis conditions. The current video segment and the historical video segments corresponding to the target historical memory unit are input into the multimodal large model to obtain the change detection results.

[0068] In one embodiment, the input to the multimodal large model is an image pair consisting of a single keyframe from the current segment and a single keyframe from a historical segment. The model outputs whether a change exists in the image pair and the type of change. In another embodiment, the input to the multimodal large model is a video pair consisting of multiple consecutive frames from the current segment and multiple consecutive frames from a historical segment. The model outputs the type of change and the time of change. In yet another embodiment, the input to the multimodal large model is an image pair with text prompts, whereby the text prompts guide the model to focus on a specific type of change.

[0069] In step S180, an evidence chain is generated based on the change detection results. The evidence chain is used to make the change detection results traceable to the original image data for verification.

[0070] In one embodiment, the evidence chain includes change detection results, an evidence index for the current video segment, evidence indexes for historical video segments, and geographic location information. In another embodiment, the evidence chain also includes the location coordinates of the changed area in the image and its confidence score. In yet another embodiment, the evidence chain further includes a review suggestion field to prompt reviewers to focus on areas of change.

[0071] In one embodiment, the output structured chain of evidence is represented as follows:

[0072]

[0073] in, For event type, The event status includes new, ongoing, disappeared, unchanged, and pending review. For geographic location or grid number, For the current evidence frame or fragment index, For temporally adjacent comparable historical evidence frames or fragments indexed, For the description of change, For confidence level, For review recommendations or handling recommendations.

[0074] For example, a chain of evidence may include the following: the event type is new material stockpile, the event status is new, the geographical location is grid 17 in area A, the current evidence is frame 10321 of the voyage on May 23, 2026, the temporally adjacent comparable reference is frame 08455 of the voyage on May 16, 2026, the change description is that light-colored stockpile appears on the right side of the road in the current image while the area in the historical reference image is empty, the confidence level is 0.88, and the review suggestion is manual review.

[0075] In this embodiment, video clips are first fused with flight metadata. Using geographic coverage information and acquisition time from the memory unit, and with a geographic grid as the spatial reference and temporal proximity as the temporal basis, the target historical memory unit is determined from the historical memory bank. Even if the drone's shooting angle and altitude differ across different flights, as long as the geographic coverage areas overlap, a correlation can be established. This solves the matching difficulty caused by perspective differences and provides a reliable historical reference for subsequent change detection.

[0076] Secondly, this embodiment calculates the difference measure between the current segment and the historical reference through the visual features in the memory unit, and combines it with the output analysis probability of the lightweight segment selection network to determine whether the current segment is worth entering the subsequent large model analysis process, instead of sending all video segments into the large model. This significantly reduces inference cost and response latency while ensuring that key changes are not lost.

[0077] Furthermore, this embodiment inputs the selected current video segment and its corresponding historical video segments into a multimodal large model, enabling the model to reason within the spatiotemporal context of the same geographical location. Compared to schemes that only analyze single-frame or single-temporal images, the determination of change type is more accurate. In addition, this embodiment associates and binds the change detection results with the current evidence frame and historical reference frames to generate a structured evidence chain. This allows the detection conclusions to be traced from textual descriptions back to the original image data, facilitating rapid location and verification by reviewers and reducing review costs.

[0078] In some possible embodiments, please refer to Figure 2 Step S120 includes:

[0079] Step S121: Based on the geographical coverage information in the current memory unit, select historical memory units from the historical memory bank whose geographical coverage area overlaps with the current video clip.

[0080] Step S122: Based on the acquisition time in the current memory unit, determine the historical memory unit whose acquisition time is closest to the current video segment from the screening results, and use it as the target historical memory unit.

[0081] In step S12, geographic coverage overlap refers to the existence of at least one identical geographic grid in two geographic coverage grid sets.

[0082] In one embodiment, the cross-union ratio (CUNR) of the geographical coverage of the current memory cell and each historical memory cell is calculated, and historical memory cells with a CUNR greater than a preset threshold are retained.

[0083] In another embodiment, the spatial distance between the center points of two geographic coverage areas is calculated, and historical memory units with a spatial distance less than a preset threshold are retained. In yet another embodiment, the overlapping area of ​​two geographic coverage areas is calculated, and historical memory units with an overlapping area greater than a preset threshold are retained.

[0084] In step S122, if there are multiple historical memory units in the screening results that have the same collection time as the current segment or a difference less than a preset time threshold, then the one with the highest geographical coverage overlap is selected as the target historical memory unit.

[0085] In one embodiment, the determination of temporally adjacent comparable references also needs to satisfy the following constraints:

[0086]

[0087]

[0088]

[0089]

[0090] in, This indicates that the historical memory unit was collected earlier than the current memory unit; This represents the intersection-union ratio (IoU) of the geographic coverage grids of the current memory unit and the historical memory unit. Geographic overlap threshold; This indicates the angle between the two shooting cone directions. The threshold for the difference in viewpoints; Image quality score for historical memory units, This is the quality threshold.

[0091] If no memory cell simultaneously meets the above conditions, the current memory cell is marked as having no comparable historical reference.

[0092] For example, during a river patrol voyage, the current segment A covers grids 5 to 8, and the data collection time is 10:30 AM on June 1, 2026. The historical memory database contains segment B (collected at 10:20 AM on the same day, covering grids 4 to 7 with an intersection-over-union ratio of 0.6) and segment C (collected at 10:25 AM on May 28, covering grids 5 to 8 with an intersection-over-union ratio of 0.9). Segment C is identified as the target historical memory unit because it has a higher intersection-over-union ratio and completely identical geographical coverage, rather than segment B, which was collected more recently.

[0093] First, spatial filtering is performed using geographic coverage information to ensure that the target historical memory unit and the current segment were captured in the same geographic area. This solves the spatial misalignment problem caused by flight path deviations in cross-flight segments, allowing subsequent change detection to be performed within the same geographic range and avoiding false detections due to area misalignment. Compared to existing solutions that rely on image feature matching, this embodiment uses a geographic grid as a spatial reference, which is unaffected by changes in lighting, texture, and shooting angle, and can still accurately establish spatial correspondences even when there are significant differences in viewing angle.

[0094] Secondly, by collecting data by time, the most recent historical memory unit is determined from the spatial filtering results. This ensures that the historical reference reflects the latest historical state of the area, avoiding the problem of distorted change judgments caused by using outdated historical data. For periodic inspection scenarios, the most recent reference in time can eliminate the interference of dealt-upon events and long-term accumulated changes on the detection results, making the determination of change status more accurate.

[0095] In some alternative embodiments, please refer to Figure 3 Step S140 includes:

[0096] Step S141: Determine the difference measure based on the visual features in the current memory unit and the visual features in the target historical memory unit.

[0097] Step S142: Input the difference metric into the segment selection network to obtain the analysis probability that the video segment corresponding to the current memory unit is selected.

[0098] Step S143: In response to the analysis probability being greater than a preset probability threshold, determine that the video segment corresponding to the current memory unit meets the analysis conditions.

[0099] In step S141, the difference metric is used to quantify the degree of change of the current segment relative to historical references. Visual features are fixed-dimensional feature vectors; in one embodiment, the difference metric is the Euclidean distance between two feature vectors, expressed as:

[0100]

[0101] in, The visual features of the current segment, Visual features of the target historical memory unit. This represents the L2 norm. A larger value indicates a more significant change in the current segment relative to a historical reference. In another embodiment, the difference metric is the cosine distance between two feature vectors:

[0102]

[0103] In yet another embodiment, the difference metric is the Manhattan distance between two feature vectors:

[0104]

[0105] In step S142, the segment selection network can be a lightweight neural network that outputs a probability value between 0 and 1, representing the confidence level of the current segment value to be fed into the multimodal large model for change detection.

[0106] In one embodiment, the segment selection network is a multilayer perceptron architecture that maps difference metrics to analytical probabilities. In another embodiment, the input to the segment selection network also includes the image quality score and capture frustum information of the current segment. In yet another embodiment, the segment selection network is a network architecture incorporating a self-attention mechanism that performs contextual modeling of difference metrics for multiple segments within the same flight and outputs the analytical probabilities for each segment.

[0107] As an example, the input vector for the fragment selection network is:

[0108]

[0109] in, The visual features of the current segment, Visual features that are temporally adjacent and comparable. For the characteristics of the change range, It is a directional change feature. and These are the view frustum codes for the current segment and the reference segment, respectively. and These are the image quality scores for the current segment and the reference image, respectively.

[0110] Furthermore, the network structure of the segment selection network can be configured by first encoding the input features using a multilayer perceptron:

[0111]

[0112] in, Let be the latent vector of the video segment. It consists of a fully connected layer, an activation function, and a normalization layer.

[0113] For the set of latent vectors in the same voyage Perform self-attention calculation:

[0114]

[0115] in, This represents the number of candidate segments for the current video segment's voyage. This is a fragment representation after incorporating the context of the same voyage.

[0116] Encode temporally adjacent comparable reference memory units:

[0117]

[0118] Then, cross-attention is calculated between the current segment representation and the historical reference representation:

[0119]

[0120] in, Selecting and representing fragments after incorporating historical reference information.

[0121] Final output fragment selection probability:

[0122]

[0123] in, This represents the probability that a candidate segment will be selected as input to a multimodal large model. and For trainable parameters, This is the Sigmoid function.

[0124] In step S143, the preset probability threshold is set according to the actual application scenario; for example, it is set to 0.6. If the analysis probability is greater than 0.6, the current segment is determined to be worth analyzing; otherwise, the segment is skipped and not sent to the subsequent large model. For segments that do not meet the analysis conditions, steps S160 and S180 are not executed, and the next candidate segment is processed directly. It can be understood that the preset probability threshold can also be 0.3, 0.8, etc., and this embodiment does not specifically limit it.

[0125] In this embodiment, firstly, the variation between the current segment and historical references is quantified through a difference metric, transforming visual similarity into a comparable numerical indicator to provide an objective basis for subsequent screening. Compared to fixed-interval frame sampling or random sampling, the difference metric can proactively capture regions that have changed relative to historical states. Moreover, the difference metric is transformed into analytical probabilities through a segment selection network, and the screening granularity is controlled by a probability threshold. Compared to directly setting a difference metric threshold, the segment selection network can fuse multiple input features and learn the optimal decision boundary, maintaining stable screening performance in different scenarios.

[0126] Finally, only the segments that meet the analysis conditions are fed into the subsequent large model, skipping a large proportion of unchanged redundant segments. This significantly reduces the amount of input data and inference cost of the multimodal large model while ensuring that key changes are not lost.

[0127] In some possible embodiments, please refer to Figure 4 Step S160 includes:

[0128] Step S161: Input the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit into the multimodal model.

[0129] Step S162: Obtain the change type output by the multimodal model as the change detection result.

[0130] In step S161, the multimodal model is a cloud-based model capable of simultaneously receiving images, video clips, and text prompts and outputting inference results. In one embodiment, the input is an image pair consisting of a single keyframe from the current clip and a single keyframe from a historical clip. In another embodiment, the input is a video pair consisting of multiple consecutive frames from the current clip and multiple consecutive frames from a historical clip. In yet another embodiment, the input is an image pair with text prompts, where the prompts guide the model to focus on specific types of changes, such as "Please compare the two images to determine if there are any new buildings, stockpiles of materials, vehicles, or floating objects."

[0131] In step S162, the change type includes addition, disappearance, change, and no change. For example, in a construction site inspection scenario, the output change type could be "addition of material stockpiles," "addition of vehicles," or "addition of buildings." In a river inspection scenario, the output change type could be "addition of floating objects," "bank damage," or "water level change." In a road inspection scenario, the output change type could be "addition of obstacles," "road surface damage," or "change in road markings." These scenarios are merely examples and do not limit the scope of protection of this embodiment.

[0132] In this embodiment, firstly, the multimodal model can simultaneously understand image content and text prompts. Compared to a pure vision model, it has higher accuracy in identifying specific types of changes, and the output results can be directly used in downstream business systems. Secondly, using video clips instead of single frames as input allows the model to utilize temporal context information, reducing false positives in single frames. For example, false positives caused by instantaneous factors such as birds flying or changes in lighting are filtered out in the context of consecutive frames, making the judgment of change types more robust.

[0133] In one embodiment, after step S100, the following may also be included:

[0134] Step S105: Store the memory unit to the historical memory bank. The memory unit includes geographical coverage information, collection time, visual features and evidence index. The evidence index is used to point to the corresponding video segment in the video.

[0135] The evidence chain includes change detection results, evidence index of the current memory unit, and evidence index of the target historical memory unit.

[0136] Please see Figure 5 Step S180 may include:

[0137] Step S181: Locate the evidence frame in the current video segment based on the evidence index of the current memory unit.

[0138] Step S182: Locate the reference frame in the historical video segment based on the evidence index of the target historical memory unit.

[0139] Step S183: Verify the change detection results by comparing the evidence frame and the reference frame.

[0140] In step S105, the memory unit includes geographic coverage information, acquisition time, visual features, and evidence index. The evidence index is used to point to the corresponding video segment in the video, including the original video file path, the segment start frame number, and the end frame number, and is used to quickly locate the original image data during review.

[0141] The historical memory bank stores memory units and their corresponding original video clip indexes from different voyages, times, and geographical coverage areas, providing a data foundation for the subsequent retrieval in step S120. As inspection voyages accumulate, the historical memory bank is continuously updated, and memory units in the same geographical area are arranged according to the collection time, forming a time-series memory for that area.

[0142] In step S181, the evidence frame can be a single frame image of the most significant change area, or it can be a multi-frame sequence containing the change process.

[0143] In step S182, the reference frame is comparable to the evidence frame in terms of shooting angle and geographical coverage, and serves as a comparison benchmark for judging whether the change has actually occurred.

[0144] In step S183, the reviewer can directly locate and compare the evidence frame and the reference frame to verify whether the change type output by the large model is accurate, and trace back to the original video segment through the index information in the evidence chain.

[0145] In this embodiment, firstly, the evidence index associates the change detection results with the original video frames. Reviewers no longer need to manually search through long video segments; they can directly jump to the scene corresponding to the conclusion, significantly improving review efficiency. Secondly, by simultaneously providing the current evidence frame and historical reference frames, reviewers can verify the change detection results within the spatiotemporal context of the same geographical location, which is more intuitive and reliable than relying solely on a single image frame or plain text description. Finally, the evidence chain encompasses complete information such as change type, geographical location, current evidence frame, historical reference frames, and confidence level, which can be directly used to generate inspection reports or push them to the business system, achieving a closed-loop business process from detection to action.

[0146] In some possible embodiments, the memory unit further includes capture frustum information, which is determined based on the UAV's position, attitude, gimbal angle, and camera parameters from the flight metadata. The capture frustum information is used to characterize the spatial observation orientation when the video clip was captured. The capture frustum includes at least the camera's principal optical axis direction (yaw angle relative to north and pitch angle relative to the horizontal plane), horizontal field of view, vertical field of view, and ground sampling distance.

[0147] Step S120 also includes:

[0148] Step S123: Based on the shooting cone information in the current memory unit, select historical memory units from the historical memory bank whose shooting angle is less than the preset angle threshold.

[0149] In step S123, specifically, each angular component in the shooting cone of the current segment is compared with the corresponding component of the historical memory unit. When the difference of all angular components is less than the preset viewing angle threshold, the historical memory unit is determined to meet the viewing angle comparability condition.

[0150] In one embodiment, the difference in shooting angle is measured by the angle between the principal optical axis directions:

[0151]

[0152] in, The unit vector of the principal optical axis direction of the current segment. This is the unit vector along the principal optical axis of the historical memory unit. The spatial angle between the two. When When the angle is less than a preset angle threshold (for example, 15°), the shooting angle is determined to be comparable.

[0153] In one embodiment, the difference in shooting perspective also includes the ratio of ground sampling distances:

[0154]

[0155] in, The ground sampling distance for the current segment. The ground sampling distance for the historical memory unit. When When the observation scales of the two segments are within a preset range (exemplarily, 0.5 to 2.0), they are determined to be comparable.

[0156] Step S123 can be executed after step S121 or in parallel with step S121. That is, both spatial filtering and perspective filtering are used to select memory units that meet the conditions from the historical memory bank. Step S122, time filtering, is executed after both spatial filtering and perspective filtering are completed. From the filtering results that simultaneously meet the conditions of geographical coverage overlap and shooting perspective comparability, the historical memory unit whose acquisition time is closest to the current segment is determined.

[0157] For example, if a historical memory unit completely overlaps with a current segment geographically, but was captured when the drone was tilted at a large angle, with the angle between the principal optical axis and the current segment reaching 40°, exceeding a preset viewing angle threshold of 15°, then this historical memory unit will not meet the comparability criteria due to the excessive difference in viewing angle and will not be identified as a target historical memory unit. By introducing a shooting cone constraint, this embodiment can exclude segments that, although capturing the same area, have incomparable image appearances due to excessive differences in shooting angles, thereby improving the reliability of target historical references and avoiding false detections caused by differences in viewing angle.

[0158] In this embodiment, firstly, the observation azimuth difference between two segments is quantified by capturing view cone information, ensuring that the target historical memory unit and the current segment not only capture the same area but also have similar shooting angles. This solves the problem of large variations in image appearance and difficulty in determining comparability due to differences in shooting angles. Compared to schemes that rely solely on image similarity matching, the capture view cone constraint can eliminate mismatches caused by different viewing angles but similar textures. Secondly, the capture view cone information is derived from the geometric calculation of flight metadata and does not depend on the image content itself, allowing it to work stably even in scenarios with poor image quality, such as changes in lighting, shadow interference, or sparse terrain textures. Finally, spatial constraints and viewing angle constraints together ensure the reliability of historical references, while acquisition time constraints ensure the timeliness of references. The combination of these three factors guarantees the comparability of cross-flight segments from three dimensions: space, viewing angle, and time.

[0159] In some possible embodiments, the memory unit further includes an image quality score, which characterizes the quality of the video segment. The image quality score, used to characterize the quality of the video segment, includes at least one or more of three dimensions: sharpness, exposure quality, and viewpoint availability, and is calculated and written into the memory unit in step S100 when the memory unit is generated.

[0160] Please see Figure 6 Step S140 includes:

[0161] Step S144: Obtain the image quality score in the current memory cell.

[0162] Step S145: In response to the image quality score being lower than a preset quality threshold, determine that the video segment corresponding to the current memory unit does not meet the analysis conditions.

[0163] In step S144, the image quality score is a numerical indicator, which in one embodiment ranges from 0 to 1. The higher the value, the better the image quality.

[0164] In step S145, if the image quality score is lower than the preset quality threshold, the segment is filtered out in advance due to substandard image quality. Steps S141 to S143 (difference measurement calculation and segment selection network inference) are no longer executed, and the subsequent change detection process is skipped directly.

[0165] Specifically, the preset quality threshold is a configurable hyperparameter, exemplarily set to 0.4. When the image quality score is below 0.4, it is determined to be a low-quality segment and is skipped directly. Only when the image quality score is not lower than 0.4 will steps S141 to S143 be executed to further determine whether the analysis conditions are met based on visual feature differences and the segment selection network.

[0166] The image quality score is calculated as follows:

[0167]

[0168] in:

[0169]

[0170]

[0171]

[0172] in, The sharpness score is obtained by Laplacian variance normalization; For better exposure quality, The number of overly dark pixels, For the number of overexposed pixels, This represents the total number of pixels. For viewpoint usability, Ground sampling distance, The target ground sampling distance, These are scale control parameters; For each weight, satisfy .

[0173] For example, a candidate clip suffers from motion blur due to the high-speed movement of a drone, resulting in a low Laplacian variance normalized value and a low sharpness score. Image quality score If the quality score is below the preset quality threshold of 0.4, it is directly determined that the analysis conditions are not met and is not sent to the subsequent difference measurement calculation and fragment selection network, nor is it sent to the multimodal large model. For example, a candidate fragment may have severely distorted ground features due to an excessively large gimbal pitch angle, resulting in poor viewpoint usability. Image quality score Footage with an exposure quality below the preset quality threshold of 0.4 is also filtered out. For example, a candidate clip might be heavily dark due to backlighting, resulting in poor exposure quality. Image quality score Fragments below a preset quality threshold of 0.4 are filtered out. Through the aforementioned quality pre-screening mechanism, this embodiment excludes low-quality segments before the difference metric is calculated, further reducing the computational burden.

[0174] In this embodiment, firstly, before the difference metric calculation and segment selection network inference, low-quality segments are pre-screened using image quality scores to avoid invalid analysis of blurry, overexposed, underexposed, or unusable segments, thus reducing the computational burden on the subsequent segment selection network and multimodal model. Secondly, the image quality score calculation relies only on image statistics and flight geometry, without requiring deep learning model inference, resulting in high computational efficiency and suitability for large-scale video processing scenarios in the cloud. Finally, the preset quality threshold can be flexibly configured according to different inspection tasks. For example, for fine recognition tasks (such as crack detection), a higher threshold (such as 0.6) can be set to retain only high-resolution segments; for large-scale change detection tasks, a lower threshold (such as 0.3) can be set to retain more candidate segments.

[0175] In some possible embodiments, after step S180, the following is included:

[0176] Step S200: Generate and output an inspection report based on the chain of evidence. The inspection report includes at least one of the following: change type, change location, current evidence frame, historical reference frame, and review suggestion.

[0177] In step S200, the inspection report is a structured output document for inspection managers, reviewers, or business systems, used to present the changes discovered during this inspection and the basis for review. The inspection report includes at least one of the following: change type, change location, current evidence frame, historical reference frame, and review recommendations.

[0178] For example, an inspection report may include the following: the change type is new stockpile, the change location is grid 17 in area A (longitude 120.1234°, latitude 31.5678°), the current evidence frame is frame 10321 of the voyage at 10:30 on June 1, 2026, the historical reference frame is frame 08455 of the voyage at 10:25 on May 25, 2026, the change description is that light-colored stockpile appears on the right side of the road in the current image while the area was empty in the historical reference image, the confidence level is 0.88, and the review recommendation is to recommend on-site review to confirm the nature of the stockpile and whether it is in violation of regulations.

[0179] In one embodiment, the inspection report is output in a structured JSON format for easy parsing and automated processing by downstream business systems. In another embodiment, the inspection report is presented in a visual interface, displaying the current evidence frame and historical reference frames side-by-side, with changed areas highlighted by annotation boxes, allowing reviewers to directly view and confirm the information. In yet another embodiment, the inspection report is output as a PDF document, containing text descriptions and image evidence for archiving and distribution.

[0180] In this embodiment, the inspection report transforms the structured chain of evidence into a readable output oriented towards business scenarios, enabling change detection conclusions to directly serve the closed loop of inspection business; at the same time, it presents the current evidence frame and historical reference frames, allowing reviewers to quickly compare the state before and after the change without manually searching in the video, greatly reducing review costs; the structured output format facilitates integration with third-party business systems, realizing full automation from change discovery to report generation.

[0181] In some possible embodiments, step S180 is followed by step S190: generating a change signature based on the change detection result, and determining the change state based on the change signature.

[0182] A change signature is a feature summary obtained by structured encoding of change detection results. In one embodiment, a change signature is represented as:

[0183]

[0184] Among them, type i For the type of change, G i For the change of the local geographic grid, loc i To represent the location of the changed area in geographic coordinates, scale i As a scale measure of the magnitude of change, For the visual feature difference between the current segment and the historical reference, t i This represents the capture time of the current segment.

[0185] The change signature is matched with historical change signatures recorded in the regional status table, and the status category of the current change is determined based on the matching result. The regional status table is used to maintain the most recent change record for each geographic grid. Specifically, if the current change signature matches a historical record in the regional status table in terms of geographic location and change type, and the time difference is within a preset range, it is determined to be a continuous change; if the geographic location corresponding to the current change signature has a historical record in the regional status table but the change is not currently detected, it is determined to be a disappearing change; if the geographic location and change type corresponding to the current change signature have no matching record in the regional status table, it is determined to be a new change; if no change is detected in the current segment, it is determined to be no change. Through this status judgment mechanism, this embodiment can avoid the same change being repeatedly reported in multiple voyages, upgrading the inspection conclusion from a single change judgment to continuous reasoning with status tracking.

[0186] For the same purpose, please refer to Figure 7 In some possible embodiments, a drone video processing system is provided, comprising:

[0187] The access and memory module is used to acquire video and flight metadata collected by the drone and generate memory units.

[0188] The retrieval module is used to determine the target historical memory unit from the historical memory bank based on the current memory unit.

[0189] The judgment module is used to determine whether the video segment corresponding to the current memory unit meets the analysis conditions based on the current memory unit and the target historical memory unit.

[0190] The detection module is used to determine the change detection result based on the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit when the analysis conditions are met.

[0191] The evidence module is used to generate a chain of evidence based on the change detection results, and the chain of evidence is used to verify the change detection results.

[0192] It should be noted that the above modules can be located in different physical devices or integrated into the same physical device. The specific implementation of each module does not constitute a limitation on this embodiment, as long as the corresponding function can be achieved. Furthermore, the specific steps performed by each module have been described in detail in the method embodiment, and will not be repeated in this embodiment.

[0193] This application also provides an electronic device, which includes one or more processors and memory resources represented by memory for storing instructions executable by the processor, such as application programs. The application programs stored in the memory may include one or more modules, each corresponding to a set of instructions.

[0194] The electronic device may also include a power supply component configured to perform power management of the electronic device, a wired or wireless network interface configured to connect the electronic device to a network, and an input / output (I / O) interface. The electronic device can be operated based on operating devices stored in memory, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0195] In one embodiment, a computer device, which may be a server, is also provided. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device stores data. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communicating with external terminals via a network connection.

[0196] In one embodiment, a computer device is provided, which may be a terminal. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interface. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The input / output interface of the computer device is used for exchanging information between the processor and external devices. The communication interface of the computer device is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0197] This application also provides a non-transitory computer-readable storage medium, which, when the instructions in the storage medium are executed by the processor of the electronic device, enables the electronic device to perform one of the above embodiments.

[0198] This application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0199] It should be noted that although the steps of the method in this application are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps, such as omitting certain steps, combining multiple steps into one step, and / or breaking down one step into multiple steps, should all be considered part of this application.

[0200] It should be understood that this application is not limited to the detailed data of the methods presented in this specification. This application is capable of other implementations and can be implemented and performed in various ways. The foregoing variations and modifications fall within the scope of this application. It should be understood that the disclosure and definition of this application extend to all alternative combinations of two or more individual features mentioned or apparent in the text and / or drawings. All these different combinations constitute multiple alternative aspects of this application. The embodiments described in this specification illustrate the best known mode for implementing this application and will enable those skilled in the art to utilize this application.

Claims

1. A method for processing video from unmanned aerial vehicles (UAVs), characterized in that, include: Acquire video data collected by the drone and associated flight metadata, and generate a memory unit based on the video data and the flight metadata; The target historical memory unit is determined from the historical memory bank based on the current memory unit; Based on the current memory unit and the target historical memory unit, determine whether the video segment corresponding to the current memory unit meets the analysis conditions; In response to the video segment corresponding to the current memory unit satisfying the analysis conditions, the change detection result is determined based on the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit. An evidence chain is generated based on the change detection results, and the evidence chain is used to verify the change detection results.

2. The UAV video processing method according to claim 1, characterized in that, The process of determining the target historical memory unit from the historical memory bank based on the current memory unit includes: Based on the geographical coverage information in the current memory unit, historical memory units whose geographical coverage area overlaps with the current video clip are selected from the historical memory bank; Based on the acquisition time in the current memory unit, the historical memory unit whose acquisition time is closest to the current video segment is determined from the screening results and used as the target historical memory unit.

3. The UAV video processing method according to claim 1, characterized in that, The step of determining whether the video segment corresponding to the current memory unit meets the analysis conditions based on the current memory unit and the target historical memory unit includes: A difference metric is determined based on the visual features in the current memory unit and the visual features in the target historical memory unit; The difference metric is input into the segment selection network to obtain the analysis probability that the video segment corresponding to the current memory unit is selected. In response to the analysis probability being greater than a preset probability threshold, it is determined that the video segment corresponding to the current memory unit satisfies the analysis condition.

4. The UAV video processing method according to claim 1, characterized in that, The step of determining the change detection result based on the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit includes: Input the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit into the multimodal model; The change type output by the multimodal model is obtained as the change detection result.

5. The UAV video processing method according to claim 1, characterized in that, After generating the memory unit, the method further includes: The memory unit is stored in the historical memory bank. The memory unit includes geographical coverage information, collection time, visual features, and evidence index. The evidence index is used to point to the corresponding video segment in the video.

6. The UAV video processing method according to claim 5, characterized in that, The chain of evidence includes the change detection result, the evidence index of the current memory unit, and the evidence index of the target historical memory unit; The chain of evidence is used to verify the change detection results, including: The evidence frame in the current video segment is located based on the evidence index of the current memory unit; Based on the evidence index of the target historical memory unit, locate the reference frame in the historical video segment; The change detection result is verified by comparing the evidence frame and the reference frame.

7. The UAV video processing method according to claim 1, characterized in that, The memory unit also includes shooting cone information, which is determined based on the UAV position, attitude, gimbal angle and camera parameters in the flight metadata. The step of determining the target historical memory unit from the historical memory bank based on the current memory unit also includes: Based on the shooting cone information in the current memory unit, historical memory units with a shooting angle less than a preset angle threshold are selected from the historical memory bank.

8. The UAV video processing method according to claim 1, characterized in that, The memory unit also includes an image quality score, which is used to characterize the quality of the video segment; The step of determining whether the video segment corresponding to the current memory unit meets the analysis conditions also includes: Obtain the image quality score in the current memory unit; In response to the image quality score being lower than a preset quality threshold, it is determined that the video segment corresponding to the current memory unit does not meet the analysis conditions.

9. The UAV video processing method according to claim 1, characterized in that, After generating the evidence chain based on the change detection results, the method further includes: An inspection report is generated and output based on the chain of evidence. The inspection report includes at least one of the following: change type, change location, current evidence frame, historical reference frame, and review suggestion.

10. A drone video processing system, characterized in that, include: The access and memory module is used to acquire video and flight metadata collected by the drone and generate memory units; The retrieval module is used to determine the target historical memory unit from the historical memory bank based on the current memory unit; The judgment module is used to determine whether the video segment corresponding to the current memory unit meets the analysis conditions based on the current memory unit and the target historical memory unit. The detection module is used to determine the change detection result based on the video segment corresponding to the current memory unit and the historical video segment corresponding to the target historical memory unit when the analysis conditions are met. An evidence module is used to generate an evidence chain based on the change detection results, and the evidence chain is used to verify the change detection results.

Images