Low-altitude video data intelligent analysis method and system

Abstract

The application relates to the technical field of low-altitude video data analysis, and discloses a low-altitude video data intelligent analysis method and system.The method comprises the following steps: acquiring original low-altitude video data shot by a UAV in a low-altitude flight state; acquiring the appearance feature of each frame of image and the optical flow motion feature between continuous frames from the original low-altitude video data in parallel; fusing the appearance feature and the optical flow motion feature to obtain a space-time joint feature sequence; deducing camera motion parameters based on the pixel change of continuous frames in the original low-altitude video data; performing motion correction on the video frame by using the camera motion parameters; and distinguishing a foreground target region in the corrected video sequence and forming a reconstructed background.The application can improve the accuracy and stability of dynamic target detection, guarantee the reliability of high-level semantic event recognition, and provide high-quality core technical support for intelligent analysis of low-altitude video data.

Description

Technical Field

[0001] This invention relates to the field of low-altitude video data analysis technology, and in particular to an intelligent analysis method and system for low-altitude video data. Background Technology

[0002] In the field of intelligent analysis of low-altitude video data, existing technologies often only extract the appearance features of video frames individually or calculate the motion information of consecutive frames in isolation, without achieving deep collaboration and fusion between the two. This processing method cannot fully capture the relationship between the dynamic changes and static details of targets in low-altitude scenes, resulting in the generated feature data being difficult to fully reflect the spatiotemporal attributes of the scene. Especially under conditions such as image shaking and complex backgrounds caused by low-altitude flight, the effectiveness of the features is greatly reduced, which becomes a fundamental obstacle to all subsequent analysis stages.

[0003] Meanwhile, the lack of high-quality spatiotemporal feature support makes it difficult to accurately offset the inter-frame shift caused by low-altitude flight, resulting in poor motion correction and insufficient separation accuracy between foreground targets and background, leading to missed target detection or background misjudgment. Furthermore, the behavioral representation of moving entities and the mapping of high-level semantic events cannot be effectively carried out. It is difficult to accurately extract the core behavioral features of moving entities, nor can a reliable correlation be established between low-level features and high-level semantics. Ultimately, the semantic analysis results fail to match the event essence of the actual scene, making it difficult to meet the practical application needs of intelligent analysis of low-altitude videos. Summary of the Invention

[0004] This invention provides an intelligent analysis method and system for low-altitude video data to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides an intelligent analysis method for low-altitude video data, comprising: S1. Acquire raw low-altitude video data captured by the drone while it is flying at low altitude; S2. Obtain the appearance features of each frame and the optical flow motion features between consecutive frames from the original low-altitude video data in parallel, and fuse the appearance features and optical flow motion features to obtain a spatiotemporal joint feature sequence. S3. Based on the pixel changes of consecutive frames in the original low-altitude video data, the camera motion parameters are derived, and the camera motion parameters are used to perform motion correction on the video frames. In the corrected video sequence, the foreground target area is distinguished and the reconstructed background is formed. S4. Based on the reconstructed background, identify moving entities in the foreground target region and determine their spatial positions to obtain dynamic target detection results; S5. Based on the spatiotemporal joint feature sequence and dynamic target detection results, construct a low-level behavioral representation of moving entities; S6. Map low-level behavioral representations to high-level semantic events, and evaluate the high-level semantic events according to preset rules to obtain semantic analysis results.

[0006] Preferably, the step of obtaining the appearance features of each frame and the optical flow motion features between consecutive frames from the original low-altitude video data in parallel includes: Extract the edge and texture structure of each frame from the raw low-altitude video data to form structural features; Simultaneously, the displacement vectors of pixels are tracked between consecutive frames, and the displacement vectors are aggregated to form motion field features; By fusing structural features with motion field features, a spatiotemporal joint feature sequence is obtained.

[0007] Preferably, the step of extracting the edge and texture structure of each frame of the original low-altitude video data to form structural features includes: Each frame of the image is decomposed into multiple scales to obtain gradient information at different scales; By applying direction- and scale-selective filtering to the gradient information, an enhanced contour response is obtained. The enhanced contour response across multiple scales is integrated to form structural features.

[0008] Preferably, the step of deriving camera motion parameters based on pixel changes in consecutive frames of the original low-altitude video data, and then using these camera motion parameters to perform motion correction on the video frames, includes: Stable feature point pairs are matched between consecutive frames, and affine transformation parameters are determined based on the coordinate correspondence of the feature point pairs. The affine transformation parameters are then used as camera motion parameters. The coordinate transformation of subsequent video frames is performed using affine transformation parameters to obtain the corrected video sequence. In the corrected video sequence, regions with significant pixel differences from the aligned background are separated using the inter-frame difference method and designated as foreground target regions.

[0009] Preferably, the step of separating regions with significant pixel differences from the aligned background using the inter-frame difference method as foreground target regions includes: Apply a morphological dilation operation to the difference region to obtain the dilated region, and then perform a morphological closing operation on the dilated region to obtain the connected component. The area value of the connected component is compared with the preset lower limit of the area, and its boundary regularity index is compared with the preset regularity threshold to filter out the connected components that satisfy both the area condition and the regularity condition. The selected connected components constitute the foreground target region.

[0010] Preferably, the step of identifying moving entities and determining their spatial positions in the foreground target region based on the reconstructed background to obtain dynamic target detection results includes: Associate the foreground target region with the motion field features at corresponding positions in the spatiotemporal joint feature sequence; Based on the continuity and consistency of the associated motion field features, a set of pixels belonging to the same moving entity is divided in the foreground target region; Each set of pixels representing a moving entity is assigned a unique identifier, and its centroid coordinates are determined as its spatial location. The set of entities carrying the identifier and spatial location constitutes the dynamic target detection result.

[0011] Preferably, the step of constructing a low-level behavioral representation of a moving entity based on the spatiotemporal joint feature sequence and dynamic target detection results includes: From the spatiotemporal joint feature sequence, extract feature segments that correspond to the spatiotemporal positions of moving entities in the dynamic target detection results; The evolution patterns of structural features and motion field features in the feature segments are analyzed, and multiple behavioral primitives, including motion direction stability, velocity change rate and regional deformation degree, are extracted. Multiple behavioral primitives are organized in chronological order to form a low-level behavioral representation of a moving entity.

[0012] Preferably, the extraction of multiple behavioral primitives, including motion direction stability, velocity change rate, and regional deformation degree, includes: Motion direction stability is defined based on the statistical distribution of the motion direction angle of a moving entity within a continuous time interval. The instantaneous velocity of the moving entity is obtained based on the characteristics of the sports field, and the rate of change of instantaneous velocity within adjacent time intervals is defined as the rate of change of velocity. By comparing the spatial positions of the boundary point set of the moving entity region across different frames, the region deformation degree is defined.

[0013] Preferably, the step of mapping low-level behavioral representations to high-level semantic events and evaluating the high-level semantic events according to preset rules to obtain semantic analysis results includes: Low-level behavioral representations are matched with predefined event templates, which define the key behavioral primitive combinations and their spatiotemporal logical relationships corresponding to high-level semantic events. When a low-level behavioral representation satisfies all the definition conditions of an event template, the high-level semantic event corresponding to that template is triggered. All triggered high-level semantic events and their corresponding behavioral primitive evidence are organized into a structured list of semantic events, which is the semantic analysis result.

[0014] To address the above problems, the present invention also provides a low-altitude video data intelligent analysis system, the system comprising: The data acquisition module is used to acquire raw low-altitude video data captured by the drone while it is flying at low altitude. The feature fusion module is used to obtain the appearance features of each frame of the original low-altitude video data and the optical flow motion features between consecutive frames in parallel, and fuse the appearance features and optical flow motion features to obtain a spatiotemporal joint feature sequence. The background reconstruction determination module is used to derive camera motion parameters based on the pixel changes of consecutive frames in the original low-altitude video data, apply the camera motion parameters to perform motion correction on the video frames, distinguish the foreground target area in the corrected video sequence, and form a reconstructed background. The detection result acquisition module is used to identify moving entities in the foreground target region and determine their spatial positions based on the reconstructed background, thereby obtaining dynamic target detection results. The low-level behavior representation construction module is used to construct low-level behavior representations of moving entities based on spatiotemporal joint feature sequences and dynamic target detection results. The event evaluation module is used to map low-level behavioral representations to high-level semantic events, and evaluate the high-level semantic events according to preset rules to obtain semantic analysis results.

[0015] Beneficial effects Compared with the prior art, the present invention has the following beneficial effects: 1. By constructing a complete technical chain from feature fusion, motion correction, target detection, behavior representation to semantic mapping, it is possible to comprehensively capture spatiotemporal dynamic information and essential target features in low-altitude video scenes. It achieves deep synergy between appearance features and optical flow motion features, and the generated spatiotemporal joint feature sequence has better scene representation capabilities. Combined with precise camera motion correction and foreground-background separation mechanisms, it can significantly improve the accuracy and stability of dynamic target detection, while ensuring the reliability of advanced semantic event recognition, providing high-quality core technical support for intelligent analysis of low-altitude video data.

[0016] 2. By optimizing the multi-scale processing and filtering strategies for feature extraction, the precise camera motion parameter estimation method, the standardized foreground region screening criteria, and the systematic behavioral primitive extraction and semantic mapping logic, not only is the processing accuracy of each link in the core technology chain enhanced, but the overall solution is also improved to adapt to complex low-altitude environments. This makes the behavioral representation of moving entities more complete and accurate. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an intelligent analysis method for low-altitude video data according to an embodiment of the present invention. Figure 2 A functional block diagram of a low-altitude video data intelligent analysis system provided in an embodiment of the present invention; The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0018] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0019] This application provides a method for intelligent analysis of low-altitude video data. The executing entity of this method includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the method for intelligent analysis of low-altitude video data can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0020] Reference Figure 1 The diagram shown is a flowchart illustrating an intelligent analysis method for low-altitude video data according to an embodiment of the present invention. In this embodiment, the intelligent analysis method for low-altitude video data includes: S1. Acquire raw low-altitude video data captured by the drone while it is flying at low altitude; S2. Obtain the appearance features of each frame and the optical flow motion features between consecutive frames from the original low-altitude video data in parallel, and fuse the appearance features and optical flow motion features to obtain a spatiotemporal joint feature sequence. S3. Based on the pixel changes of consecutive frames in the original low-altitude video data, the camera motion parameters are derived, and the camera motion parameters are used to perform motion correction on the video frames. In the corrected video sequence, the foreground target area is distinguished and the reconstructed background is formed. S4. Based on the reconstructed background, identify moving entities in the foreground target region and determine their spatial positions to obtain dynamic target detection results; S5. Based on the spatiotemporal joint feature sequence and dynamic target detection results, construct a low-level behavioral representation of moving entities; S6. Map low-level behavioral representations to high-level semantic events, and evaluate the high-level semantic events according to preset rules to obtain semantic analysis results.

[0021] In a preferred embodiment, the appearance features of each frame and the optical flow motion features between consecutive frames are obtained in parallel from the raw low-altitude video data, including: Extract the edge and texture structure of each frame from the raw low-altitude video data to form structural features; Simultaneously, the displacement vectors of pixels are tracked between consecutive frames, and the displacement vectors are aggregated to form motion field features; By fusing structural features with motion field features, a spatiotemporal joint feature sequence is obtained.

[0022] Specifically, for each frame of the original low-altitude video data, each pixel is scanned row by row and column by column. The grayscale value of the pixel is compared with the grayscale values ​​of the adjacent pixels above, below, left, and right. When the difference in grayscale values ​​reaches a set threshold, the pixel is marked as an edge point. All edge points are then aggregated to form the image edge.

[0023] It should be noted that the threshold here is determined based on the brightness and contrast of the video shooting scene, and should be adjusted according to actual needs during implementation.

[0024] Furthermore, the image is divided into multiple equally sized local regions, and the distribution pattern of pixel grayscale and the arrangement relationship between pixels in each region are statistically analyzed to obtain image texture. Edge and texture information are then integrated to form structural features.

[0025] This is done because edges and textures are core components of an image's appearance, accurately reflecting the shape and contour of the target. Compared to extracting only a single feature, structural features better match the needs of appearance features, thus improving the basic accuracy of subsequent analysis.

[0026] Furthermore, in two consecutive frames of the original low-altitude video, each pixel in the first frame is taken as the tracking starting point. In the preset neighborhood of the corresponding pixel in the second frame, the gray values ​​of the pixels are compared one by one. The pixel whose gray value is closest to that of the starting pixel is found as the tracking endpoint. The displacement direction and distance of the pixel are determined by the coordinate difference between the starting point and the endpoint, i.e., the displacement vector.

[0027] The neighborhood range here is determined by the video frame rate; the higher the frame rate, the smaller the neighborhood.

[0028] Then, the displacement vectors of all pixels are aggregated and grouped according to their positions in the image to form motion field features that reflect the motion state of the entire frame. This is done because pixel displacement directly reflects the motion changes between consecutive frames. Aggregating displacement vectors can transform scattered motion information into overall motion field features, avoiding the fragmentation of motion information of a single pixel. Compared with analyzing the displacement of a single pixel, motion field features can more accurately capture the motion trend in the video.

[0029] Furthermore, the structural features and motion field features corresponding to the same frame of image are associated one-to-one according to the coordinate position of the pixels in the image. First, the structural features are used as the basic framework, and the motion information of the corresponding position in the motion field features is filled into the framework. Then, the associated features are processed to unify the format and eliminate the dimensional differences between the two features.

[0030] Furthermore, the features associated with each frame are arranged sequentially according to the frame order of the original low-altitude video to form a spatiotemporal joint feature sequence.

[0031] It should be noted that this is done because structural features reflect spatial appearance information, while motion field features reflect temporal motion information. Combining the two can take into account both spatiotemporal information. Compared with using structural features or motion field features alone, spatiotemporal joint feature sequences can more comprehensively present the core information of video data, providing more complete and accurate data support for subsequent intelligent analysis.

[0032] In a preferred embodiment, the edge and texture structure of each frame of the original low-altitude video data are extracted to form structural features, including: Each frame of the image is decomposed into multiple scales to obtain gradient information at different scales; By applying direction- and scale-selective filtering to the gradient information, an enhanced contour response is obtained. The enhanced contour response across multiple scales is integrated to form structural features.

[0033] Specifically, an image pyramid is constructed for each frame to achieve multi-scale decomposition. First, the original image is retained as the base scale, and then the original image is gradually reduced by a fixed ratio to obtain multiple images at reduced scales. The reduction ratio is determined according to the size range of the target in the video.

[0034] Furthermore, the image at each scale is processed pixel by pixel. In the horizontal direction, the gray value of the current pixel is extracted, and then the gray values ​​of its left and right adjacent pixels are extracted respectively. The gray value of the current pixel is then compared with the gray values ​​of its left and right adjacent pixels. The two difference results are converted into absolute values ​​respectively. The two absolute value results are added together and the average value is taken. This average value is the degree of gray change of the current pixel in the horizontal direction.

[0035] Furthermore, in the vertical direction, the grayscale value of the current pixel and the grayscale values ​​of the adjacent pixels above and below are extracted. The grayscale value of the current pixel is then compared with the grayscale values ​​of the adjacent pixels above and below. The two difference results are converted into absolute values, added together, and the average value is taken. This average value is the degree of grayscale change of the current pixel in the vertical direction.

[0036] Furthermore, the horizontal and vertical grayscale changes of all pixels in the image are summarized to form the gradient information at that scale.

[0037] It should be noted that this is done because a single scale cannot cover edges and textures of different sizes. Multi-scale decomposition, combined with this pixel-by-pixel bidirectional interpolation, taking the absolute value and then summing and averaging, can comprehensively capture features from details to the whole. Compared with single-scale processing or simple comparison in a single direction, it can avoid missing key information. Moreover, the method of taking the average after bidirectional comparison can reduce the interference caused by anomalies in a single adjacent pixel, making the gradient information more consistent with the actual edge and texture distribution of the image.

[0038] Furthermore, for the gradient information at each scale, filtering templates of corresponding scale and orientation are used for processing. Each template only allows gradient information consistent with its own orientation and belonging to the corresponding scale to pass through, filtering out noise that does not conform to the orientation or is not at the target scale. The response value of each pixel after processing is the enhanced contour response. This is done because the original gradient contains noise, and this filtering can accurately enhance the effective contour, significantly improving contour clarity and effectiveness compared to ordinary filtering.

[0039] Furthermore, the enhanced contour responses at all scales are aligned to the original image pixel coordinates. For each pixel location, its response values ​​at each scale are summarized, and the response value that best reflects the edge or texture features at that location is selected as the result. The set of all pixel results forms the structural feature. This is done because single-scale responses have limitations, and integrating multiple scales can combine the advantages of each scale. Compared with single-scale responses, the structural feature is more complete and accurate, and can better reflect the image edge and texture structure.

[0040] In a preferred embodiment, camera motion parameters are derived based on pixel changes in consecutive frames of the original low-altitude video data, and motion correction is performed on the video frames using these camera motion parameters, including: Stable feature point pairs are matched between consecutive frames, and affine transformation parameters are determined based on the coordinate correspondence of the feature point pairs. The affine transformation parameters are then used as camera motion parameters. The coordinate transformation of subsequent video frames is performed using affine transformation parameters to obtain the corrected video sequence. In the corrected video sequence, regions with significant pixel differences from the aligned background are separated using the inter-frame difference method and designated as foreground target regions.

[0041] Specifically, in two consecutive frames of images, corner points with drastic gray-level changes are extracted as initial feature points. Gray-level distribution information of a fixed range around each feature point in the previous frame is extracted. In the next frame, the gray-level distribution is compared region by region to locate the region with the same gray-level distribution around the feature point. The core point in the region is taken as the corresponding feature point. Points with incorrect matching are eliminated through a second check of gray-level distribution consistency, and finally, stable feature point pairs are obtained.

[0042] Furthermore, based on the coordinates of these feature point pairs in the two frames, the mapping relationship between the coordinates of the feature points in the previous frame and the corresponding coordinates of the feature points in the next frame is established by matching the coordinates of the feature point pairs. This mapping relationship is the affine transformation parameter, which is used as the camera motion parameter.

[0043] It should be noted that this is done because low-altitude video cameras are prone to translation, rotation and other motions. Stable feature point pairs can accurately reflect the motion correlation between frames. Affine transformation can completely describe this planar motion. Compared with directly offsetting the entire pixel, it can accurately capture the camera's motion pattern and avoid errors caused by the offset of a single pixel.

[0044] Furthermore, the previous frame is selected as the reference frame. Based on the mapping rules established by the affine transformation parameters, the coordinate transformation is performed on each pixel of each subsequent frame to obtain the corresponding coordinates of the pixel in the reference frame. The position of each pixel in the image is adjusted according to the corresponding coordinates so that the background area of ​​the subsequent frame is completely aligned with the background area of ​​the reference frame. All the adjusted frames are arranged in the time order of the original video to obtain the corrected video sequence.

[0045] This embodiment does this because camera movement in the original video causes inter-frame background shifts that interfere with foreground recognition. Coordinate transformation can completely eliminate these shifts. Compared to the uncorrected video sequence, the corrected sequence can keep the background stable, providing a reliable foundation for subsequent separation of foreground targets.

[0046] Furthermore, in the corrected video sequence, adjacent frames are selected and their grayscale values ​​are compared pixel by pixel to obtain the grayscale difference results of each pixel. A difference threshold is set according to the grayscale fluctuation range of the background. When the grayscale difference result of a pixel exceeds the threshold, the connected region where the pixel is located is marked as a region that is significantly different from the aligned background. All marked connected regions are integrated to obtain the foreground target region. It can be understood that the difference threshold here can be adjusted according to the lighting conditions of the actual shooting scene.

[0047] It should also be noted that this is done because the background has been aligned after correction, and the grayscale difference between frames mainly comes from the movement of the foreground target. The grayscale comparison between frames can directly highlight this difference. Compared with separating the foreground in the uncorrected video, it can greatly reduce the interference caused by the background offset and improve the accuracy of foreground target area recognition.

[0048] In a preferred embodiment, regions with significant pixel differences from the aligned background are separated using an inter-frame difference method and designated as foreground target regions, including: Apply a morphological dilation operation to the difference region to obtain the dilated region, and then perform a morphological closing operation on the dilated region to obtain the connected component. The area value of the connected component is compared with the preset lower limit of the area, and its boundary regularity index is compared with the preset regularity threshold to filter out the connected components that satisfy both the area condition and the regularity condition. The selected connected components constitute the foreground target region.

[0049] Specifically, for the difference region obtained by inter-frame difference, a morphological dilation operation is applied using a structuring element of a preset shape. The structuring element is traversed through each pixel of the difference region one by one. If there are overlapping pixels between the structuring element and the difference region, the position is retained and the adjacent pixels are extended to the difference region to obtain the dilated region.

[0050] Furthermore, the same structuring element is used to apply a morphological closing operation to the dilated region. First, the dilation operation is repeated to fill the small gaps in the region, and then a morphological erosion operation is performed to shrink the region boundary to near the original contour, finally obtaining the connected region.

[0051] This is done because the difference region may contain fractures or tiny holes. Expansion can connect the fractured parts, and closing operations can fill the gaps and smooth the boundaries. Compared to using the difference region directly, it can form a complete regional shape and avoid missing foreground targets due to regional fragmentation.

[0052] Furthermore, the total number of pixels contained in each connected component is counted to obtain the area value of the connected component. This area value is then compared with a preset lower limit, which is determined based on the minimum size of common foreground targets in low-altitude videos and is adjusted as needed during implementation.

[0053] By outlining the edge contours of connected components, the ratio of the area enclosed by the contour to the area of ​​the smallest bounding rectangle of the contour is calculated to obtain the boundary regularity index. This index is then compared with a preset regularity threshold, which is determined based on the regularity of the contours of common foreground targets. Connected components that simultaneously satisfy the condition that the area value is not less than the preset lower limit of area and the boundary regularity index is not lower than the preset regularity threshold are retained.

[0054] This is done because there may be small connected regions or irregular interference regions formed by noise in the difference region. Dual screening can eliminate these invalid regions, which can significantly improve the accuracy of foreground target recognition compared to single screening conditions.

[0055] Furthermore, all connected components that have passed the dual screening of area and regularity are integrated, and the specific pixel location range of these connected components in the image is marked. These integrated connected components together constitute the foreground target region.

[0056] Because the selected connected components in this solution accurately correspond to the real foreground targets, the integrated operation can fully present the overall shape and location information of the foreground targets. Compared with retaining a single connected component, this solution can provide complete and accurate target data for subsequent low-altitude video target analysis, avoiding the impact of target splitting on the analysis results.

[0057] In a preferred embodiment, based on the reconstructed background, moving entities are identified and their spatial positions are determined in the foreground target region to obtain dynamic target detection results, including: Associate the foreground target region with the motion field features at corresponding positions in the spatiotemporal joint feature sequence; Based on the continuity and consistency of the associated motion field features, a set of pixels belonging to the same moving entity is divided in the foreground target region; Each set of pixels representing a moving entity is assigned a unique identifier, and its centroid coordinates are determined as its spatial location. The set of entities carrying the identifier and spatial location constitutes the dynamic target detection result.

[0058] Specifically, we first determine the specific coordinates of each pixel in the foreground target region in the image, locate the pixel position that completely corresponds to these coordinates in the spatiotemporal joint feature sequence, extract the motion field features of the position, and bind each pixel in the foreground target region to the motion field features of the corresponding coordinates one by one, so that each foreground pixel carries its own motion information.

[0059] This is done because the foreground target area only reflects spatial location and lacks motion attributes. Associating it with motion field features can provide a basis for distinguishing different moving entities. Compared with dividing entities based solely on spatial regions, this adds a basis for judging motion dimensions and improves the accuracy of entity division.

[0060] Furthermore, observe all pixels carrying motion field features within the foreground target area frame by frame, compare the motion direction and amplitude of adjacent pixels, and group pixels with the same motion direction and similar motion amplitude into temporary pixel groups; then track the motion state of each temporary pixel group across consecutive frames. If the motion direction and amplitude of the pixels in this group do not change abruptly in consecutive frames and remain stable and consistent, then the temporary pixel group is determined as a set of pixels belonging to the same moving entity.

[0061] This is done because all pixels of the same moving entity follow the same motion rules, and the motion state is continuous and consistent. Compared with indiscriminate merging of foreground pixels, it can accurately separate multiple overlapping or adjacent moving entities and avoid target confusion.

[0062] Furthermore, each defined set of moving entity pixels is assigned a unique and non-repeating identifier in a preset order. Here, the identifiers are assigned sequentially according to the actual number of moving entities detected, ensuring that each entity is unique and distinguishable. The edge contour of each set of moving entity pixels is delineated, and the center point that can evenly bisect the area of ​​the contour is found. The coordinates of the center point in the image are used as the centroid coordinates of the moving entity, i.e., its spatial position.

[0063] Then, all moving entity pixels carrying unique identifiers and corresponding centroid coordinates are integrated to form the dynamic target detection result.

[0064] This is done because the unique identifier enables the differentiation and tracking of different moving entities, and the centroid coordinates can accurately represent the spatial position of the target. Compared with simply outputting the target area, the detection results are more practical, facilitating further analysis and processing of dynamic targets and enhancing the application value of the detection results.

[0065] In a preferred embodiment, a low-level behavioral representation of the moving entity is constructed based on the spatiotemporal joint feature sequence and the dynamic target detection results, including: From the spatiotemporal joint feature sequence, extract feature segments that correspond to the spatiotemporal positions of moving entities in the dynamic target detection results; The evolution patterns of structural features and motion field features in the feature segments are analyzed, and multiple behavioral primitives, including motion direction stability, velocity change rate and regional deformation degree, are extracted. Multiple behavioral primitives are organized in chronological order to form a low-level behavioral representation of a moving entity.

[0066] Specifically, the time interval and spatial location of each moving entity in the dynamic target detection results are clearly defined. Here, the time interval corresponds to the range of frames in the video in which it appears, and the spatial location corresponds to the range of pixel coordinates in the image.

[0067] In the spatiotemporal joint feature sequence, the corresponding frame segment is locked according to the time interval, and then in each frame of the frame segment, the features of the corresponding region are extracted according to the spatial coordinate range to obtain the feature segment that matches the spatiotemporal position of the moving entity.

[0068] This is done because the spatiotemporal joint feature sequence contains all the features of the entire video. Extracting the corresponding segment can focus on a single moving entity. Compared with directly using the overall features, it can avoid interference from irrelevant regional features and ensure the focus of subsequent analysis.

[0069] Furthermore, a continuous time interval for the moving entity is selected. It should be noted that the interval length is determined based on the time span of the entity's movement and is adjusted as needed during implementation. Then, the motion direction angles of the moving entity within the interval are extracted frame by frame, and the distribution of all direction angles is statistically analyzed. The stability of the motion direction is defined based on whether most direction angles are concentrated in the same range.

[0070] This is done because the direction of motion for the same behavior is usually regular, and this primitive can reflect the smoothness of the entity's motion. Compared to analyzing only the direction of a single frame, statistics over continuous intervals are closer to the actual behavior state of the entity, thus improving the reliability of the behavior primitive.

[0071] Furthermore, from the motion field features of the feature fragments, the instantaneous velocity of the moving entity is extracted frame by frame, these instantaneous velocities are sorted out in chronological order, and the instantaneous velocities of adjacent time intervals are taken. The velocity change rate is defined by the difference between the two. This is done because velocity change is the core attribute of entity behavior, which can reflect its dynamic state such as acceleration and deceleration. Compared with only focusing on instantaneous velocity, this primitive can capture the dynamic change law of motion and enrich the dimensions of behavior representation.

[0072] Furthermore, the region boundary of the moving entity pixel set is delineated frame by frame, the spatial coordinates of all points on the boundary of each frame are recorded, the spatial coordinate offset of the corresponding boundary points between different frames is compared, and the region deformation degree is defined according to the overall degree of offset.

[0073] It is understandable that physical movement may be accompanied by morphological changes. This primitive can supplement behavioral information in the morphological dimension. Compared with ignoring regional deformation, it can make the behavioral primitive more comprehensive and avoid missing key behavioral features.

[0074] Furthermore, according to the chronological order of the occurrence of the moving entity's behavior, the extracted stability of the motion direction, rate of change of velocity, and degree of regional deformation are arranged in sequence to form an ordered combination of behavioral primitives, which is the low-level behavioral representation of the moving entity.

[0075] It is understandable that chronological order can reconstruct the process of behavior, and orderly combination can accurately reflect the temporal logic of behavior. Compared with disordered stacking or single primitives, the temporal combination of multiple primitives makes the representation more complete and accurate, providing a reliable foundation for subsequent advanced behavior analysis. Compared with traditional single-dimensional behavior representation, it is richer in information.

[0076] In a preferred embodiment, multiple behavioral primitives are extracted, including motion direction stability, velocity change rate, and regional deformation degree, including: Motion direction stability is defined based on the statistical distribution of the motion direction angle of a moving entity within a continuous time interval. The instantaneous velocity of the moving entity is obtained based on the characteristics of the sports field, and the rate of change of instantaneous velocity within adjacent time intervals is defined as the rate of change of velocity. By comparing the spatial positions of the boundary point set of the moving entity region across different frames, the region deformation degree is defined.

[0077] Specifically, the continuous time interval in which the moving entity appears in the video is locked. The length of the interval is determined according to the actual time span of the entity's movement and is adjusted as needed during implementation. The motion direction angle of the entity is extracted frame by frame from the associated motion field features. After summarizing these direction angles, their distribution is sorted out. The motion direction stability is defined based on whether most direction angles are concentrated in the same preset range.

[0078] The direction angle statistics over the continuous time interval can avoid single-frame direction deviation and better reflect the continuity of entity behavior compared to single-frame direction judgment. Combined with the spatiotemporal joint features mentioned above, it can also filter out the direction noise caused by airflow jitter in low-altitude videos and improve the reliability of primitives.

[0079] Furthermore, by extracting the instantaneous velocity of moving entities frame by frame from the motion field features corresponding to the feature segments, it can be seen that the velocity information originates from the aggregation result of the pixel displacement vectors of the previous consecutive frames. By sorting these instantaneous velocities in chronological order, selecting two adjacent time intervals, and comparing the overall difference in instantaneous velocity between the two intervals, the velocity change rate is defined.

[0080] It should be noted that speed changes can intuitively reflect the dynamic states of an entity, such as acceleration and deceleration. Compared to focusing on speed at a single moment, it can capture the dynamic evolution of behavior. Combined with the results of dynamic target detection, it can be accurately associated with a single moving entity, unexpectedly distinguishing the speed changes of overlapping entities and avoiding target confusion.

[0081] Furthermore, based on the determined set of moving entity pixels, the region boundary of the entity is delineated frame by frame, the spatial coordinates of all points on the boundary of each frame are recorded, different frames are selected as comparison benchmarks, the spatial coordinate offset of the corresponding boundary points is compared one by one, and the region deformation degree is defined according to the overall magnitude of the offset.

[0082] It should be noted that, compared with conventional technologies, this implementation method can supplement the morphological dimension information of entity behavior and, combined with the characteristics of the motion field, can achieve accurate representation of the composite motion of "translation + deformation". For example, it can distinguish the difference between the translation of a low-altitude flying drone and the deformation of its wing swing.

[0083] In this embodiment, a preferred approach is to calculate the rate of change of velocity of the moving entity in adjacent time intervals using the following formula to accurately define the rate of change of velocity: in, and These represent the motion entities extracted based on motion field features in time. and The instantaneous velocity scalar value; - Indicates the time span between adjacent time intervals; This indicates the absolute value operation, used to quantify the magnitude of speed changes without distinguishing the direction of acceleration or deceleration.

[0084] It should also be noted that the instantaneous velocity scalar value is the velocity scalar result of the moving entity at the corresponding time point extracted from the motion field features. The time span is the difference between two adjacent time points, which is derived from the time series setting of the video frames. The specific value is determined according to the actual video frame rate and time interval division.

[0085] The formula can accurately quantify the magnitude of velocity change of a moving entity within adjacent time intervals. By taking the absolute value of the velocity difference, the directional differences of acceleration or deceleration are masked, focusing only on the degree of change, thus providing a basis for extracting the behavioral primitive of velocity change rate.

[0086] In general, within the same time span, the larger the speed difference between two points in time, the larger the result, indicating a more drastic speed change; when the speed difference is fixed, the smaller the time span, the larger the result, reflecting a more obvious speed change per unit time.

[0087] In a preferred embodiment, low-level behavioral representations are mapped to high-level semantic events, and the high-level semantic events are evaluated according to preset rules to obtain semantic analysis results, including: Low-level behavioral representations are matched with predefined event templates, which define the key behavioral primitive combinations and their spatiotemporal logical relationships corresponding to high-level semantic events. When a low-level behavioral representation satisfies all the definition conditions of an event template, the high-level semantic event corresponding to that template is triggered. All triggered high-level semantic events and their corresponding behavioral primitive evidence are organized into a structured list of semantic events, which is the semantic analysis result.

[0088] Specifically, the low-level behavioral representation of the moving entity obtained above is locked. Here, the representation is a combination of behavioral primitives composed of motion direction stability, velocity change rate, and regional deformation degree in chronological order. A predefined event template library is retrieved. It should be noted that the template library is based on common behaviors in low-altitude scenes. Each event template clearly marks the corresponding key behavioral primitive combination and the spatiotemporal logical relationships such as the temporal sequence and synchronous association between primitives.

[0089] Furthermore, the behavioral primitives in the low-level behavioral representation are aligned one by one with the key primitive combinations of each event template, while verifying whether the temporal order and association between the primitives conform to the spatiotemporal logical relationship set by the template.

[0090] In this embodiment, the combination of spatiotemporal logic can avoid semantic misjudgment, and combined with the precise behavioral primitives mentioned above, it can distinguish between "similar primitives and different events" in low-altitude scenarios, such as distinguishing the primitive differences between "smooth cruise" and "slow descent" of a drone.

[0091] Furthermore, the definition conditions of each event template are verified one by one. If all behavioral primitives in the low-level behavioral representation are completely consistent with the key primitives of the template, and the spatiotemporal logical relationship between the primitives meets all the requirements of the template and there are no missing or biased conditions, the high-level semantic event corresponding to the event template is directly activated, and the key primitive information that is successfully matched during the triggering process is recorded.

[0092] This is combined with multi-dimensional behavioral primitives to achieve precise triggering of subdivided semantic events, such as subdividing "motion" into specific events such as "uniform linear motion" and "variable speed spiral motion".

[0093] Specifically, all triggered high-level semantic events are sorted out and arranged in chronological order of their occurrence in the video. Corresponding behavioral primitive evidence is added to each event, and the motion entity identifier corresponding to each event is clearly marked. Finally, a structured list of semantic events containing event name, occurrence time, associated entities, and primitive evidence is compiled, which is the semantic analysis result.

[0094] In summary, the structured organization here makes semantic information clear and traceable, which is more convenient for subsequent scene analysis compared to scattered event output; and combined with the dynamic target detection results, it enables the chaining of multiple semantic events of the same moving entity, fully restores its behavioral trajectory, and improves the integrity of low-altitude video semantic analysis.

[0095] like Figure 2 The diagram shown is a functional module diagram of intelligent analysis of low-altitude video data provided in an embodiment of the present invention.

[0096] This invention discloses a low-altitude video data intelligent analysis system 100, which can be installed in an electronic device. Depending on the functions implemented, the low-altitude video data intelligent analysis system 100 may include a data acquisition module 101, a feature fusion module 102, a background reconstruction and determination module 103, a detection result acquisition module 104, a low-level behavior representation construction module 105, and an event evaluation module 106. The modules of this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0097] In this embodiment, the functions of each module / unit are as follows: The data acquisition module 101 is used to acquire raw low-altitude video data captured by the UAV in low-altitude flight mode; The feature fusion module 102 is used to obtain the appearance features of each frame of the original low-altitude video data and the optical flow motion features between consecutive frames in parallel, and fuse the appearance features and optical flow motion features to obtain a spatiotemporal joint feature sequence. The background reconstruction determination module 103 is used to deduce camera motion parameters based on the pixel changes of consecutive frames in the original low-altitude video data, use the camera motion parameters to perform motion correction on the video frames, distinguish the foreground target area in the corrected video sequence, and form a reconstructed background. The detection result acquisition module 104 is used to identify moving entities in the foreground target region and determine their spatial positions based on the reconstructed background, thereby obtaining dynamic target detection results. The low-level behavior representation construction module 105 is used to construct low-level behavior representations of moving entities based on spatiotemporal joint feature sequences and dynamic target detection results. The event evaluation module 106 is used to map low-level behavioral representations to high-level semantic events, and evaluate the high-level semantic events according to preset rules to obtain semantic analysis results.

[0098] In the several embodiments provided by this invention, it should be understood that the disclosed methods and systems can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0099] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0101] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0102] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A low-altitude video data intelligent analysis method, characterized in that, The method includes: S1. Acquire raw low-altitude video data captured by the drone while it is flying at low altitude; S2. Obtain the appearance features of each frame and the optical flow motion features between consecutive frames from the original low-altitude video data in parallel, and fuse the appearance features and optical flow motion features to obtain a spatiotemporal joint feature sequence. S3. Based on the pixel changes of consecutive frames in the original low-altitude video data, the camera motion parameters are derived, and the camera motion parameters are used to perform motion correction on the video frames. In the corrected video sequence, the foreground target area is distinguished and the reconstructed background is formed. S4. Based on the reconstructed background, identify moving entities in the foreground target region and determine their spatial positions to obtain dynamic target detection results; S5. Based on the spatiotemporal joint feature sequence and dynamic target detection results, construct a low-level behavioral representation of moving entities; S6. Map low-level behavioral representations to high-level semantic events, and evaluate the high-level semantic events according to preset rules to obtain semantic analysis results.

2. The intelligent analysis method for low-altitude video data as described in claim 1, characterized in that, The method of obtaining the appearance features of each frame of the image and the optical flow motion features between consecutive frames from the raw low-altitude video data in parallel includes: Extract the edge and texture structure of each frame from the raw low-altitude video data to form structural features; Simultaneously, the displacement vectors of pixels are tracked between consecutive frames, and the displacement vectors are aggregated to form motion field features; By fusing structural features with motion field features, a spatiotemporal joint feature sequence is obtained.

3. The intelligent analysis method for low-altitude video data as described in claim 2, characterized in that, The process of extracting the edge and texture structure of each frame of the original low-altitude video data to form structural features includes: Each frame of the image is decomposed into multiple scales to obtain gradient information at different scales; By applying direction- and scale-selective filtering to the gradient information, an enhanced contour response is obtained. The enhanced contour response across multiple scales is integrated to form structural features.

4. The intelligent analysis method for low-altitude video data as described in claim 1, characterized in that, The process of deriving camera motion parameters based on pixel changes in consecutive frames of raw low-altitude video data, and then using these parameters to perform motion correction on the video frames, includes: Stable feature point pairs are matched between consecutive frames, and affine transformation parameters are determined based on the coordinate correspondence of the feature point pairs. The affine transformation parameters are then used as camera motion parameters. The coordinate transformation of subsequent video frames is performed using affine transformation parameters to obtain the corrected video sequence. In the corrected video sequence, regions with significant pixel differences from the aligned background are separated using the inter-frame difference method and designated as foreground target regions.

5. The intelligent analysis method for low-altitude video data as described in claim 4, characterized in that, The method of separating regions with significant pixel differences from the aligned background using inter-frame difference as foreground target regions includes: Apply a morphological dilation operation to the difference region to obtain the dilated region, and then perform a morphological closing operation on the dilated region to obtain the connected component. The area value of the connected component is compared with the preset lower limit of the area, and its boundary regularity index is compared with the preset regularity threshold to filter out the connected components that satisfy both the area condition and the regularity condition. The selected connected components constitute the foreground target region.

6. The intelligent analysis method for low-altitude video data as described in claim 1, characterized in that, The process of identifying moving entities and determining their spatial positions in the foreground target region based on the reconstructed background to obtain dynamic target detection results includes: Associate the foreground target region with the motion field features at corresponding positions in the spatiotemporal joint feature sequence; Based on the continuity and consistency of the associated motion field features, a set of pixels belonging to the same moving entity is divided in the foreground target region; Each set of pixels representing a moving entity is assigned a unique identifier, and its centroid coordinates are determined as its spatial location. The set of entities carrying the identifier and spatial location constitutes the dynamic target detection result.

7. The intelligent analysis method for low-altitude video data as described in claim 1, characterized in that, The construction of a low-level behavioral representation of moving entities based on spatiotemporal joint feature sequences and dynamic target detection results includes: From the spatiotemporal joint feature sequence, extract feature segments that correspond to the spatiotemporal positions of moving entities in the dynamic target detection results; The evolution patterns of structural features and motion field features in the feature segments are analyzed, and multiple behavioral primitives, including motion direction stability, velocity change rate and regional deformation degree, are extracted. Multiple behavioral primitives are organized in chronological order to form a low-level behavioral representation of a moving entity.

8. The intelligent analysis method for low-altitude video data as described in claim 7, characterized in that, The extracted behavioral primitives include motion direction stability, velocity change rate, and regional deformation degree, including: Motion direction stability is defined based on the statistical distribution of the motion direction angle of a moving entity within a continuous time interval. The instantaneous velocity of the moving entity is obtained based on the characteristics of the sports field, and the rate of change of instantaneous velocity within adjacent time intervals is defined as the rate of change of velocity. By comparing the spatial positions of the boundary point set of the moving entity region across different frames, the region deformation degree is defined.

9. The intelligent analysis method for low-altitude video data as described in claim 1, characterized in that, The process of mapping low-level behavioral representations to high-level semantic events and evaluating these high-level semantic events according to preset rules to obtain semantic analysis results includes: Low-level behavioral representations are matched with predefined event templates, which define the key behavioral primitive combinations and their spatiotemporal logical relationships corresponding to high-level semantic events. When a low-level behavioral representation satisfies all the definition conditions of an event template, the high-level semantic event corresponding to that template is triggered. All triggered high-level semantic events and their corresponding behavioral primitive evidence are organized into a structured list of semantic events, which is the semantic analysis result.

10. A low-altitude video data intelligent analysis system, characterized in that, The system includes: The data acquisition module is used to acquire raw low-altitude video data captured by the drone while it is flying at low altitude. The feature fusion module is used to obtain the appearance features of each frame of the original low-altitude video data and the optical flow motion features between consecutive frames in parallel, and fuse the appearance features and optical flow motion features to obtain a spatiotemporal joint feature sequence. The background reconstruction determination module is used to derive camera motion parameters based on the pixel changes of consecutive frames in the original low-altitude video data, apply the camera motion parameters to perform motion correction on the video frames, distinguish the foreground target area in the corrected video sequence, and form a reconstructed background. The detection result acquisition module is used to identify moving entities in the foreground target region and determine their spatial positions based on the reconstructed background, thereby obtaining dynamic target detection results. The low-level behavior representation construction module is used to construct low-level behavior representations of moving entities based on spatiotemporal joint feature sequences and dynamic target detection results. The event evaluation module is used to map low-level behavioral representations to high-level semantic events, and evaluate the high-level semantic events according to preset rules to obtain semantic analysis results.

Images