Remote sensing video event interpretation method, device, computer equipment and storage medium

Abstract

The present invention provides a remote sensing video event interpretation method, device, computer equipment and storage medium. The method obtains a target image by segmenting an original remote sensing image according to candidate areas, recognizes the target image through a preset category recognition model, obtains a preset classification of the candidate area, and accurately determines the candidate area based on this, thereby tracking the movement of the candidate area, obtaining speed information of the target corresponding to the candidate area, and then interpreting the behavior of the candidate area according to the preset behavior model to obtain the target dynamic event corresponding to the candidate area, thereby being able to mine the dynamic events of typical targets from the remote sensing video, effectively improving the accuracy of the event interpretation results.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a remote sensing video event interpretation method, device, computer equipment and storage medium. Background Art

[0002] Compared with satellites, video satellites add information in the time dimension, which can provide a more accurate description of dynamic events in a certain area over a period of time, and provide first-hand information for disaster relief, wartime symptom detection, and planning and decision-making. Especially in the field of national defense and military, video satellites can continuously detect and track important global ports, borders, air force bases, missile positions, railway transport stations, oil depots, large armored groups and other targets, and dynamic events of key targets such as aircraft takeoffs and landings, and ships entering and leaving ports may contain huge military strategic value. With the characteristics of staring imaging and shooting videos of global hot spots, video satellites have information advantages in the time dimension, and can detect and grasp symptom events early, providing an important source of information for grasping changes in regional situations in the first place.

[0003] Semantics refers to the meaning of events in the real world that data represents. In the traditional field of computer vision, the main information for semantic understanding includes image shape, contour, texture, color, activity, etc. The key to semantic understanding is not only to accurately locate and identify targets in images, but also to understand the interactive relationship between targets and scenes in the video. Natural understanding of visual information, future state prediction, and correlation analysis are all analytical methods for advanced semantic understanding.

[0004] However, the current application scenarios of video semantic understanding algorithms are mainly concentrated in daily video images represented by traffic target detection and human behavior recognition, and the work completed is mainly quantity detection, collision detection, and scene classification. The recognition effect for dynamic events is not very ideal, especially for small moving targets in remote sensing videos. In fact, video images are not simply a stack of continuous images. Compared with static images, they also have a time dimension. Consider using the target motion information added by the time dimension to realize the dynamic event interpretation of typical targets. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, computer equipment and storage medium for event interpretation of remote sensing video in response to the above technical problems.

[0006] A remote sensing video event interpretation method, comprising:

[0007] Acquire at least two frames of original remote sensing images of the remote sensing video;

[0008] Processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image;

[0009] Segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area;

[0010] Inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas;

[0011] Tracking each of the candidate regions containing the preset classification to obtain position information of each of the candidate regions at different time points;

[0012] Calculating the speed information of each candidate area according to the position information of each candidate area at different time points;

[0013] The behavior of the candidate area is interpreted according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

[0014] In one embodiment, the step of tracking each candidate region containing the preset classification to obtain position information of each candidate region at different time points includes:

[0015] Detecting a preset classification of each candidate region, and determining the candidate region containing the preset classification in the original remote sensing image for a continuous preset number of frames as a target candidate region;

[0016] Each of the target candidate regions is tracked to obtain position information of each of the candidate regions at different time points.

[0017] In one embodiment, the steps of processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image include:

[0018] The original remote sensing image is processed by using a frame difference method and a filtering algorithm to obtain a frame difference image containing a moving target and a background image;

[0019] According to the moving target, at least one candidate region in the remote sensing image is determined from the frame difference image.

[0020] In one embodiment, the step of processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image and determining at least one candidate region in the processed remote sensing image further includes:

[0021] The Euclidean distance of each candidate region is calculated, and candidate regions whose Euclidean distance is less than a preset distance threshold are determined to be the same candidate region.

[0022] In one embodiment, the preset category recognition model is a YOLOV4 model.

[0023] In one embodiment, the step of inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate regions includes:

[0024] Divide each of the target images into K×K grids;

[0025] The target image divided into grids is input into a preset category recognition model for classification and recognition to obtain a preset classification of each candidate area.

[0026] In one embodiment, the step of dividing each target image into a K×K grid comprises:

[0027] Perform DarkNet convolution on each of the target images, and perform RelusBlock_body processing for a preset number of times;

[0028] Each of the target images processed by RelusBlock_body is divided into K×K grids.

[0029] A remote sensing video event interpretation device, comprising:

[0030] An original remote sensing image acquisition module is used to acquire at least two frames of original remote sensing images of a remote sensing video;

[0031] A candidate region determination module, configured to process the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determine at least one candidate region in the processed remote sensing image;

[0032] A target image acquisition module, used for segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area;

[0033] A classification and recognition module, used for inputting each of the target images into a preset category recognition model for classification and recognition, and obtaining a preset classification of each of the candidate areas;

[0034] A target tracking module, used for tracking each candidate area containing the preset classification to obtain position information of each candidate area at different time points;

[0035] A speed information calculation module, used to calculate the speed information of each candidate area according to the position information of each candidate area at different time points;

[0036] The target dynamic event interpretation module is used to interpret the behavior of the candidate area according to the speed information of each candidate area and a preset behavior model to obtain the target dynamic event.

[0037] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements the following steps when executing the computer program:

[0038] Acquire at least two frames of original remote sensing images of the remote sensing video;

[0039] Processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image;

[0040] Segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area;

[0041] Inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas;

[0042] Tracking each of the candidate regions containing the preset classification to obtain position information of each of the candidate regions at different time points;

[0043] Calculating the speed information of each candidate area according to the position information of each candidate area at different time points;

[0044] The behavior of the candidate area is interpreted according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

[0045] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0046] Acquire at least two frames of original remote sensing images of the remote sensing video;

[0047] Processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image;

[0048] Segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area;

[0049] Inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas;

[0050] Tracking each of the candidate regions containing the preset classification to obtain position information of each of the candidate regions at different time points;

[0051] Calculating the speed information of each candidate area according to the position information of each candidate area at different time points;

[0052] The behavior of the candidate area is interpreted according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

[0053] The event interpretation method, device, computer equipment and storage medium of the remote sensing video obtain a target image by segmenting the original remote sensing image according to the candidate area, and recognize the target image through a preset category recognition model to obtain a preset classification of the candidate area, and accurately determine the candidate area based on this, so as to track the movement of the candidate area and obtain the speed information of the target corresponding to the candidate area, and then interpret the behavior of the candidate area according to the preset behavior model to obtain the target dynamic event corresponding to the candidate area, so as to mine the dynamic events of typical targets from the remote sensing video, and effectively improve the accuracy of the event interpretation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a flowchart of a method for interpreting events of remote sensing videos in one embodiment;

[0055] Figure 2 It is a structural block diagram of an event interpretation device for remote sensing video in one embodiment;

[0056] Figure 3 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment.

[0057] Figure 4 It is a flowchart of a method for intelligent interpretation of typical dynamic events in remote sensing videos based on motion information in one embodiment;

[0058] Figure 5 A frame difference image containing target motion information after frame difference method and filtering processing in one embodiment;

[0059] Figure 6 A schematic diagram of candidate region overlap in one embodiment;

[0060] Figure 7 A schematic diagram of Euclidean distance in one embodiment;

[0061] Figure 8 A schematic diagram of performing image segmentation on a candidate area of ​​an original remote sensing image in an embodiment;

[0062] Fig. 9A backbone extraction network of a YOLOV4 model in one embodiment;

[0063] Fig.10 A schematic diagram showing a comparison between direct detection and downsampling detection in an embodiment;

[0064] Fig.11 A schematic diagram of classification results of candidate regions identified by a YOLOV4 model in one embodiment;

[0065] Fig.12 A schematic diagram of segmented speed of a moving target in one embodiment;

[0066] Fig.13 A schematic diagram of a moving target speed analysis in one embodiment;

[0067] Fig.14 The figure shows the interpretation result of the remote sensing image in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0069] Embodiment 1

[0070] In this embodiment, Figure 1 As shown, a method for interpreting events in remote sensing videos is provided, which includes:

[0071] Step 110: Acquire at least two frames of original remote sensing images of the remote sensing video.

[0072] In this step, the remote sensing video contains multiple continuous remote sensing images, and the remote sensing image is a frame of the remote sensing video. In this embodiment, multiple remote sensing images in the remote sensing video are obtained as original remote sensing images. The multiple remote sensing images are continuous multiple frames, and there are moving targets in the images.

[0073] Step 120: Process the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determine at least one candidate region in the processed remote sensing image.

[0074] In this embodiment, a preset image processing algorithm is used to process the original remote sensing image, thereby obtaining a remote sensing image containing a moving target and a small amount of background. Since the background image is relatively fixed and the moving target is in motion between different frames, the area corresponding to the moving target is clearer and more prominent in the remote sensing image, which is convenient for extraction and confirmation.

[0075] In one embodiment, the steps of using a preset image processing algorithm to process the original remote sensing image to obtain a processed remote sensing image, and determining at least one candidate area in the processed remote sensing image include: using a frame difference method and a filtering algorithm to process the original remote sensing image to obtain a frame difference image map containing a moving target and a background image; and determining at least one candidate area in the remote sensing image from the frame difference image map based on the moving target.

[0076] In this embodiment, the remote sensing image is processed by using the frame difference method and filtering algorithm to remove most of the background and noise of the image to obtain a frame difference image map, which is the processed remote sensing image. The candidate area is the area in the image corresponding to the moving target.

[0077] It is worth mentioning that the candidate region image segmentation depends on the target motion characteristics in the remote sensing video. Since the position of the target in different time-series images is obviously different and the background image is relatively fixed, the image processed by the frame difference method and filtering only retains the target motion information and the satellite motion causes the mapping of the target area objects on the imaging element to change, which greatly reduces the detection area range. Figure 5 As shown, the moving target can be distinguished from the background image, thereby making it more accurate to determine the candidate area corresponding to the moving target according to the moving target, and effectively reducing the area range that needs to be detected in the remote sensing image, thereby improving the efficiency of selecting the candidate area.

[0078] In one embodiment, the step of using a preset image processing algorithm to process the original remote sensing image to obtain a processed remote sensing image and determining at least one candidate area in the processed remote sensing image further includes: calculating the Euclidean distance of each candidate area, and determining the candidate areas whose Euclidean distance is less than a preset distance threshold as the same candidate area.

[0079] In this embodiment, the pixel value is used as the candidate region search basis to obtain the position of the moving target candidate region in the image P, and the overlap of the candidate regions is suppressed by the Euclidean distance to reduce the number of candidate regions. Figure 6 shown.

[0080] Due to the morphological opening operation, the target is split, resulting in a large number of redundant candidate frames. According to the distribution law of typical moving targets in remote sensing images, there is no overlap of targets. When the center position of any two candidate frames is less than a certain threshold, they can be considered to be the same target, such as Figure 7As shown, (X, Y) represents the center position of different candidate boxes, d represents the Euclidean distance between the centers of two candidate boxes. According to different application scenarios, the set threshold of the Euclidean distance can be adjusted, and the center position of the preferred candidate area is stored in the list L.

[0081] In the above embodiment, multiple candidate regions whose Euclidean distance is less than a preset distance threshold are determined as the same candidate region, which can effectively reduce redundant candidate regions, reduce the calculation of candidate regions, facilitate the calculation of subsequent steps, and improve calculation efficiency.

[0082] Step 130 , segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area.

[0083] In this embodiment, after the position of the candidate region is determined, the corresponding position in the original remote sensing image is segmented based on the position of the candidate region, such as Figure 8 As shown, a color image T corresponding to the candidate area is obtained, and the color image T is the target image.

[0084] Step 140: Input each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate regions.

[0085] In this embodiment, the preset category recognition model is a pre-trained model for identifying the classification of the target image. In this way, by inputting the target image into the preset category recognition model for classification recognition, the preset classification of each candidate area can be output.

[0086] In one embodiment, the preset category recognition model is a YOLOV4 model. The target recognition module uses the YOLOV4 model, and its feature extraction network is CSPDarknet53, see Fig. 9 It mainly targets images with an input size of 416×416 or 618×618. If the original image is directly used as input, most algorithms including YOLO cannot detect small targets in high-resolution images. Fig.10 As shown, the left picture is direct detection, and the right picture is downsampling detection. It can be seen that downsampling detection will cause a decrease in image resolution.

[0087] Based on the above reasons, in this embodiment, by selecting candidate areas and performing image segmentation on the image based on the candidate areas, image detection redundancy is avoided while greatly improving detection efficiency, indirectly overcoming the difficulty of small target detection in high-resolution images.

[0088] In this embodiment, the training data sets of the YOLOV4 model are all from the moving target area detected by the Jilin-1 video satellite after frame difference processing. A total of more than 4,000 high-quality images are selected for annotation. After 50 rounds of training, relatively accurate classification and recognition of typical targets can be achieved.

[0089] In one embodiment, the step of inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas includes: dividing each of the target images into a K×K grid; inputting the target images divided into grids into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas.

[0090] In one embodiment, the step of segmenting each of the target images into a K×K grid includes: performing DarkNet convolution on each of the target images and performing a preset number of RelusBlock_body processes; and segmenting each of the target images after the RelusBlock_body process into a K×K grid.

[0091] In this embodiment, the input image is the target image T of several rectangular candidate areas obtained in the above steps, and it is divided into K×K grids, each grid is responsible for detecting the target whose center point falls within the grid. In order to prevent the input image from being distorted, images that do not meet the requirements will be surrounded by gray strips. The input target image is first subjected to ordinary DarkNet convolution, and then five RelusBlock_body images are completed to obtain the image and then divided into 13×13, 26×26, and 52×52 grids, which are used to detect objects of different sizes.

[0092] After recognition by the YOLOV4 model, the candidate area images can be accurately classified, such as Fig.11 shown.

[0093] Step 150 , tracking each of the candidate regions containing the preset classification to obtain position information of each of the candidate regions at different time points.

[0094] In this step, after the candidate regions are determined, the candidate regions containing the preset classification are tracked in the continuous multiple frames of remote sensing images. Since the position of the candidate regions changes with the movement of the moving target, the position information of the candidate regions in the remote sensing images of different frames can be obtained by tracking the candidate regions. Since each frame is arranged based on time, the position information obtained at different time points is the position information of the candidate regions in the sensing images of different frames.

[0095] In one embodiment, the step of tracking each of the candidate areas containing the preset classification to obtain the position information of each of the candidate areas at different time points includes: detecting the preset classification of each of the candidate areas, and determining the candidate areas containing the preset classification in the original remote sensing images for a preset number of consecutive frames as target candidate areas; tracking each of the target candidate areas to obtain the position information of each of the candidate areas at different time points.

[0096] In this embodiment, the preset number of frames is three frames. In this embodiment, according to the recognition results of the preset classification of each candidate area, the candidate area containing the preset classification in three consecutive frames can be regarded as the target to be tracked, determined as the target candidate area, and the position information of the target candidate area is stored in the list of targets to be tracked W. After completing the statistics of the classification results of all candidate areas, the corresponding number of CSRT trackers are initialized, and the position information of all targets to be tracked in each frame of the image is stored in the list A as the data source for the interpretation of target dynamic events.

[0097] Step 160: Calculate the speed information of each candidate area according to the position information of each candidate area at different time points.

[0098] In this embodiment, according to the position information of the candidate area in the remote sensing images of different time sequences, the speed information of the moving target corresponding to the candidate area being tracked is calculated according to the change of the position and time.

[0099] Step 170 , interpreting the behavior of the candidate area according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

[0100] In this embodiment, the preset behavior model is a pre-trained model, which is used to interpret the corresponding target dynamic event according to the speed information of the candidate area. In this embodiment, the preset behavior model is a set of dynamic event interpretation models based on speed changes and target types established by interpreting dynamic events in a large number of remote sensing images. The model is shown in Table 1:

[0101] Table 1 Remote sensing target dynamic event interpretation model

[0102]

[0103]

[0104] Under the premise of knowing the position information of each target to be tracked, the position information can be converted into the information of the target in different time periods, and the target displacement accumulation in a period of time is used to represent the average speed of the target in this period of time, such as Fig.12 shown.

[0105] The formula for converting cumulative displacement into velocity is as follows:

[0106]

[0107] Among them (x i ,y i ) represents the relative position of the target in the image at different times, di represents the Euclidean distance of the target's start and end positions in a unit time, and represents the speed information in a unit time. In order to prevent the phenomenon of large fluctuations in the speed prediction value calculated frame by frame, the method of increasing the time period is adopted to reduce the speed estimation error. It is known that the frame rate of the video is 25 frames / s, so the target displacement distance of 25 frames is used as the benchmark. The speed of each target in different time periods is calculated and stored in the target speed information database V.

[0108] Combine the target's velocity information library and the programmed interpretation model to judge the target's motion state. Take the aircraft target's motion state as an example. Fig.13 shown.

[0109] Through intuitive observation of the speed change line graph, the motion state of the aircraft target can be clearly identified. In order to achieve programmed intelligent recognition, the following rules will be formulated based on the target's speed information:

[0110]

[0111] Action indicates the operating status of the aircraft, 1 represents take-off status, -1 represents landing status, and 0 represents other status, so it is classified as taxiing status. The interpretation results of the specific images are as follows Fig.14 As shown:

[0112] The technical effects of this application include:

[0113] The present application provides an intelligent interpretation method for typical dynamic events in remote sensing videos based on target motion information and deep learning. The remote sensing video is preliminarily processed through filtering algorithms such as frame difference method, bilateral filtering, and morphological opening operation to filter out most of the background information. At the same time, the YOLOV4 model is used to assist in recognition. While completing target classification, the number of candidate areas to be tracked is reduced, and the detection of small-scale moving targets has a better detection effect.

[0114] The candidate region target association algorithm provided in the present application is based on the distribution relationship of moving targets in remote sensing videos and uses Euclidean distance as a criterion, which greatly suppresses the overlapping phenomenon of candidate frames, reduces the amount of computation for target classification, and improves the timeliness of the algorithm.

[0115] The typical moving target classification model for remote sensing images constructed in this application classifies six types of dynamic events of two types of targets based on the speed change characteristics of the targets. It can mine the dynamic events of typical targets from remote sensing videos to assist manual interpretation.

[0116] This application supports the expansion of recognition types, improves the applicability of the algorithm by further labeling the detection candidate areas, makes a more detailed division of moving targets, and expands the types of recognition, which can mine more dynamic events in remote sensing images in more scenarios.

[0117] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0118] Embodiment 2

[0119] In this embodiment, the frame difference method is used to process the input image, and most of the background and noise of the image are removed after bilateral filtering and morphological opening operations; the processed image is detected, and the candidate area image of the suspected moving target is segmented in real time; the YOLOV4 model trained with target sample data is used to classify and identify the candidate area images that may contain moving targets; CSRT is used to track the candidate areas containing the preset targets after classification and identification in real time, and their position information is recorded; the moving target position information obtained in the previous step is used to calculate the speed information of the moving target in different time periods, and the dynamic events of the target are interpreted after analyzing the target motion information.

[0120] When the candidate region image is classified as a specific target set, the candidate region image is expanded into the target sample set as a sample for further training the deep convolutional neural network in the moving target image recognition module to achieve continuous subdivision of the target category and improve the accuracy of the detection results. When the candidate region image is classified as a specific target set, the candidate region image is expanded into the target sample set as a sample for further training the deep convolutional neural network in the moving target image recognition module to achieve continuous subdivision of the target category and improve the accuracy of the detection results.

[0121] The specific steps of event interpretation of remote sensing video are as follows: Figure 4 As shown, it includes:

[0122] Step 1: Use the frame difference method and filtering algorithm to process the input remote sensing video to obtain a frame difference image P containing only the moving target and a small amount of background information. Figure 5 .

[0123] The image segmentation of the candidate area depends on the target motion characteristics in the remote sensing video. Since the position of the target in different time-series images is obviously different and the background image is relatively fixed, the image after frame difference method and filtering processing only retains the target motion information and the mapping change of the target area objects on the imaging element caused by satellite motion, which greatly reduces the detection area range.

[0124] Step 2: Use pixel value as the basis for candidate region retrieval to obtain the location of the candidate region of the moving target in the image P, and suppress the overlap of the candidate regions through the Euclidean distance to reduce the number of candidate regions. Figure 6 ;

[0125] Due to the morphological opening operation, the target is split, resulting in a large number of redundant candidate frames. According to the distribution law of typical moving targets in remote sensing images, there is no overlap of targets. When the center position of any two candidate frames is less than a certain threshold, they can be considered to be the same target, such as Figure 7 As shown;

[0126] Where (X, Y) represents the center positions of different candidate boxes, and d represents the Euclidean distance between the centers of two candidate boxes. The threshold of the Euclidean distance can be adjusted according to different application scenarios, and the center positions of the selected candidate regions are stored in the list L;

[0127] Step 3: Under the premise of obtaining the center position of the candidate region of the optimal moving target in step 2, Figure 8 As shown, an image segmentation operation is performed on the corresponding position of the original image to obtain a color image T of the candidate area;

[0128] Step 4: Use the YOLOV4 model to classify and identify the target images T of all candidate areas to obtain the target category Ti of each candidate area;

[0129] The target recognition module in this paper uses the YOLOV4 model, and its feature extraction network is CSPDarknet53. Fig. 9 It mainly targets images with an input size of 416×416 or 618×618. If the original image is directly used as input, most algorithms including YOLO cannot detect small targets in high-resolution images. Downsampling detection will cause a decrease in image resolution. Fig.10 .

[0130] In view of the above reasons, the candidate region extraction module in step 2 avoids image detection redundancy through candidate region extraction technology and image segmentation technology, while greatly improving detection efficiency, indirectly overcoming the difficulty of small target detection in high-resolution images. The input image is a number of rectangular candidate region images T obtained in step 3, and it is divided into grids. Each grid is responsible for detecting the target whose center point falls within the grid. In order to prevent the input image from being distorted, images that do not meet the requirements will be surrounded by gray strips. The input image is first subjected to ordinary DarkNet convolution, and then the image is obtained after five RelusBlock_body operations. It is divided into 13×13, 26×26, and 52×52 grids, which are used to detect objects of different sizes.

[0131] After the YOLOV4 model recognizes the candidate area images, it can accurately classify them. Fig.11 .

[0132] In this embodiment, all training data sets are from the moving target area detected by the Jilin-1 video satellite after frame difference processing. A total of more than 4,000 high-quality images are selected for annotation. After 50 rounds of training, relatively accurate classification and recognition of typical targets can be achieved.

[0133] Step 5: Continuously track the candidate areas with preset classifications after identification to complete the detection and tracking of the moving target;

[0134] Retrieve the detection and classification results of all candidate areas detected in step 4. Candidate areas that contain preset classifications in three consecutive frames can be regarded as targets to be tracked, and their position information is stored in the list of targets to be tracked W. After completing the statistics of the classification results of all candidate areas, initialize the corresponding number of CSRT trackers, and store the position information of all targets to be tracked in each frame of the image in the list A as the data source for the interpretation of target dynamic events.

[0135] Step six, after obtaining the position information of all the targets to be tracked in different time-series images, convert it into speed information, and interpret the targets according to the behavior model of typical targets.

[0136] After interpreting dynamic events in a large number of remote sensing images, a dynamic event interpretation model based on speed changes and target types was established. See Table 1

[0137] Table 1 Remote sensing target dynamic event interpretation model

[0138]

[0139]

[0140] Under the premise of knowing the position information of each target to be tracked, the position information can be converted into the information of the target in different time periods, and the average speed of the target in this period of time can be expressed by the accumulated displacement of the target in a period of time. Fig.12 .

[0141] The formula for converting cumulative displacement into velocity is as follows:

[0142]

[0143] Among them (x i ,y i ) represents the relative position of the target in the image at different times, di represents the Euclidean distance of the target's start and end positions in a unit time, and represents the speed information in a unit time. In order to prevent the phenomenon of large fluctuations in the speed prediction value calculated frame by frame, the method of increasing the time period is adopted to reduce the speed estimation error. It is known that the frame rate of the video is 25 frames / s, so the target displacement distance of 25 frames is used as the benchmark. The speed of each target in different time periods is calculated and stored in the target speed information database V.

[0144] Combine the target's velocity information library and the programmed interpretation model to judge the target's motion state. Take the aircraft target's motion state as an example. Fig.13 shown.

[0145] Through intuitive observation of the speed change line graph, the motion state of the aircraft target can be clearly identified. In order to achieve programmed intelligent recognition, the following rules will be formulated based on the target's speed information:

[0146]

[0147] Action indicates the operating status of the aircraft, 1 represents take-off status, -1 represents landing status, and 0 represents other status, so it is classified as taxiing status. The interpretation results of the specific images are as follows Fig.14 As shown:

[0148] The technical effects of this application include:

[0149] The present application provides an intelligent interpretation method for typical dynamic events in remote sensing videos based on target motion information and deep learning. The remote sensing video is preliminarily processed through filtering algorithms such as frame difference method, bilateral filtering, and morphological opening operation to filter out most of the background information. At the same time, the YOLOV4 model is used to assist in recognition. While completing target classification, the number of candidate areas to be tracked is reduced, and the detection of small-scale moving targets has a better detection effect.

[0150] The candidate region target association algorithm provided in the present application is based on the distribution relationship of moving targets in remote sensing videos and uses Euclidean distance as a criterion, which greatly suppresses the overlapping phenomenon of candidate frames, reduces the amount of computation for target classification, and improves the timeliness of the algorithm.

[0151] The typical moving target classification model for remote sensing images constructed in this application classifies six types of dynamic events of two types of targets based on the speed change characteristics of the targets. It can mine the dynamic events of typical targets from remote sensing videos to assist manual interpretation.

[0152] This application supports the expansion of recognition types, improves the applicability of the algorithm by further labeling the detection candidate areas, makes a more detailed division of moving targets, and expands the types of recognition, which can mine more dynamic events in remote sensing images in more scenarios.

[0153] Embodiment 3

[0154] In this embodiment, Figure 2 As shown, a remote sensing video event interpretation device is provided, comprising:

[0155] The remote sensing image acquisition module 210 is used to acquire at least two frames of original remote sensing images of the remote sensing video;

[0156] A candidate region determination module 220 is used to process the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determine at least one candidate region in the processed remote sensing image;

[0157] A target image acquisition module 230 is used to segment the original remote sensing image based on the candidate area to obtain a target image of the candidate area;

[0158] The classification and recognition module 240 is used to input each of the target images into a preset category recognition model for classification and recognition, and obtain a preset classification of each of the candidate areas;

[0159] The target tracking module 250 is used to track each candidate area containing the preset classification to obtain the position information of each candidate area at different time points;

[0160] A speed information calculation module 260, configured to calculate speed information of each candidate area according to position information of each candidate area at different time points;

[0161] The target dynamic event interpretation module 270 is used to interpret the behavior of the candidate area according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

[0162] In one embodiment, the target tracking module includes:

[0163] A target candidate region determination unit, configured to detect a preset classification of each candidate region, and determine the candidate region containing the preset classification in the original remote sensing image for a continuous preset number of frames as a target candidate region;

[0164] The target tracking unit is used to track each of the target candidate regions to obtain the position information of each of the candidate regions at different time points.

[0165] In one embodiment, the candidate region determination module includes:

[0166] A frame difference image acquisition unit is used to process the original remote sensing image using a frame difference method and a filtering algorithm to obtain a frame difference image containing a moving target and a background image;

[0167] The candidate region determination unit is used to determine at least one candidate region in the remote sensing image from the frame difference image according to the moving target.

[0168] In one embodiment, the remote sensing video event interpretation device further comprises:

[0169] The candidate region merging module is used to calculate the Euclidean distance of each candidate region, and determine the candidate regions whose Euclidean distance is less than a preset distance threshold as the same candidate region.

[0170] In one embodiment, the preset category recognition model is a YOLOV4 model.

[0171] In one embodiment, the classification identification module includes:

[0172] A grid segmentation acquisition unit, used for segmenting each target image into a K×K grid;

[0173] The preset classification and recognition unit is used to input the target image divided into grids into a preset category recognition model for classification and recognition to obtain the preset classification of each candidate area.

[0174] In one embodiment, the grid segmentation acquisition unit is further used to perform DarkNet convolution on each of the target images and perform RelusBlock_body processing for a preset number of times; and segment each of the target images after the RelusBlock_body processing into K×K grids.

[0175] For the specific definition of the remote sensing video event interpretation device, please refer to the definition of the remote sensing video event interpretation method above, which will not be repeated here. Each unit in the above remote sensing video event interpretation device can be implemented in whole or in part by software, hardware and a combination thereof. The above units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above units.

[0176] Embodiment 4

[0177] In this embodiment, a computer device is provided. Its internal structure diagram can be shown as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program, and the non-volatile storage medium is deployed with a database, which is used to store a preset category recognition model and a preset behavior model. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other computer devices deployed with application software. When the computer program is executed by the processor, a method for interpreting events of remote sensing video is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse, etc.

[0178] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0179] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0180] Acquire at least two frames of original remote sensing images of the remote sensing video;

[0181] Processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image;

[0182] Segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area;

[0183] Inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas;

[0184] Tracking each of the candidate regions containing the preset classification to obtain position information of each of the candidate regions at different time points;

[0185] Calculating the speed information of each candidate area according to the position information of each candidate area at different time points;

[0186] The behavior of the candidate area is interpreted according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

[0187] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0188] Detecting a preset classification of each candidate region, and determining the candidate region containing the preset classification in the original remote sensing image for a continuous preset number of frames as a target candidate region;

[0189] Each of the target candidate regions is tracked to obtain position information of each of the candidate regions at different time points.

[0190] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0191] The original remote sensing image is processed by using a frame difference method and a filtering algorithm to obtain a frame difference image containing a moving target and a background image;

[0192] According to the moving target, at least one candidate region in the remote sensing image is determined from the frame difference image.

[0193] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0194] The Euclidean distance of each candidate region is calculated, and candidate regions whose Euclidean distance is less than a preset distance threshold are determined to be the same candidate region.

[0195] In one embodiment, the preset category recognition model is a YOLOV4 model.

[0196] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0197] Divide each of the target images into K×K grids;

[0198] The target image segmented into grids is input into a preset category recognition model for classification and recognition to obtain a preset classification of each candidate area.

[0199] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0200] Perform DarkNet convolution on each of the target images, and perform RelusBlock_body processing for a preset number of times;

[0201] Each of the target images processed by RelusBlock_body is divided into K×K grids.

[0202] Embodiment 5

[0203] In this embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0204] Acquire at least two frames of original remote sensing images of the remote sensing video;

[0205] Processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image;

[0206] Segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area;

[0207] Inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas;

[0208] Tracking each of the candidate regions containing the preset classification to obtain position information of each of the candidate regions at different time points;

[0209] Calculating the speed information of each candidate area according to the position information of each candidate area at different time points;

[0210] The behavior of the candidate area is interpreted according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

[0211] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0212] Detecting a preset classification of each candidate region, and determining the candidate region containing the preset classification in the original remote sensing image for a continuous preset number of frames as a target candidate region;

[0213] Each of the target candidate regions is tracked to obtain position information of each of the candidate regions at different time points.

[0214] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0215] The original remote sensing image is processed by using a frame difference method and a filtering algorithm to obtain a frame difference image containing a moving target and a background image;

[0216] According to the moving target, at least one candidate region in the remote sensing image is determined from the frame difference image.

[0217] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0218] The Euclidean distance of each candidate region is calculated, and candidate regions whose Euclidean distance is less than a preset distance threshold are determined to be the same candidate region.

[0219] In one embodiment, the preset category recognition model is a YOLOV4 model.

[0220] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0221] Divide each of the target images into K×K grids;

[0222] The target image divided into grids is input into a preset category recognition model for classification and recognition to obtain a preset classification of each candidate area.

[0223] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0224] Perform DarkNet convolution on each of the target images, and perform RelusBlock_body processing for a preset number of times;

[0225] Each of the target images processed by RelusBlock_body is divided into K×K grids.

[0226] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0227] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0228] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A remote sensing video event interpretation method, characterized in that: include: Acquire at least two frames of original remote sensing images of the remote sensing video; Processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image; Segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area; Inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas; Tracking each of the candidate regions containing the preset classification to obtain position information of each of the candidate regions at different time points; Calculating the speed information of each candidate area according to the position information of each candidate area at different time points; The behavior of the candidate area is interpreted according to the speed information of each candidate area and a preset behavior model to obtain a target dynamic event.

2. The method according to claim 1, characterized in that The step of tracking each candidate region containing the preset classification to obtain position information of each candidate region at different time points includes: Detecting a preset classification of each candidate region, and determining the candidate region containing the preset classification in the original remote sensing image for a continuous preset number of frames as a target candidate region; Each of the target candidate regions is tracked to obtain position information of each of the candidate regions at different time points.

3. The method according to claim 1, characterized in that The steps of processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determining at least one candidate region in the processed remote sensing image include: The original remote sensing image is processed by using a frame difference method and a filtering algorithm to obtain a frame difference image containing a moving target and a background image; According to the moving target, at least one candidate region in the remote sensing image is determined from the frame difference image.

4. The method according to claim 1, characterized in that: The step of processing the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image and determining at least one candidate region in the processed remote sensing image further includes: The Euclidean distance of each candidate region is calculated, and candidate regions whose Euclidean distance is less than a preset distance threshold are determined to be the same candidate region.

5. The method according to any one of claims 1 to 4, characterized in that: The preset category recognition model is the YOLOV4 model.

6. The method according to any one of claims 1 to 4, characterized in that: The step of inputting each of the target images into a preset category recognition model for classification and recognition to obtain a preset classification of each of the candidate areas comprises: Divide each of the target images into K×K grids; The target image divided into grids is input into a preset category recognition model for classification and recognition to obtain a preset classification of each candidate area.

7. The method according to claim 6, characterized in that The step of dividing each target image into K×K grids comprises: Perform DarkNet convolution on each of the target images, and perform RelusBlock_body processing for a preset number of times; Each of the target images processed by RelusBlock_body is divided into K×K grids.

8. A remote sensing video event interpretation device, characterized in that: include: An original remote sensing image acquisition module is used to acquire at least two frames of original remote sensing images of a remote sensing video; A candidate region determination module, configured to process the original remote sensing image using a preset image processing algorithm to obtain a processed remote sensing image, and determine at least one candidate region in the processed remote sensing image; A target image acquisition module, used for segmenting the original remote sensing image based on the candidate area to obtain a target image of the candidate area; A classification and recognition module, used for inputting each of the target images into a preset category recognition model for classification and recognition, and obtaining a preset classification of each of the candidate areas; A target tracking module, used for tracking each candidate area containing the preset classification to obtain position information of each candidate area at different time points; A speed information calculation module, used to calculate the speed information of each candidate area according to the position information of each candidate area at different time points; The target dynamic event interpretation module is used to interpret the behavior of the candidate area according to the speed information of each candidate area and a preset behavior model to obtain the target dynamic event.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images