ISAR image simulation method and system based on key frame extraction

Abstract

The invention discloses an ISAR image simulation method and system based on key frame extraction, relates to the technical field of ISAR image processing, and aims to solve the problem of scarcity of inverse synthetic aperture radar image data. The method comprises the following steps: acquiring a specific video frame sequence; for each video frame image in the specific video frame sequence, extracting an edge feature value and an HSV inter-frame feature value; determining a maximum value in edge feature values or HSV inter-frame feature values corresponding to each frame of image and containing the frame and multiple frames of images in a preset number before and after the frame, judging whether the maximum value exceeds a preset threshold value or not, and determining the frame of image as a candidate key frame if the maximum value exceeds the preset threshold value; screening the plurality of candidate key frames by using a DS evidence theory to obtain a key frame set; and processing each key frame in the key frame set to obtain an ISAR image sequence. A solution is provided for the ISAR for the problem that a large number of data sets are difficult to acquire for non-cooperative target imaging, and the acquisition difficulty of radar data can be greatly reduced.

Description

Technical Field

[0001] The invention relates to the technical field of ISAR image processing, and in particular to an ISAR image simulation method and system based on key frame extraction. Background Art

[0002] Inverse Synthetic Aperture Radar (ISAR) can perform two-dimensional imaging of distant targets, so it has received much attention in the military and civilian fields. However, when using ISAR images for target recognition, the most prominent contradiction is the lack of training samples. Since ISAR often images non-cooperative targets, it is difficult to obtain a large number of target data sets to be identified, so the simulation of ISAR images for complex targets is now in the ascendant. In addition, these algorithms often require accurate three-dimensional models of the target, or optical images from multiple angles, which often also takes a lot of manpower and material resources to construct the data set.

[0003] Nowadays, the emergence of a large amount of video information has made it possible to use video data. Therefore, continuous multi-angle optical images can be obtained through video, and sequential ISAR images can be obtained by simulating these images. However, due to the large amount of video data, how to automatically process it is still a key issue. Summary of the invention

[0004] In view of the above problems, the present invention proposes an ISAR image simulation method based on key frame extraction, in an effort to solve or alleviate one or more of the above problems.

[0005] According to one aspect of the present invention, a method for simulating an ISAR image based on key frame extraction is proposed, the method comprising:

[0006] Acquire a specific video frame sequence; the specific video frame sequence is a video frame sequence containing a moving satellite, an aircraft, or a ship;

[0007] For each video frame image in a specific video frame sequence, edge feature values ​​and HSV inter-frame feature values ​​are extracted;

[0008] Determine the maximum value of edge feature values ​​of each frame of image, including the frame and a preset number of frames before and after the frame, and judge whether the maximum value exceeds a preset threshold. If so, determine the frame of image as a candidate key frame;

[0009] Determine the maximum value of the HSV inter-frame feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and judge whether the maximum value exceeds a preset threshold. If so, determine the frame of image as a candidate key frame;

[0010] DS evidence theory is used to screen multiple candidate key frames to obtain a key frame set;

[0011] Each key frame in the key frame set is processed to obtain an ISAR image sequence.

[0012] Furthermore, the extracting edge feature values ​​includes: extracting edge values ​​of the video frame image using a Canny algorithm; and performing a difference and summation between the edge values ​​of the current frame image and the next frame to obtain the edge feature values ​​of the current frame image.

[0013] Furthermore, the use of the Canny algorithm to extract edge values ​​of the video frame image includes: smoothing the video frame image using a two-dimensional Gaussian function; convolving the smoothed video frame image in the x and y directions using a Sobel convolution kernel; performing non-maximum suppression on the amplitude of the convolved video frame image, and setting the amplitude of a pixel to zero if it is not the maximum value among the surrounding pixels.

[0014] Furthermore, the extracting of HSV inter-frame feature values ​​includes: using the HSV color model to convert each video frame image to obtain its corresponding HSV value, and processing the current frame image with the brightness V of the next frame to obtain the HSV inter-frame feature values.

[0015] Furthermore, the HSV inter-frame feature value The calculation formula is:

[0016]

[0017] Where I represents the video frame image, V represents the brightness of the image, t represents the time corresponding to the video frame, p represents the pixel point of the image, and N represents the number of pixel points of the image.

[0018] Furthermore, the preset threshold is obtained by an adaptive threshold selection method.

[0019] Furthermore, the screening of multiple candidate key frames using DS evidence theory includes: calculating the DS confidence probability according to the edge feature value and HSV inter-frame feature value corresponding to each candidate key frame; if the DS confidence probability of a candidate key frame is greater than a preset confidence probability threshold, the candidate key frame is determined as a key frame.

[0020] Furthermore, the calculation formula of the DS confidence probability is:

[0021]

[0022] In the formula, m DS represents the DS confidence probability, H represents the key frame proposition, 1 / (1-K) represents the normalization coefficient, and K represents the conflict factor. The calculation formula is as follows:

[0023]

[0024] Among them, H i represents the marginal eigenvalue proposition, H j Represents the HSV inter-frame eigenvalue proposition, m 1 and m 2 They represent the confidence probabilities of edge eigenvalues ​​and HSV inter-frame eigenvalues ​​respectively.

[0025] Furthermore, the processing of each key frame in the key frame set includes: extracting edges from each key frame to obtain a key edge frame, and performing convolution on each key edge frame to obtain an ISAR image sequence.

[0026] According to another aspect of the present invention, an ISAR image simulation system based on key frame extraction is proposed, the system comprising:

[0027] A video frame sequence acquisition module, configured to acquire a specific video frame sequence; the specific video frame sequence is a video frame sequence containing a moving satellite, an airplane or a ship;

[0028] A feature value extraction module is configured to extract edge feature values ​​and HSV inter-frame feature values ​​for each video frame image in the video frame sequence;

[0029] A candidate key frame extraction module is configured to determine the maximum value of edge feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and determine whether the maximum value exceeds a preset threshold. If so, the frame of image is determined as a candidate key frame; determine the maximum value of HSV inter-frame feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and determine whether the maximum value exceeds a preset threshold. If so, the frame of image is determined as a candidate key frame;

[0030] A key frame screening module, configured to screen multiple candidate key frames using DS evidence theory to obtain a key frame set;

[0031] The key frame processing module is configured to process each key frame in the key frame set to obtain an ISAR image sequence.

[0032] The beneficial technical effects of the present invention are:

[0033] The present invention proposes an ISAR image simulation method and system based on key frame extraction, makes full use of a large number of existing video sets, designs an automatic video key frame extraction method, and converts the key frames into ISAR images, providing a solution to the problem that it is difficult to obtain a large number of data sets for ISAR imaging of non-cooperative targets, which is conducive to solving the problem of small ISAR image data sets. The present invention uses video key frame extraction technology to automatically segment key frames, which is conducive to finding target key frames. The present invention can extract targets of interest in video data, and perform inverse synthetic aperture radar data simulation on the target, which can greatly reduce the difficulty of obtaining radar data. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The present invention can be better understood by referring to the description given below in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification, and are used to further illustrate the preferred embodiments of the present invention and explain the principles and advantages of the present invention.

[0035] Figure 1 The present invention is a flowchart of an ISAR image simulation method based on key frame extraction according to an embodiment of the present invention.

[0036] Figure 2 It is an intuitive schematic diagram of the DS synthesis formula in the embodiment of the present invention.

[0037] Figure 3 2 is a representative example diagram of a video frame in an embodiment of the present invention.

[0038] Figure 4 1 is an example diagram of simulation results in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the scheme of the present invention, exemplary implementations or embodiments of the present invention will be described below in conjunction with the accompanying drawings. Obviously, the described implementations or embodiments are only implementations or embodiments of a part of the present invention, not all of them. Based on the implementations or embodiments of the present invention, all other implementations or embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present invention.

[0040] The embodiment of the present invention proposes an ISAR image simulation method based on key frame extraction, such as Figure 1 As shown, the method comprises the following steps:

[0041] S1. Acquire a specific video frame sequence; the specific video frame sequence is a video frame sequence containing a moving satellite, an airplane or a ship;

[0042] S2. For each video frame image in a specific video frame sequence, extract edge feature values ​​and HSV inter-frame feature values;

[0043] S3, determining the maximum value of edge feature values ​​of each frame of image, including the frame and a preset number of frames before and after the frame, and judging whether the maximum value exceeds a preset threshold. If so, determining the frame of image as a candidate key frame;

[0044] Determine the maximum value of the HSV inter-frame feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and judge whether the maximum value exceeds a preset threshold. If so, determine the frame of image as a candidate key frame;

[0045] S4, using DS evidence theory to screen multiple candidate key frames and obtain a key frame set;

[0046] S5. Process each key frame in the key frame set to obtain an ISAR image sequence.

[0047] The method starts at S1. In S1, a specific video frame sequence is obtained.

[0048] According to an embodiment of the present invention, the specific video frame sequence is a video frame sequence including a satellite, an airplane or a ship moving. Each frame of the video is extracted into each separate folder to obtain an image of each frame of the video, which is called a video frame image.

[0049] Then S2 is executed. In S2, for each video frame image in the specific video frame sequence, edge feature values ​​and HSV inter-frame feature values ​​are extracted.

[0050] According to an embodiment of the present invention, the edge value of each frame image is extracted, and the edge values ​​of the current frame and the next frame are differentiated and summed to obtain the edge feature value of the current frame; the HSV model is used to convert each frame image to an HSV value, and the brightness of the current frame and the next frame are differentiated, squared, summed, and squared to obtain the HSV inter-frame feature value.

[0051] In this embodiment, optionally, the Canny algorithm is used to extract edge feature values ​​of each frame image. The process is as follows:

[0052] First, use a two-dimensional Gaussian function to smooth the frame image. The two-dimensional Gaussian function is in the following form:

[0053]

[0054] Among them, x, y are the pixel coordinates, and σ is the standard deviation of the Gaussian function.

[0055] Then, the smoothed frame image is convolved in the x and y directions using the Sobel convolution kernel. The Sobel convolution kernel has the following form:

[0056]

[0057] The amplitude value of the convolved frame image is subjected to non-maximum suppression. If the pixel amplitude value is not the maximum value among the surrounding pixels, it is set to zero.

[0058] In this embodiment, optionally, the frame image is converted from the RGB model to the HSV model, wherein the brightness V is calculated as follows:

[0059]

[0060] Among them, R, G and B are the red image, green image and blue image in the frame image respectively.

[0061] In this embodiment, optionally, the brightness of the current frame and the next frame is differentiated, squared, summed, and squared to obtain the calculation formula of the HSV inter-frame feature value:

[0062]

[0063] Where I represents the video frame image, V represents the brightness of the image, t represents the time corresponding to the video frame, p represents the pixel point of the image, and N represents the number of pixel points of the image.

[0064] Then execute S3, in which the maximum value of the edge feature values ​​corresponding to each frame image, including the frame and multiple frames within a preset number of frames before and after the frame, is determined, and it is judged whether the maximum value exceeds the preset threshold. If so, the frame image is determined as a candidate key frame; the maximum value of the HSV inter-frame feature values ​​corresponding to each frame image, including the frame and multiple frames within a preset number of frames before and after the frame, is determined, and it is judged whether the maximum value exceeds the preset threshold. If so, the frame image is determined as a candidate key frame.

[0065] According to an embodiment of the present invention, the maximum value of the edge feature value within the 10 frames before and after the current frame is found, and it is determined whether the maximum value exceeds the preset threshold. If the maximum value exceeds the preset threshold, the current frame is saved as the segmentation position; if the maximum value is lower than the preset threshold, jump to the next frame; the maximum value of the HSV inter-frame feature value within the 10 frames before and after the current frame is found, and it is determined whether the maximum value exceeds the preset threshold. If the maximum value exceeds the preset threshold, the current frame is saved as the segmentation position; if the maximum value is lower than the preset threshold, jump to the next frame.

[0066] In this embodiment, optionally, the method for determining the preset threshold is an adaptive threshold selection method, and its calculation formula is:

[0067]

[0068] Among them, T d is a positive constant, μ left is the mean value in the left half window of the center point; σ left is the standard deviation in the left half window of the center point; μ right is the mean value in the right half window of the center point; σ right is the standard deviation in the window to the right of the center point.

[0069] Then, S4 is executed. In S4, multiple candidate key frames are screened using DS evidence theory to obtain a key frame set.

[0070] According to an embodiment of the present invention, the edge feature values ​​and HSV inter-frame feature values ​​of each candidate key frame are fused according to the DS evidence theory to obtain the DS confidence probability. If the DS confidence probability of a candidate key frame is greater than a preset confidence probability threshold, the candidate key frame is determined as a key frame.

[0071] The DS feature theory is used to fuse the edge feature value and the HSV inter-frame feature value to obtain the DS confidence probability, which is calculated as follows:

[0072]

[0073] In the formula, m DS represents the DS confidence probability; H represents the key frame proposition; 1 / (1-K) represents the normalization coefficient, which can ensure that the support of the new evidence body for the empty set is 0 and the sum of the support for all propositions is 1, that is, to ensure non-negativity and normalization; K represents the conflict factor, which represents the overall conflict of the evidence body, and its calculation formula is as follows:

[0074]

[0075] Among them, H i represents the marginal eigenvalue proposition, H j Represents the HSV inter-frame eigenvalue proposition, m 1 and m 2 They represent the confidence probabilities of edge eigenvalues ​​and HSV inter-frame eigenvalues ​​respectively.

[0076] m 1 and m 2 Calculated according to the following formula:

[0077]

[0078] Among them, m True Represents the probability of judging this frame as the shot boundary, arctan() is the inverse tangent function, and s is the eigenvalue value of the frame.

[0079] Figure 2 is a schematic diagram of the DS synthesis formula. The horizontal and vertical axes in the figure represent the evidence volume m 1 ,m 2 The intersection rectangle marked by the black box indicates the support degree of the fused evidence body to the proposition. If there is an item that supports the union of multiple propositions, the corresponding box of this item needs to be added. In the fusion stage, different features are fused according to different confidence levels. According to the synthesis criteria, the basic support degree of the sample for the final synthesis of each category is calculated, and a judgment is given as to whether the frame belongs to the shot boundary.

[0080] Finally, S5 is executed, in which each key frame in the key frame set is processed to obtain an ISAR image sequence.

[0081] According to an embodiment of the present invention, the Canny algorithm is used to extract the edge of each key frame to obtain a Canny edge frame, and then a two-dimensional sinc function is used to convolve each Canny edge frame to obtain a sequence ISAR image. The two-dimensional sinc function has the following form:

[0082]

[0083] Among them, x and y are the pixel coordinates.

[0084] The technical effect of the present invention is further verified through experiments.

[0085] Figure 3 is the selected video clip. Among them, frames 1-22 are meaningless interference shots, frames 23-37 are a darker shot of the back of the satellite; frames 38-68 are close-up shots of the satellite with the solar panels slowly opening; frames 69-83 are close-up shots of a part of the satellite, which will be quite different from the actual target and need to be excluded; frames 84-115 are close-up shots of the satellite. However, there is a lot of background interference in the second half of the target, which also has a great impact on the simulated ISAR image.

[0086] Figure 4 is the ISAR image generated by simulation. Figure 4 It can be seen from the figure that the present invention can generate an ISAR image sequence based on video clips.

[0087] Another embodiment of the present invention provides an ISAR image simulation system based on key frame extraction, the system comprising:

[0088] A video frame sequence acquisition module, configured to acquire a specific video frame sequence; the specific video frame sequence is a video frame sequence containing a moving satellite, an airplane or a ship;

[0089] A feature value extraction module is configured to extract edge feature values ​​and HSV inter-frame feature values ​​for each video frame image in the video frame sequence;

[0090] A candidate key frame extraction module is configured to determine the maximum value of edge feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and determine whether the maximum value exceeds a preset threshold. If so, the frame of image is determined as a candidate key frame; determine the maximum value of HSV inter-frame feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and determine whether the maximum value exceeds a preset threshold. If so, the frame of image is determined as a candidate key frame;

[0091] A key frame screening module, configured to screen multiple candidate key frames using DS evidence theory to obtain a key frame set;

[0092] The key frame processing module is configured to process each key frame in the key frame set to obtain an ISAR image sequence.

[0093] In this embodiment, preferably, the edge feature value extraction in the feature value extraction module includes: extracting the edge value of the video frame image using the Canny algorithm; performing a difference and summation between the edge value of the current frame image and the edge value of the next frame to obtain the edge feature value of the current frame image. Wherein, the edge value extraction of the video frame image using the Canny algorithm includes: smoothing the video frame image using a two-dimensional Gaussian function; convolving the smoothed video frame image in the x and y directions using a Sobel convolution kernel; performing non-maximum suppression on the amplitude of the convolved video frame image, and setting the amplitude of a pixel to zero if it is not the maximum value among the surrounding pixels.

[0094] In this embodiment, preferably, the extracting of HSV inter-frame feature values ​​in the feature value extraction module includes: using the HSV color model to convert each video frame image to obtain its corresponding HSV value, and processing the current frame image with the brightness V of the next frame to obtain the HSV inter-frame feature value. The calculation formula is:

[0095]

[0096] Where I represents the video frame image, V represents the brightness of the image, t represents the time corresponding to the video frame, p represents the pixel point of the image, and N represents the number of pixel points of the image.

[0097] In this embodiment, preferably, the key frame screening module uses the DS evidence theory to screen multiple candidate key frames, including: calculating the DS confidence probability according to the edge feature value and HSV inter-frame feature value corresponding to each candidate key frame; if the DS confidence probability of a candidate key frame is greater than a preset confidence probability threshold, then the candidate key frame is determined as a key frame.

[0098] Among them, the calculation formula of DS confidence probability is:

[0099]

[0100] In the formula, m DS represents the DS confidence probability, H represents the key frame proposition, 1 / (1-K) represents the normalization coefficient, and K represents the conflict factor. The calculation formula is as follows:

[0101]

[0102] Among them, H i represents the marginal eigenvalue proposition, H j Represents the HSV inter-frame eigenvalue proposition, m 1 and m 2 They represent the confidence probabilities of edge eigenvalues ​​and HSV inter-frame eigenvalues ​​respectively.

[0103] In this embodiment, preferably, the processing of each key frame in the key frame set in the key frame processing module includes: extracting edges from each key frame to obtain a key edge frame, and performing convolution on each key edge frame to obtain an ISAR image sequence.

[0104] The function of the ISAR image simulation system based on key frame extraction described in this embodiment can be explained by the aforementioned ISAR image simulation method based on key frame extraction. Therefore, for the parts not described in detail in this embodiment, please refer to the above method embodiments and will not be repeated here.

[0105] Although the present invention has been described according to a limited number of embodiments, it will be apparent to those skilled in the art, with the benefit of the above description, that other embodiments are contemplated within the scope of the invention thus described. The disclosure of the present invention is intended to be illustrative rather than restrictive of the scope of the invention, which is defined by the appended claims.

Claims

1. An ISAR image simulation method based on key frame extraction, characterized in that: include: Acquire a specific video frame sequence; the specific video frame sequence is a video frame sequence containing a moving satellite, an aircraft, or a ship; For each video frame image in a specific video frame sequence, edge feature values ​​and HSV inter-frame feature values ​​are extracted; Determine the maximum value of edge feature values ​​corresponding to each frame of image, including the frame and a preset number of frames before and after the frame, and judge whether the maximum value exceeds a preset threshold. If so, determine the frame of image as a candidate key frame; Determine the maximum value of the HSV inter-frame feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and judge whether the maximum value exceeds a preset threshold. If so, determine the frame of image as a candidate key frame; DS evidence theory is used to screen multiple candidate key frames to obtain a key frame set; Each key frame in the key frame set is processed to obtain an ISAR image sequence.

2. The ISAR image simulation method based on key frame extraction according to claim 1, characterized in that: The edge feature value extraction includes: extracting the edge value of the video frame image by using the Canny algorithm; performing difference and summation on the edge value of the current frame image and the edge value of the next frame to obtain the edge feature value of the current frame image.

3. The ISAR image simulation method based on key frame extraction according to claim 2, characterized in that: The method of extracting edge values ​​of a video frame image using the Canny algorithm includes: smoothing the video frame image using a two-dimensional Gaussian function; convolving the smoothed video frame image in the x and y directions using a Sobel convolution kernel; and performing non-maximum suppression on the amplitude of the convolved video frame image, and setting the amplitude of a pixel to zero if it is not the maximum value among the surrounding pixels.

4. The ISAR image simulation method based on key frame extraction according to claim 1, characterized in that: The extracting of HSV inter-frame feature values ​​includes: using the HSV color model to convert each video frame image to obtain its corresponding HSV value, and processing the current frame image and the brightness V of the next frame to obtain the HSV inter-frame feature values.

5. The ISAR image simulation method based on key frame extraction according to claim 4 is characterized in that: The HSV inter-frame feature value The calculation formula is: Where I represents the video frame image, V represents the brightness of the image, t represents the time corresponding to the video frame, p represents the pixel point of the image, and N represents the number of pixel points of the image.

6. The ISAR image simulation method based on key frame extraction according to claim 1, characterized in that: The preset threshold is obtained by an adaptive threshold selection method.

7. The ISAR image simulation method based on key frame extraction according to claim 1, characterized in that: The screening of multiple candidate key frames using DS evidence theory includes: calculating the DS confidence probability according to the edge feature value and HSV inter-frame feature value corresponding to each candidate key frame; if the DS confidence probability of a candidate key frame is greater than a preset confidence probability threshold, the candidate key frame is determined as a key frame.

8. The ISAR image simulation method based on key frame extraction according to claim 7, characterized in that: The calculation formula of the DS confidence probability is: In the formula, m DS represents the DS confidence probability, H represents the key frame proposition, 1 / (1-K) represents the normalization coefficient, and K represents the conflict factor. The calculation formula is as follows: Among them, H i represents the marginal eigenvalue proposition, H j Represents the HSV inter-frame eigenvalue proposition, m 1 and m 2 They represent the confidence probabilities of edge eigenvalues ​​and HSV inter-frame eigenvalues ​​respectively.

9. The ISAR image simulation method based on key frame extraction according to claim 1, characterized in that: The processing of each key frame in the key frame set includes: extracting edges from each key frame to obtain a key edge frame, and performing convolution on each key edge frame to obtain an ISAR image sequence.

10. An ISAR image simulation system based on key frame extraction, characterized in that: include: A video frame sequence acquisition module, configured to acquire a specific video frame sequence; the specific video frame sequence is a video frame sequence containing a moving satellite, an airplane or a ship; A feature value extraction module is configured to extract edge feature values ​​and HSV inter-frame feature values ​​for each video frame image in the video frame sequence; A candidate key frame extraction module is configured to determine the maximum value of edge feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and determine whether the maximum value exceeds a preset threshold. If so, the frame of image is determined as a candidate key frame; determine the maximum value of HSV inter-frame feature values ​​corresponding to each frame of image, including the frame and multiple frames of image within a preset number before and after the frame, and determine whether the maximum value exceeds a preset threshold. If so, the frame of image is determined as a candidate key frame; A key frame screening module, configured to screen multiple candidate key frames using DS evidence theory to obtain a key frame set; The key frame processing module is configured to process each key frame in the key frame set to obtain an ISAR image sequence.

Images