Airborne head-up display static symbol verification method based on image processing

Abstract

The airborne head-up display static symbol verification method based on image processing is suitable for verification of airborne head-up display symbols, the airborne avionics system transmits the head-up display symbols, and the airborne head-up display static symbol verification method based on image processing is suitable for verification of the airborne head-up display symbols. The method comprises the following steps: S1, identifying a head-up display symbol which is fixed in shape and has no character, and determining a corresponding coordinate position; s2, identifying the flickering alarm head-up display symbol and determining the flickering frequency; and S3, verifying the head-up display symbol according to the coordinate position and the flicker frequency. The problems that the manual judgment process of the verification result of part of airborne head-up display picture symbols is complex, and the steps are tedious are solved, and automatic verification of part of airborne head-up display symbols is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of airborne symbol verification, and in particular relates to an airborne head-up display static symbol verification method based on image processing. Background Art

[0002] The head-up display system receives information from sensors such as the atmospheric inertial navigation system, radio altimeter, flight control computer, air collision avoidance system, flight management system, ground proximity warning computer, multi-module receiver, omnidirectional beacon / pointing beacon receiver, digital rangefinder, etc., and after processing and calculation, generates a symbolic image representing the aircraft's flight information, which is projected in an equiangular manner onto a transparent combination mirror directly in front of the pilot's field of vision and imaged at infinity, thus improving the pilot's situational awareness in low visibility conditions.

[0003] In the verification of airborne HUD symbol images, most of the symbol images are judged manually. However, during the verification process of some symbols, the manual judgment process is more complicated and the steps are more cumbersome, and there are a large number of screenshots and videos of the test results, such as determining the exact location coordinates of the symbol, determining the flashing frequency of the symbol, etc., which reduces the efficiency of image processing. Summary of the invention

[0004] In view of this, the method for verifying static symbols of an airborne HUD based on image processing provided by the present invention can automatically verify some airborne HUD symbols, thereby improving the efficiency of image processing.

[0005] A method for verifying static symbols of an airborne HUD based on image processing, the method for verifying static symbols of an airborne HUD based on image processing is applicable to verifying airborne HUD symbols, the airborne avionics system transmits the HUD symbols, the method comprises:

[0006] S1: Recognize the HUD symbol with fixed shape and no characters and determine the corresponding coordinate position;

[0007] S2: Identify the flashing warning HUD symbol and determine the flashing frequency;

[0008] S3: Verify the HUD symbol based on the coordinate position and the flashing frequency.

[0009] The technical beneficial effects of the present invention are:

[0010] The present invention adopts an image recognition method, which can realize the recognition of fixed-shape and character-free symbols and determine their coordinate positions, the recognition of flashing warning symbols and determine the flashing frequency, etc. It solves the problem that the manual judgment process of the verification results of some onboard HUD screen symbols is relatively complicated and the steps are relatively cumbersome, and realizes the automatic verification of some onboard HUD symbols. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0012] Figure 1 A schematic flow chart of an airborne head-up display static symbol verification method based on image processing provided by one embodiment of the present disclosure;

[0013] Figure 2 A symbol that needs to be identified and its coordinate position determined provided in one embodiment of the present disclosure;

[0014] Figure 3 An embodiment of the present disclosure provides a warning symbol that needs to be identified and has a flashing frequency determined. DETAILED DESCRIPTION

[0015] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0016] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0017] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0018] like Figures 1 to 3 The method for verifying static symbols of an airborne HUD based on image processing is applicable to verifying the symbols of an airborne HUD. The airborne avionics system transmits the HUD symbols. The method comprises:

[0019] S1: Recognize a HUD symbol with a fixed shape and no characters and determine the corresponding coordinate position, for example, the coordinates in a screenshot in bmp format;

[0020] S2: Identify the flashing warning HUD symbol and determine the flashing frequency, such as pull-up warning, wind shear warning, etc.;

[0021] S3: Verify the HUD symbol based on the coordinate position and the flashing frequency.

[0022] As a specific implementation method provided in this case, S1 includes:

[0023] S11: pre-storing template images corresponding to different HUD symbols;

[0024] The image currently displayed by the onboard HUD is processed (the image is a screenshot sent by the processor), the read bmp format image is converted into a grayscale image, and the area where the HUD symbol with a fixed shape and no characters is located is cropped as a pre-processing image, for example, the read bmp format image is converted into a grayscale image, and the image under test is cropped according to the area where the symbol appears to remove the influence of surrounding symbols;

[0025] S12: Compare the corresponding template image with the pre-processed image through a sliding window and similarity calculation, and output a result matrix with the same size as the pre-processed image parameters, where the image parameters include an array;

[0026] S13: using the minMaxLoc function to find the maximum value and position of the array in the result matrix, the maximum value of the array is determined as the area with the highest matching degree in the area corresponding to the pre-processed image, and a rectangular box (initial image) is drawn at the corresponding position in the image and marked;

[0027] S14: Screenshot the area with the highest matching degree, use Canny edge detection to identify the outline of the symbol in the screenshot area, identify the leftmost and rightmost coordinates of the outline according to the outline of the HUD symbol and calculate the midpoint coordinates of the symbol, output the midpoint coordinates, such as positioning the symbol coordinates, screenshot the area with the highest matching degree found, set the high threshold in the hysteresis threshold processing in the Canny edge detection algorithm to 100 and the low threshold to 200, and the aperture size of the Sobel operator to 3. Convert the edge detection result into a binary image, traverse the edge map, find special points in the outline, such as edge points, corner vertices, etc., and then calculate the coordinates of the specific points required in the detected symbol through the special points in the outline.

[0028] As a specific implementation method provided in this case, S2 includes:

[0029] S21: Video preprocessing, decoding the video displayed by the HUD in MP4 format to obtain video frames and total video duration parameters, and preprocessing each frame of the image, including cropping, denoising, grayscale conversion and binarization, to obtain a preprocessed image;

[0030] S22: target symbol detection and flicker judgment, feature detection and matching are performed on the pre-processed image of each frame, wherein the homography matrix of the pre-processed image is estimated using the RANSAC algorithm and the matched feature points;

[0031] The template image is affine transformed according to the homography matrix, and the pixel difference between the transformed template image and the designated area in the pre-processed image (the designated area is the area including the HUD sign) is compared. If the pixel difference is less than or equal to the set threshold, the pre-processed image is the image when the sign is flashing; if the pixel difference is greater than the set threshold or no suitable homography transformation matrix is ​​found, the pre-processed image is the image when the sign is not flashing;

[0032] S23: Calculate the target frequency. Select a position that is one-third of the video length. Take the time when the next symbol is displayed as the starting time. Determine the time required for two consecutive flashing cycles and calculate the flashing frequency. Verify whether the design requirements of the HUD symbol are met based on the flashing frequency. Generally, the detection is performed based on the preset frequency.

[0033] As a specific implementation method provided in this case, S12 includes:

[0034] S121: setting the size of the sliding window to the size of the template image, and adjusting the sliding step size according to the template matching result and gradient information with a weight of 8:2;

[0035] S122: adding the weight of the gradient information, using the Sobel operator when calculating the gradient information and normalizing the gradient amplitude to the interval [-1, 1];

[0036] S123: Template matching uses the matchTemp l ate function to calculate the similarity between the template and the current window, and uses the standardized correlation coefficient matching method TM_CCOEFF_NORMED, such as calculating the correlation coefficient through the pixel values ​​of the scene image and the template image, and the average values ​​of the scene image sub-region and the template image, and normalizing it to the [-1,1] interval. A value close to 1 indicates that the template and the scene image sub-region are very similar, while a value close to -1 indicates that they are very dissimilar.

[0037] As a specific implementation method provided in this case, S14 includes:

[0038] S141: taking a screenshot of the district with the highest matching degree, setting the high threshold in the hysteresis threshold processing of the Canny edge detection algorithm to 100 and the low threshold to 200, and the aperture size of the Sobel operator to 3;

[0039] S142: Convert the edge detection result into a binary image, traverse the edge map, find special points in the contour, such as edge points, corner vertices, etc., and then calculate the coordinates of the specific points required in the detected symbol through the special points in the contour.

[0040] As a specific implementation method provided in this case, S22 includes:

[0041] S221: Feature detection and matching: Use SI FT feature detection algorithm to perform feature detection and feature description on the template image and the preprocessed image. Use BruteForce to match the feature descriptions of the two sets of images, and use the matching results as a good matching point set. For example, the distance between all matched feature descriptors is limited to between 0 and 1000, and the threshold is set to the larger of three times the minimum distance and 0.2. If the distance of the current matching point is less than this threshold, it is considered to be a good matching point, and all good matching points constitute a good matching point set.

[0042] S222: performing homography matrix estimation and geometric transformation, using the RANSAC algorithm to estimate the homography matrix between the template image and the well-matched point set of the preprocessed image, and performing affine transformation on the template image using a bilinear interpolation method according to the homography matrix;

[0043] S223: Similarity evaluation and flicker judgment, by comparing the pixel differences of the template image after affine transformation and the corresponding area of ​​the HUD symbol in the preprocessed image to evaluate their similarity, and judging whether it is displayed in flickering according to the similarity. The similarity is judged by setting a threshold. If it is greater than the threshold, it is judged to be the same, otherwise, it is different. For example, a similarity threshold is set. When it is less than the set threshold, this frame is considered to be in a display state with a flickering symbol. When no suitable homography transformation is found or the similarity is greater than the threshold, this frame is considered to be in a non-display state with a flickering symbol.

[0044] The present invention adopts an image recognition method, which can realize the recognition of fixed-shape and character-free symbols and determine their coordinate positions, the recognition of flashing warning symbols and determine the flashing frequency, etc. It solves the problem that the manual judgment process of the verification results of some onboard HUD screen symbols is relatively complicated and the steps are relatively cumbersome, and realizes the automatic verification of some onboard HUD symbols.

[0045] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for verifying static symbols of an airborne HUD based on image processing, which is suitable for verifying airborne HUD symbols, wherein the airborne avionics system transmits the HUD symbols, and is characterized in that: The method comprises, S1: Recognize the HUD symbol with fixed shape and no characters and determine the corresponding coordinate position; S2: Identify the flashing warning HUD symbol and determine the flashing frequency; S3: Verify the HUD symbol based on the coordinate position and the flashing frequency.

2. The method according to claim 1, characterized in that S1 includes: S11: pre-storing template images corresponding to different HUD symbols; The image currently displayed by the onboard HUD is processed, the read bmp format image is converted into a grayscale image, and the area where the HUD symbol with a fixed shape and no characters is located is cropped as a pre-processing image; S12: Compare the corresponding template image with the pre-processed image through a sliding window and similarity calculation, and output a result matrix with the same size as the pre-processed image parameters, where the image parameters include an array; S13: using the minMaxLoc function to find the maximum value and position of the array in the result matrix, the area corresponding to the pre-processed image where the maximum value of the array is located is determined as the area with the highest matching degree, and a rectangular box is drawn at the corresponding position in the image and marked; S14: taking a screenshot of the area with the highest matching degree, and using Canny edge detection on the screenshot area to identify the outline of the symbol; According to the outline of the head-up display symbol, the leftmost and rightmost coordinates of the outline are identified, the midpoint coordinates of the symbol are calculated, and the midpoint coordinates are output.

3. The method according to claim 2, characterized in that S2 includes: S21: decoding the video displayed by the HUD in MP4 format to obtain video frames and total video duration parameters, and preprocessing each frame of the image, including cropping, denoising, grayscale conversion, and binarization, to obtain a preprocessed image; S22: performing feature detection and matching on the preprocessed image of each frame, wherein the homography matrix of the preprocessed image is estimated using the RANSAC algorithm and the matched feature points; The homography matrix performs an affine transformation on the template image, and compares the pixel difference between the transformed template image and the specified area in the preprocessed image. If the pixel difference is less than or equal to the set threshold, the preprocessed image is an image when the symbol flickers; if the pixel difference is greater than the set threshold or no suitable homography transformation matrix is ​​found, the preprocessed image is an image when the symbol does not flicker; S23: Select a position that is one-third of the video length, take the time when the next symbol is displayed as the starting time, determine the time required for two consecutive flashing cycles, and calculate the flashing frequency, and verify whether the design requirements of the HUD symbol are met based on the flashing frequency.

4. The method according to claim 3, characterized in that S12 includes: S121: setting the size of the sliding window to the size of the template image, and adjusting the sliding step size according to the template matching result and gradient information with a weight of 8:2; S122: Add the weight of the gradient information, use the Sobel operator when calculating the gradient information and normalize the gradient amplitude to the interval [-1,1]; S123: Template matching uses the matchTemplate function to calculate the similarity between the template and the current window, and uses the standardized correlation coefficient matching method TM_CCOEFF_NORMED.

5. The method according to claim 4, characterized in that S14 includes: S141: Take a screenshot of the district with the highest matching degree, set the high threshold in the hysteresis threshold processing of the Canny edge detection algorithm to 100, the low threshold to 200, and the aperture size of the Sobel operator to 3; S142: Convert the edge detection result into a binary image, traverse the edge map, find the special points in the contour, and calculate the coordinates of the specific points required in the detected symbol through the special points in the contour.

6. The method according to claim 5, characterized in that S22 includes: S221: Use SIFT feature detection algorithm to perform feature detection and feature description on the template image and the preprocessed image, use brute force search matcher to match the feature descriptions of the two groups of images, and use the matching results as a good matching point set; S222: performing the homography matrix estimation and geometric transformation, using the RANSAC algorithm to estimate the homography matrix between the template image and the good matching point set of the preprocessed image, and performing an affine transformation on the template image using a bilinear interpolation method according to the homography matrix; S223: Similarity evaluation and flicker judgment, by comparing the pixel differences of the template image after affine transformation and the corresponding area of ​​the HUD symbol in the preprocessed image to evaluate their similarity, and judging whether it is displayed in flicker based on the similarity. The similarity is judged by setting a threshold. If it is greater than the threshold, it is judged to be the same, otherwise, it is different.

Images