Flight guidance symbol image recognition method

Abstract

This invention discloses a method for recognizing flight guidance symbols. The method involves binarization, weighted feature processing, and feature template searching on the display screen to identify flight path vector (FPV) symbols and flight path guidance (FPD) symbols with specific color and shape characteristics. Based on the identified symbols, reverse calculations are performed to obtain the corresponding parameter values, which are then used for monitoring and processing. This flight guidance symbol image recognition method enables high-integrity monitoring of flight guidance symbols on the cockpit display unit, ensuring data integrity during data processing and image generation.

Description

Technical Field

[0001] This invention relates to the field of image processing design for complex airborne hardware, and specifically to a method for recognizing flight guidance symbols. Background Technology

[0002] With the widespread application of highly complex commercial off-the-shelf electronic components such as CPUs and GPUs in aircraft cockpit display systems, the cockpit display unit (CMU) encompasses every stage from data processing to image generation. While the increased integration of CMU functions brings performance and cost-effectiveness, it also presents challenges to product safety and reliability. Therefore, airworthiness regulations impose high requirements on the development process of CMUs. On the one hand, the data integrity of key flight parameters within the CMU must reach less than 1.0E-8 per flight hour; on the other hand, the development assurance level of each component module within the entire CMU must reach Level B or higher. Analysis of the current CMU architecture reveals that various input data undergoes data processing by the CPU and image generation by the GPU within the CMU before being displayed on the screen. Based on generally accepted industry experience, this data processing channel only achieves an integrity level of 1.0E-6 per flight hour. Furthermore, the lack of effective historical service data for CPUs and GPUs makes it impossible to fully describe abnormal states caused by CPU and GPU failures or design errors, thus hindering an accurate assessment of their reliability and safety. Ultimately, this results in the inability to effectively guarantee the airworthiness credibility of the generated data graphics. To ensure the airworthiness confidence level of critical display data, high integrity monitoring must be designed, covering the entire data path from data processing to image generation, to demonstrate that the integrity of the data content within this path meets security requirements. To guarantee the implementation of integrity monitoring, it is necessary to design image recognition methods for different display methods of critical parameters, extract the corresponding parameter values, and monitor them to ensure that the process of generating graphics for critical parameters meets airworthiness requirements. Summary of the Invention

[0003] The purpose of this invention is to provide a method for image recognition of flight guidance symbols. This method searches for symbols with specific color and shape characteristics in the displayed screen, and performs reverse calculations based on the recognized symbols to obtain the corresponding parameter values, which are then used for monitoring and processing. This image recognition method enables high-integrity monitoring of flight guidance symbols in the cockpit display unit, ensuring data integrity during data processing and image generation.

[0004] The objective of this invention is achieved through the following technical solution:

[0005] A method for recognizing flight guidance symbols includes the following steps:

[0006] Step 1: Based on the display areas of the flight path vector (FPV) symbol and the flight path guidance (FPD) symbol, extract the corresponding target range; whereby the FPV symbol is displayed in the first color and the FPD symbol is displayed in the second color;

[0007] Step 2: Parallel identification of the center coordinates of FPV and FPD symbols; where:

[0008] When identifying the center coordinates of an FPV symbol, the first color within the target area is first binarized to obtain a first-color binarized matrix. Then, the FPV feature template is used to scan the first-color binarized matrix and record the matching value with the highest matching degree and its corresponding coordinate information (x1, y1). If the matching value of the coordinate information (x1, y1) exceeds the minimum matching threshold, it is determined to be a valid match. After obtaining valid FPV matching data, the center coordinates of the FPV symbol are calculated based on the recorded coordinate information (x1, y1).

[0009] When identifying the center coordinates of an FPD symbol, the first and second colors within the target area are first weighted. Pixels of the first color are set to 1, pixels of the second color are set to 2, and pixels of other colors are set to 0, resulting in a weighted data matrix. Then, the FPD feature template is used to scan this weighted data matrix using FPD features, and the matching value with the highest matching degree and its corresponding coordinate information (x2, y2) are recorded. If the matching value of coordinate information (x2, y2) exceeds the minimum matching threshold, the match is considered valid. After obtaining valid FPD matching data, the center coordinates of the FPD symbol are calculated based on the recorded coordinate information (x2, y2).

[0010] Step 3: Calculate the value of VerticalFD based on the center coordinates of the FPV and FPD symbols: VerticalFD is the distance between the center of the FPV symbol and the center of the FPD symbol.

[0011] Furthermore, it also includes calculating the value of LateralFD in the horizontal direction, where LateralFD is the angle of inclination of the FPD line: First, with the center of the FPD symbol as the center, scan the points of the second color within the surrounding annular area, and record the coordinate information (x, y, y) of the points of the second color within the annular area. i y i Using the coordinate information of the points of the second color, the least squares method is used to fit a straight line. The best-matching straight line function is found by minimizing the sum of squared errors. The detailed process is as follows:

[0012] 1) The sum of squared errors between the coordinates of the recorded points and the estimated values ​​on the fitted line is minimized:

[0013] s=∑[y i -f(x i )] 2 =∑[y i -(a+bx i )] 2 →min

[0014] 2) Using differentiation, we obtain a and b, where b is the slope of the line:

[0015]

[0016] a=(∑y i ) / nb(∑x i ) / n

[0017] 3) Calculate the angle of the line using inverse trigonometric functions based on the slope b:

[0018] LateralFD=α=arctan(b).

[0019] Furthermore, it also includes calculating the value of the drift angle, Drift Angle, which is the distance from the center of the circle in the FPD symbol to the perpendicular bisector of the pitch ladder: Assuming the coordinates of the aircraft center point are (x0, y0) and the roll angle is β, a virtual straight line is obtained based on the aircraft center point at an angle of (90-β). Then, combined with the center coordinates of the FPD symbol obtained in step two, the distance between each point on this virtual straight line and the center of the FPD circle is calculated, and the minimum value among these distances is recorded as the display pixel distance. The value of Drift Angle is obtained according to the conversion relationship between the display pixel distance and the parameter Drift Angle.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention utilizes the image processing capabilities and parallel computing features of FPGAs, combined with the display characteristics of flight parameters (symbols) in cockpit display systems, to customize dedicated weighted feature templates and linear fitting techniques, enabling complete monitoring of flight guidance symbols. This invention can cover every frame of the displayed image, ensuring that the graphic processing of flight guidance symbols is under monitoring, improving the completeness of flight guidance symbol display, and enhancing the airworthiness confidence of complex commercial off-the-shelf electronic devices such as CPUs and GPUs in the aviation field, thereby improving flight safety. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the flight path vector symbol, flight path guidance symbol, and horizontal parameter LateralFD.

[0023] Figure 2This is a schematic diagram of VerticalFD as a parameter in the vertical direction.

[0024] Figure 3 This is a schematic diagram of the parameter Drift Angle.

[0025] Figure 4 This is a schematic diagram showing the overlap of flight path vector symbols and flight path guidance symbols.

[0026] Figure 5 This is a diagram illustrating the weighted transformation.

[0027] Figure 6 This is a schematic diagram for recognizing flight path vector symbols.

[0028] Figure 7 A schematic diagram for identifying flight path guidance symbols.

[0029] Figure 8 This is a schematic diagram of the feature template for the FPV symbol.

[0030] Figure 9 This is a schematic diagram of the feature template for the FPD symbol.

[0031] Figure 10 A schematic diagram of the least squares linear fitting algorithm.

[0032] Figure 11 This is a flowchart of the flight guidance symbol image recognition method of the present invention. Detailed Implementation

[0033] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0034] Based on the design principles of aircraft human-machine interfaces (HMIs), flight guidance symbols from the flight control system are typically displayed using special symbols, and these symbols are displayed at a specific location within a designated area based on the current parameter input values. See also... Figures 1 to 3As shown, the flight guidance symbol includes a Flight Path Vector (FPV) symbol 14 and a Flight Path Guide (FPD) symbol 15. In this embodiment, the FPV symbol is displayed in a first color, and the FPD symbol is displayed in a second color. The flight guidance symbol also includes three parameters (LateralFD, VerticalFD, and Drift Angle), where LateralFD 11 is the angle between the two ends of the straight lines of the FPD symbol 15 and the horizontal axis of the image; VerticalFD 12 is the distance between the centers of the circles in the FPD symbol 15 and the FPV symbol 14; and Drift Angle 13 is the distance from the center of the circle in the FPD symbol 15 or the center of the FPV symbol to the perpendicular bisector. These three parameters are the target values ​​that the image recognition method shown in this embodiment needs to calculate. By using these three parameters, the graphic processing of the flight guidance symbol is monitored to ensure the integrity of the flight guidance symbol.

[0035] Based on the display characteristics of FPV symbols and FPD symbols, FPV symbols have a more advanced display layer; see [link to FPV symbols]. Figure 4 As shown, when FPV symbol 14 and FPD symbol 15 overlap, the FPV symbol will obscure the FPD symbol. Since the FPV symbol is a hollow circle, the FPD symbol will be partially obscured. Furthermore, within the entire display area, the interference information of the same color mainly includes the airport runway symbol of the first color and the thrust indicator symbol of the second color. Considering the above-mentioned obscuring and same-color interference factors, the image recognition method shown in this embodiment will use a weighted feature approach to perform feature filtering on the circular portion of the FPV symbol and the solid sphere portion of the FPD symbol within the target area.

[0036] The weighting described in this embodiment refers to a different weighting coefficient for each pixel color. Considering the shapes of the FPV and FPD symbols, in this embodiment, the weighting attribute of the first color is 1, the weighting attribute of the second color is 2, and the weighting attribute of other colors is 0. According to the above weighting rules, Figure 4 The two symbols in can be converted to Figure 5 The pattern shown in the image.

[0037] In actual FPGA processing, FPV symbols and FPD symbols are recognized in parallel. Since FPV symbols have the highest display layer, only pixels of the first color are processed during FPV symbol recognition. The binarization result is as follows: Figure 6 As shown in the figure, during FPD symbol recognition, the color weighting rule mentioned above is used for processing, and the weighting result is as follows. Figure 7 As shown in the image (this is a software illustration; values ​​0 are displayed in black, while values ​​1 and 2 are displayed in white).

[0038] Within the display areas of the FPD and FPV symbols, there are color-related interference factors. To eliminate these interference factors, this embodiment employs feature matching to extract the feature templates of the FPV and FPD symbols. The feature template for the FPV symbol is shown below. Figure 8 As shown, this is a 20x24 matrix. Inside the matrix, a ring-shaped array of points is distributed according to the circular shape of the FPV symbol, totaling 120 feature points. The feature template for the FPD symbol is shown below. Figure 9 As shown, this is a 15x16 matrix. Inside the matrix, spherical dots are distributed according to the shape of the solid sphere of the FPD symbol, totaling 200 feature points. According to the weighted attributes, the weighted sum of the 200 feature points is 400. Two templates are used for the recognition and localization of FPV and FPD symbols, respectively.

[0039] Additionally, the parameter LateralFD is represented in the image as the rotation angle of a straight line within the FPD symbol. To identify this parameter LateralFD, it is necessary to identify the second-color straight line within the FPD symbol and calculate the corresponding angle. See also Figure 10 As shown, in this implementation, a line fitting method based on the least squares approach is used to identify and calculate the line. This method requires the coordinates of the center of the solid sphere in the previous FPD symbol as input. Using these coordinates as the center, the points of the second color in the nearby annular area are scanned, and the coordinate information of these points is recorded. With radius R1 set to 63 and radius R2 set to 20, the row and column coordinates of the second-color points within the annular area are recorded. Then, the least squares approach is used to fit the line, and the slope of the line is calculated. Finally, the slope is used to calculate the inverse trigonometric function to obtain the angle of the line, which is the LateralFD angle value.

[0040] See Figure 11 As shown, the specific process of a flight guidance symbol image recognition method is as follows:

[0041] Step 1: Extract the target range: Based on the display area of ​​the FPV and FPD symbols, extract the corresponding target range, which is a square matrix of 480*480. Feature matching of FPV and FPD will be carried out simultaneously in the future.

[0042] Step 2: Parallel identification of the center coordinates of FPV and FPD symbols:

[0043] When identifying the center coordinates of the FPV symbol, the first color within the target area is first binarized. The image data within the target area is RGB video data, where bits 23-16 represent the R primary color, bits 15-8 represent the G primary color, and bits 7-0 represent the B primary color. Assuming the first color is green, the binarization formula is 0x5A <= R <= 0x6E, G >= 0xE6, B <= 0x14. Pixels that meet the above pixel value requirements are set to 1, and others are set to 0, resulting in the first color binarized matrix. Then, the FPV feature template is used to scan the first color binarized matrix using FPV features. The FPV feature template is a matrix of circles containing the circles in the FPV symbol. This matrix of circles is scanned row by row and column by column in the first color binarized data matrix, and the highest matching value and its corresponding coordinate information (x1, y1) are recorded. The circular dot matrix in the FPV feature template contains 120 points. During the matching process, a minimum matching threshold of 80 is set, meaning that if the highest recorded matching value is less than 80, the match is considered to have failed. Once valid FPV matching data is obtained, the FPV circle center coordinates are calculated based on the recorded coordinate information. Since the FPV feature template is a 20-column * 24-row FPV circle matrix, and the recorded coordinate information (x1, y1) is the lower right corner of this FPV circle matrix, the FPV circle center coordinates (x1, y1) are calculated. FPV ,y FPV ) = (x1-10, y1-12).

[0044] When identifying the center coordinates of the FPD symbol, a weighted average is applied to the first and second colors within the target area. For the RGB data of the video data, assuming the first color is green and the second color is magenta, the following weighted average is applied using the formula:

[0045] 1) When R>=0xEB, 0x23<=G<=0x41, and B>=0xEB, the data of this pixel is set to 1;

[0046] 2) When 0x5A <= R <= 0x6E, G >= 0xE6, and B <= 0x14, the data of this pixel is set to 2;

[0047] 3) In other cases, the pixel data is set to 0.

[0048] The weighted data matrix is ​​obtained using the above formula. Then, the FPD feature template is used to scan the matrix row by row and column by column, recording the highest matching value and its corresponding coordinates (x2, y2). The FPD feature template is a matrix of solid spheres containing the solid spheres in the FPD symbol. This matrix is ​​15 rows and 16 columns long, containing 200 feature points of the FPD solid spheres. Weighted according to the data, the sum of the numerical information of these feature points is 400. During the matching process, a minimum matching threshold of 280 is set; if the highest recorded matching value is less than 280, the match is considered a failure. After obtaining valid FPD matching data, the center coordinates of the FPD symbol are calculated based on the recorded coordinates. Since the FPD feature template is a 16-column * 15-row matrix, the recorded coordinates (x2, y2) are the lower right corner of this matrix. Therefore, the center coordinates of the FPD symbol (x2, y2) are... FPD ,y FPD ) = (x²-7, y²-8).

[0049] Step 3: Calculate the value of VerticalFD: According to the displayed meaning of the parameters, VerticalFD is the distance between the center of the circle in the FPV symbol and the center of the circle in the FPD symbol. Based on the center coordinates of the FPV and FPD symbols obtained in Step 2, calculate the value of VerticalFD according to the following formula:

[0050] 1) Calculate Δx and Δy for the two centers of the circle:

[0051] Δx=x FPD -x FPV ; Δy = y FPD -y FPV

[0052] 2) According to the distance calculation formula, take the square root of the sum of the squares of Δx and Δy to obtain the display pixel distance:

[0053]

[0054] 3) Based on the conversion relationship between the displayed pixel distance and the VerticalFD parameter value, the 91 rows of pixel distance represent 10 parameter units:

[0055] VerticalFD = d * 10 / 91

[0056] Step 4: Calculate the value of LateralFD: According to the displayed meaning of the parameter LateralFD, LateralFD represents the angle of inclination of the FPD line. The FPD line needs to be identified and calculated. First, using the center of the FPD symbol as the center, scan the points of the second color within the surrounding annular region. The scanning range is an annular region of 20 ≤ R ≤ 63, and record the coordinate information of the points of the second color within this annular region. Using the coordinate information of these points of the second color, perform line fitting using the least squares method. Find the best-matching line function by minimizing the sum of squared errors. The detailed process is as follows:

[0057] 1) The sum of squared errors between the coordinates of the recorded points and the estimated values ​​on the fitted line is minimized:

[0058] s=∑[y i -f(x i )] 2 =∑[y i -(a+bx i )] 2 →min

[0059] 2) Using differentiation, we obtain a and b, where b is the slope of the line:

[0060]

[0061] a=(∑y i ) / nb(∑x i ) / n

[0062] 3) Calculate the angle of the line using inverse trigonometric functions based on the slope b:

[0063] LateralFD=α=arctan(b)

[0064] Step 5: Calculate the Drift Angle value: According to the displayed meaning of the Drift Angle parameter, Drift Angle is the distance from the center of the circle in the FPD symbol to the perpendicular bisector of the pitch ladder. Since there is no actual perpendicular bisector in the image, a virtual perpendicular bisector needs to be created based on the actual roll angle and the aircraft's center point. Assuming the aircraft's center point coordinates are (x0, y0) and the roll angle is β, a straight line is obtained based on the aircraft's center point at an angle of (90-β). Then, using the center coordinates of the FPD symbol obtained in Step 2, the distance between each point on this virtual straight line and the FPD center is calculated. The minimum value among these distances is recorded as the display pixel distance.

[0065] Finally, based on the conversion relationship between the displayed pixel distance and the Drift Angle parameter value, the 91 rows of pixel distance represent 10 parameter units:

[0066] Drift Angle = d * 10 / 91

[0067] Step 6: Report VerticalFD, LateralFD, and Drift Angle data to the monitoring and comparison module.

[0068] In summary, this embodiment utilizes the parallel processing capabilities of FPGA to simultaneously scan and identify FPV and FPD symbols within a region, achieving an identification time of less than 16.7ms for each parameter.

[0069] It is understood that those skilled in the art can make equivalent substitutions or modifications to the technical solution and inventive concept of the present invention, and all such substitutions or modifications should fall within the protection scope of the appended claims.

Claims

1. A method for recognizing flight guidance symbols, characterized in that... Includes the following steps: Step 1: Based on the display areas of the flight path vector (FPV) symbol and the flight path guidance (FPD) symbol, extract the corresponding target range; whereby the FPV symbol is displayed in the first color and the FPD symbol is displayed in the second color; Step 2: Parallel identification of the center coordinates of FPV and FPD symbols; where: When identifying the center coordinates of an FPV symbol, the first color within the target area is first binarized to obtain a first-color binarized matrix. Then, the FPV feature template is used to scan the first-color binarized matrix and record the matching value with the highest matching degree and its corresponding coordinate information (x1, y1). If the matching value of the coordinate information (x1, y1) exceeds the minimum matching threshold, it is determined to be a valid match. After obtaining valid FPV matching data, the center coordinates of the FPV symbol are calculated based on the recorded coordinate information (x1, y1). When identifying the center coordinates of an FPD symbol, the first and second colors within the target area are first weighted. Pixels of the first color are set to 1, pixels of the second color are set to 2, and pixels of other colors are set to 0, resulting in a weighted data matrix. Then, the FPD feature template is used to scan this weighted data matrix using FPD features, and the matching value with the highest matching degree and its corresponding coordinate information (x2, y2) are recorded. If the matching value of coordinate information (x2, y2) exceeds the minimum matching threshold, the match is considered valid. After obtaining valid FPD matching data, the center coordinates of the FPD symbol are calculated based on the recorded coordinate information (x2, y2). Step 3: Calculate the value of VerticalFD based on the center coordinates of the FPV and FPD symbols: VerticalFD is the distance between the center of the FPV symbol and the center of the FPD symbol.

2. The method for recognizing flight guidance symbols according to claim 1, characterized in that... Also includes: Calculate the value of LateralFD in the horizontal direction, where LateralFD is the angle of inclination of the FPD line: First, using the center of the FPD symbol as the center, scan the points of the second color within the surrounding annular region and record the coordinate information of the points of the second color within this annular region. Using the coordinate information of the points of the second color, perform line fitting using the least squares method. Find the best-matching line function by minimizing the sum of squared errors. The detailed process is as follows: 1) The sum of squared errors between the coordinates of the recorded points and the estimated values ​​on the fitted line is minimized: s=∑[y i -f(x i )] 2 =∑[y i -(a+bx i )] 2 →min 2) Using differentiation, we obtain a and b, where b is the slope of the line: a=(∑y i ) / n-b(∑x i ) / n 3) Calculate the angle of the line using inverse trigonometric functions based on the slope b: LateralFD=α=arctan(b).

3. The method for recognizing flight guidance symbols according to claim 1, characterized in that... Also includes: Calculate the drift angle (Drift Angle), which is the distance from the center of the FPD symbol to the perpendicular bisector of the pitch ladder. Assuming the aircraft center point coordinates are (x0, y0) and the roll angle is β, a virtual straight line is obtained based on the aircraft center point at an angle of (90-β). Then, combining the center coordinates of the FPD symbol obtained in step two, the distance between each point on this virtual straight line and the FPD center is calculated. The minimum value among these distances is recorded as the display pixel distance. The value of Drift Angle is obtained based on the conversion relationship between the display pixel distance and the parameter Drift Angle.

Images