Computer vision-based method and system for identifying aviation navigation control instruments

Abstract

The present application relates to the field of computer vision and avionics, in particular to an aircraft navigation control instrument identification method and system based on computer vision. The method comprises: acquiring image data containing multiple types of flight navigation and control instruments in the cockpit of an aircraft; performing visual analysis on the image data to obtain position information and category information of each flight navigation and control instrument; processing the image region corresponding to each instrument based on the position information, automatically identifying the character or numerical information in the instrument image region, and obtaining the corresponding instrument reading information; analyzing and processing the color information of the region where the corresponding character or number is located to determine the color semantic information corresponding to the instrument reading information; integrating the above information to generate a structured instrument identification result and outputting the same. The present application can output complete instrument information of multiple types of aircraft flight navigation and control instruments in a structured form under a complex cockpit scene, thereby improving the accuracy and reliability of aircraft instrument information acquisition.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and avionics technology, and in particular to a computer vision-based method and system for identifying aviation navigation and control instruments. Background Technology

[0002] In existing technologies, the automatic reading of cockpit or similar industrial instruments typically employs two main approaches: The first is character / region detection and recognition methods based on traditional image processing and manual features. For example, candidate regions are extracted using methods such as connected component analysis, maximum stable extremum region (MSER), and histogram of oriented gradients (HOG), and then combined with template matching or classifiers to complete character recognition. The second approach is object detection and scene text recognition methods based on deep learning. For instance, detection models such as YOLO and Faster R-CNN are used to locate target regions, and end-to-end text recognition models such as CRNN and SVTR are combined to complete character and number reading. In engineering implementation, many solutions tend to separate "instrument detection / location" and "OCR recognition" into multiple independent stages, or to perform customized training and deployment for a single instrument / single reading area.

[0003] However, the aforementioned existing technologies still suffer from the same shortcomings as the present invention in the scenario of aviation flight navigation and control instruments: First, the cockpit image background is complex, with changing viewing angles and lighting conditions. The fonts and symbols displayed on the instruments are specialized and compactly laid out. General OCR or traditional feature methods are not robust enough under conditions of low contrast, noise interference, dynamic backgrounds, and small target characters, and are prone to false detection, missed detection, and unstable recognition. Second, existing solutions are often designed for a single instrument type or a few fixed reading areas. They lack the generalized ability to uniformly detect, accurately distinguish, and collaboratively process multiple heterogeneous instruments such as PFD, ND, FCU, EFIS, and LDG in the same cockpit image, making it difficult to adapt to complete cockpit-level applications. Third, existing recognition results usually only output character values ​​and cannot simultaneously provide color information closely related to the semantics of aviation displays (such as status activation colors, alarm prompt colors, etc.), thus limiting the semantic understanding of flight status and subsequent alarm / decision support capabilities. Fourth, the fragmented process is prone to error accumulation. Even a small deviation in the detection box can significantly affect the quality of subsequent OCR input, making it difficult to balance overall end-to-end reliability and real-time performance. Summary of the Invention

[0004] To address the problems of insufficient accuracy, weak generalization ability, lack of universal models, and inability to identify key color information in existing technologies for aviation flight navigation and control instruments, this invention provides a computer vision-based intelligent identification method and system for aviation flight navigation and control instruments. This method enables end-to-end intelligent identification of various key navigation and control instruments such as PFD, ND, FCU, EFIS, and LDG, and features high accuracy, strong robustness, and high practicality.

[0005] In a first aspect, embodiments of the present invention provide a computer vision-based method for identifying aviation navigation control instruments, the method comprising:

[0006] S1. Acquire image data containing various flight navigation and control instruments within the aircraft cockpit as input; S2. Perform visual analysis processing on the image data, automatically locate the flight navigation and control instrument in the image data, determine the instrument category corresponding to the located flight navigation and control instrument, and obtain the location information and instrument category information of the located flight navigation and control instrument. S3. Based on the location information, extract the instrument image area corresponding to each flight navigation and control instrument from the image data, and automatically identify the character or number information in the instrument image area to obtain the instrument reading information corresponding to each flight navigation and control instrument; S4. Based on the automatic recognition of the character or number information, analyze and process the color information of the image area where the character or number information is located to determine the color semantic information corresponding to the instrument reading information; S5. Integrate and process the location information, instrument category information, instrument reading information and color semantic information to generate and output structured instrument recognition results.

[0007] Optionally, S1 may include: taking screen recording data from flight simulation software and synchronously acquired data from multiple cameras on a flight simulator platform as input, performing time alignment processing on the synchronously acquired data from the multiple cameras to obtain a synchronous video stream; taking the synchronous video stream as input, performing adaptive frame extraction processing based on the image change amplitude at different flight stages to obtain a candidate image set; taking the candidate image set as input, comparing and filtering the changes in instrument readings of adjacent frames, removing redundant frames with unchanged readings, and outputting an image dataset for instrument recognition.

[0008] Optionally, the generation of the image dataset may include: taking the candidate image set as input, performing data augmentation processing on each image to form an enhanced image set, wherein the data augmentation processing includes at least geometric augmentation and color augmentation; the geometric augmentation includes performing horizontal flipping processing with a preset probability and performing random rotation processing within a preset angle range; the color augmentation includes randomly selecting at least one from brightness and contrast adjustment, hue saturation and brightness adjustment, and contrast-limited adaptive histogram equalization processing; and taking the enhanced image set as output to obtain consistent input data for training and inference.

[0009] Optionally, S2 may include: using the enhanced image set as input, training a general instrument detection model based on a target detection architecture; using the cockpit image to be identified as input, calling the general instrument detection model to detect and classify multiple types of flight navigation and control instruments; and using the detection and classification results as output, obtaining the bounding box position information, instrument category information, and corresponding confidence information of each instrument.

[0010] Optionally, S3 may include: taking the bounding box position information of each instrument as input, performing region cropping processing on the original cockpit image to obtain an instrument image region; taking the instrument image region as input, sequentially performing text region detection processing and text content recognition processing to obtain character or number readings; taking the character or number readings as output, obtaining reading information corresponding to each instrument category.

[0011] Optionally, S4 may include: taking the image region where the character or number is located as input, converting the pixel color data of the image region from the original color representation to a color representation that includes hue, saturation, and brightness components; taking the converted pixel color data as input, performing clustering and grouping processing on the pixel color data, dividing the pixels into a preset number of pixel clusters, wherein the affiliation relationship between pixels and pixel clusters is adjusted iteratively to reduce the sum of the differences between each pixel and the representative color of its pixel cluster; taking each pixel cluster as input, averaging the hue, saturation, and brightness components of the pixels within each pixel cluster to obtain the representative color corresponding to each pixel cluster; taking the pixel count of each pixel cluster as input, filtering the pixel cluster with the largest number of pixels, and determining the representative color corresponding to this pixel cluster as the main color output of the region where the character or number is located.

[0012] Optionally, determining the color semantic information corresponding to the instrument reading information may include: taking a pre-constructed aviation instrument color dictionary as input, wherein hue range, saturation range, and lightness range are preset for various standard colors respectively; taking the main color as input, performing interval comparison processing on the hue, saturation, and lightness components of the main color with the hue range, saturation range, and lightness range corresponding to each standard color respectively; when each component of the main color falls into the respective range corresponding to a certain standard color, determining that the main color matches the standard color, and obtaining the standard color label corresponding to the character or number as the color semantic information output.

[0013] Secondly, embodiments of the present invention provide an aviation navigation control instrument recognition system based on computer vision, the system comprising: The input module is used to acquire image data containing various flight navigation and control instruments in the aircraft cockpit as input. The detection module, connected to the input module, is used to perform visual analysis processing on the image data, automatically locate the flight navigation and control instrument in the image data, determine the instrument category corresponding to the located flight navigation and control instrument, and obtain the location information and instrument category information of the located flight navigation and control instrument. The identification module, connected to the detection module, is used to extract the instrument image area corresponding to each flight navigation and control instrument from the image data based on the location information, and to automatically identify the character or number information in the instrument image area to obtain the instrument reading information corresponding to each flight navigation and control instrument. The analysis module, connected to the recognition module, is used to analyze and process the color information of the image area where the character or number information is located, based on the automatic recognition of the character or number information, to determine the color semantic information corresponding to the instrument reading information; The output module, connected to the detection module, recognition module, and analysis module, is used to integrate and process the location information, instrument category information, instrument reading information, and color semantic information to generate and output structured instrument recognition results.

[0014] Optionally, the analysis module may include: The color conversion unit is used to convert the color representation of the pixel color data of the image area containing the character or number as input, so as to obtain pixel color data containing hue, saturation and lightness components. A clustering processing unit, connected to the color conversion unit, is used to take the converted pixel color data as input and perform clustering and grouping processing on the pixel color data, dividing the pixels into a preset number of pixel clusters. The main color determination unit, connected to the clustering processing unit, is used to perform average processing on the pixel color components in each pixel cluster to obtain the representative color of the pixel cluster, and to select the pixel cluster with the largest number of pixels based on the number of pixels, and to determine the corresponding representative color as the main color of the region where the character or number is located.

[0015] Optionally, the analysis module may further include: The color mapping unit is used to take a pre-built aviation instrument color dictionary as input and the main color as input, and perform interval comparison processing on the hue, saturation and lightness components of the main color with the preset ranges corresponding to each standard color. The semantic determination unit, connected to the color mapping unit, is used to determine that the main color matches the standard color when each component of the main color falls within the respective range of a standard color, and outputs the corresponding standard color label as color semantic information.

[0016] The computer vision-based aviation navigation control instrument identification method and system according to the present invention has at least the following beneficial effects: This invention presents a computer vision-based method for identifying aviation navigation and control instruments. It uses image data containing multiple types of flight navigation and control instruments within the aircraft cockpit as a unified input. Through automated visual analysis of the image data, it achieves simultaneous localization and category determination of multiple instruments, enabling automatic identification of heterogeneous instruments in complex cockpit scenarios without human intervention. Furthermore, for the located instrument image areas, it automatically identifies character or numerical information and further analyzes the color information of the corresponding areas to obtain color semantic information related to the instrument display status. Finally, it integrates instrument category, position information, reading information, and color semantic information to generate a structured recognition result output. Through this technical solution, this invention effectively solves the problem in existing technologies of simultaneously achieving accurate instrument localization, stable reading recognition, and color semantic acquisition under complex backgrounds and multiple instrument coexistence conditions. It achieves end-to-end automatic acquisition of aviation navigation and control instrument information, thereby improving the completeness and reliability of flight instrument information identification.

[0017] This invention relates to a computer vision-based aviation navigation and control instrument recognition system. By constructing functional modules corresponding to the steps of the method, the system can continuously process aircraft cockpit image data and automatically output structured recognition results. The system uses an instrument detection module to automatically locate and classify various types of flight navigation and control instruments, a reading recognition module to acquire character or numerical readings from the instruments, a color semantic analysis module to extract color semantic information corresponding to the readings, and finally a result integration module to uniformly integrate and output the various recognition results. Through the collaborative work between modules, this system avoids the process fragmentation problems caused by relying on manual area division or multiple independent algorithms in existing technologies. It achieves systematic integration of multiple types of aviation instrument recognition functions, thereby improving the consistency, stability, and deployability of the system in engineering applications.

[0018] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the steps of the computer vision-based aviation navigation control instrument recognition method according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the detection and labeling of an aviation flight navigation and control instrument according to Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the labeling of the reading area of ​​an aviation flight navigation and control instrument according to Embodiment 2 of the present invention; Figure 4 This is a graph showing the training process and performance index changes of the general instrument detection model in Embodiment 2 of the present invention. Figure 5 This is a schematic diagram of the improved optical character recognition and color recognition process according to Embodiment 2 of the present invention; Figure 6 This is a block diagram of the computer vision-based aviation navigation control instrument recognition system according to Embodiment 3 of the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments. All technologies implemented based on the content of the present invention fall within the scope of protection of the present invention.

[0021] Unless otherwise specified, the terms "upper," "lower," "left," "right," "center," "inner," "outer," and "side" used in the description of specific embodiments of the present invention to indicate orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationship in which the product / equipment / device is usually placed during use. These terms are merely for the purpose of facilitating the description of the present invention or simplifying the description in specific embodiments, and for enabling those skilled in the art to quickly understand the solution, and do not indicate or imply that a particular device / component / element must have a specific orientation, or be constructed and operated in a specific positional relationship. Therefore, they should not be construed as limitations on the present invention.

[0022] In the description of the embodiments of this invention, technical terms such as "first" and "second" only distinguish one entity or operation from another, and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary or secondary relationship of the indicated technical features. In the description of the embodiments of this invention, "multiple" means two or more, unless otherwise explicitly defined.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] Example 1 During their long-term research in avionics systems and computer vision, the inventors focused on the automatic perception and information acquisition of flight navigation and control instruments in aircraft cockpits. These instruments, serving as the core human-machine interface for pilots to obtain attitude, heading, altitude, speed, and system status, encompass various types including the Primary Flight Display (PFD), Navigation Display (ND), Flight Control Unit (FCU), Electronic Flight Instrument System (EFIS), and Landing Gear Indicators (LDG), characterized by diverse display formats, compact layouts, and highly specialized display semantics. The inventors' key focus was on how to utilize computer vision and deep learning technologies to achieve unified image understanding, information extraction, and structured representation of these diverse instruments to serve flight status monitoring, simulation training, and smart cockpit applications.

[0025] In specific research and engineering practice, the inventors found that if existing general visual recognition or traditional OCR technology solutions are adopted, it is often necessary to design detection rules or models for different instrument types and different reading areas, supplemented by complex image preprocessing, region cropping, and multi-stage correction processes. The overall implementation steps are cumbersome and rely on human experience, making it difficult to maintain stable performance under complex cockpit backgrounds, different lighting conditions, and changing viewing angles. At the same time, most existing solutions only focus on the recognition results of characters or numbers themselves, making it difficult to simultaneously acquire color information closely related to the semantics of aviation displays. In engineering applications, this can only achieve limited "reading" and is insufficient to support in-depth understanding and evaluation of flight status. When solving practical engineering problems, especially when it is necessary to simultaneously meet technical goals such as high precision, multi-instrument collaboration, strong robustness, and real-time performance, existing technical solutions are clearly unable to fully meet application requirements.

[0026] Based on the above understanding, and through systematic analysis and repeated experiments on related problems, the inventors proposed a computer vision-based method and system for identifying aviation flight navigation and control instruments. This technical solution addresses complex cockpit scenarios by constructing a dedicated dataset and introducing deep learning object detection and improved optical character recognition technology to achieve unified positioning and accurate classification of multiple key instruments. Furthermore, it robustly recognizes the characters and numbers within the instruments and simultaneously extracts their color semantic information, ultimately outputting structured recognition results. Through these technical means, this invention effectively improves recognition accuracy and generalization ability, reduces system implementation complexity, and enhances engineering usability when addressing the technical problem of multi-instrument, highly complex aviation cockpit recognition, thereby achieving the technical effect of supporting application scenarios such as flight status monitoring and training evaluation.

[0027] Please refer to Figure 1 , Figure 1 This is a schematic diagram illustrating the steps of a computer vision-based aviation navigation control instrument recognition method provided in an embodiment of the present invention. The method may include: S1. Acquire image data containing various flight navigation and control instruments within the aircraft cockpit as input; S2. Perform visual analysis processing on the image data, automatically locate the flight navigation and control instrument in the image data, determine the instrument category corresponding to the located flight navigation and control instrument, and obtain the location information and instrument category information of the located flight navigation and control instrument. S3. Based on the location information, extract the instrument image area corresponding to each flight navigation and control instrument from the image data, and automatically identify the character or number information in the instrument image area to obtain the instrument reading information corresponding to each flight navigation and control instrument; S4. Based on the automatic recognition of the character or number information, analyze and process the color information of the image area where the character or number information is located to determine the color semantic information corresponding to the instrument reading information; S5. Integrate and process the location information, instrument category information, instrument reading information and color semantic information to generate and output structured instrument recognition results.

[0028] In some specific implementations, image data containing various flight navigation and control instruments within the aircraft cockpit is first acquired. Specifically, screen recording data from flight simulation software and synchronously acquired data from multiple cameras on the flight simulator platform are used as input. Time alignment processing is performed on the input data to obtain a synchronized video stream. Subsequently, using the synchronized video stream as input, adaptive frame extraction processing is performed based on the magnitude of image changes at different flight stages to obtain a candidate image set. On this basis, using the candidate image set as input, the changes in instrument readings of adjacent frames are compared and filtered, and redundant frames with unchanged readings are removed to obtain an image dataset output for subsequent processing.

[0029] After obtaining the image dataset, data augmentation processing is performed on the image dataset to improve its adaptability to changes in viewing angle and lighting. Specifically, using the image dataset as input, geometric enhancement and color enhancement are performed on each image. The geometric enhancement includes performing horizontal flipping processing with a preset probability and random rotation processing within a preset angle range. The color enhancement includes randomly selecting at least one of the following: brightness and contrast adjustment, hue saturation and brightness adjustment, and contrast-limited adaptive histogram equalization processing. The images with enhanced processing are used as output to form an enhanced image set.

[0030] After data preparation, visual analysis processing is performed on the image data to achieve automatic localization and category determination of various types of flight navigation and control instruments. Specifically, a general instrument detection model is trained using an enhanced image set as input; during the inference phase, the cockpit image to be identified is used as input, and the general instrument detection model is invoked to detect and classify various types of instruments in the image, outputting the bounding box position information, instrument category information, and corresponding confidence information of each instrument.

[0031] After obtaining the location information of each instrument, the image region corresponding to each instrument is processed based on the location information, and the character or number information within the instrument image region is automatically recognized. Specifically, using the bounding box location information as input, the original cockpit image is cropped to obtain the instrument image region; using the instrument image region as input, text region detection processing and text content recognition processing are performed sequentially to obtain the character or number readings within the instrument image region, thereby obtaining the reading information corresponding to each instrument category.

[0032] After character or number recognition is completed, the color information of the corresponding character or number area is analyzed and processed to determine color semantic information. Specifically, taking the image area where the character or number is located as input, the pixel color data of the image area is converted into a color representation method, transforming the pixel color from the original color representation method to a color representation method that includes hue, saturation, and lightness components, to obtain the converted pixel color data. Taking the converted pixel color data as input, the pixel color data is clustered and grouped, dividing the pixels into a preset number of pixel clusters. The affiliation relationship between pixels and pixel clusters is iteratively adjusted to reduce the sum of the differences between each pixel and the representative color of its pixel cluster. Taking the clustering result as input, the hue, saturation, and lightness components of the pixels in each pixel cluster are averaged to obtain the representative color corresponding to each pixel cluster. Taking the pixel count of each pixel cluster as input, the pixel cluster with the largest number of pixels is selected, and the representative color corresponding to this pixel cluster is determined as the main color output of the character or number area.

[0033] After obtaining the primary color, it is mapped to a color semantic label. Specifically, using a pre-constructed aviation instrument color dictionary as input, the dictionary presets corresponding hue, saturation, and lightness ranges for various standard colors. Using the primary color as input, the hue, saturation, and lightness components of the primary color are compared with the preset ranges corresponding to each standard color. When all components of the primary color fall within their respective ranges corresponding to a certain standard color, the primary color is determined to match that standard color, and the standard color label corresponding to that standard color is output as the color semantic information of the character or number. Through the above embodiments, automatic positioning, reading extraction, and color semantic output of various types of aviation navigation and control instruments can be achieved in complex cockpit scenarios, providing basic data for the subsequent generation of structured recognition results.

[0034] Based on the above embodiments, it can be seen that the present invention integrates the positioning, reading recognition, and color semantic extraction of multiple types of flight navigation and control instruments in aircraft cockpit images into a unified automatic processing flow. This enables stable and complete acquisition of key aircraft instrument information even under conditions such as complex backgrounds, coexistence of multiple instruments, and changes in lighting. The method adopted in this embodiment avoids the uncertainty and error accumulation problems caused by relying on manual pre-division of areas or separate processing for individual instruments in existing technologies. It achieves end-to-end automatic identification of instrument information, thereby effectively improving the accuracy, consistency, and reliability of aircraft instrument information acquisition.

[0035] Based on the aforementioned technical effects, the computer vision-based aviation navigation and control instrument recognition method provided in this invention can be applied to a wide range of technical fields, such as avionics system information processing, flight status monitoring and evaluation, and intelligent flight assistance and simulation training, including but not limited to flight simulation training systems, intelligent cockpit monitoring systems, and automatic flight data acquisition and analysis systems. In these applications, when acquiring information from flight navigation and control instruments in the cockpit, multiple types of instruments can be automatically located and their readings and color semantics identified, reducing manual intervention and human error. Simultaneously, the recognition results can be output in a structured form, facilitating data integration with subsequent monitoring, evaluation, or control systems. Through these methods, efficient, unified, and scalable processing of aviation instrument information is achieved, thereby enhancing the system's automation level and engineering application adaptability.

[0036] Example 2 Building upon the method described in Example 1, this embodiment provides a computer vision-based intelligent identification method for aviation flight navigation and control instruments to further illustrate its feasibility in specific engineering scenarios. This method addresses the complex environment of aircraft cockpits by constructing a dedicated dataset of aviation flight navigation and control instruments and combining deep learning models with optical character recognition technology to achieve automatic positioning, reading recognition, and color information acquisition for various types of flight navigation and control instruments. This embodiment focuses on providing a specific data acquisition, model training, and information fusion implementation process to illustrate the feasibility and effectiveness of the method in actual research and development and application.

[0037] In response to the scarcity of publicly available datasets for aviation flight navigation and control instruments (PFD, ND, FCU, EFIS, LDG), this embodiment has independently constructed a high-quality and diverse dedicated dataset.

[0038] First, the data acquisition module: some data comes from a real flight simulator platform. An Airbus A320-200 flight simulator was used, equipped with four 4K resolution (3840×2160), 30FPS synchronously triggered cameras. Camera 1 is fixed in front of the main instrument panel to capture PFD and ND; camera 4 is fixed directly in front of the FCU; EFIS and LDG indicator information are covered by the main instrument panel camera. All cameras are time-stamped to ensure spatiotemporal consistency of multi-view data. Some data comes from flight simulation software. Using Prepar3D (P3D) simulation software, screen recordings were made from the co-pilot's perspective, and OBS software was used to capture lossless images of instruments such as PFD, ND, and FCU, facilitating variable control and batch data generation. These two methods cover the complete flight profile, including takeoff, climb, cruise, descent, approach, and landing phases. Simultaneously, both daytime and nighttime lighting conditions were set in P3D to simulate instrument displays under different ambient light conditions, enhancing the diversity of the dataset. A total of 27 hours of video data from 18 flights was collected.

[0039] The second step is the data preprocessing module: First, adaptive frame extraction is performed on the video data, employing a phased frame extraction strategy to balance data volume and information density. A higher frame extraction rate (3 frames / second) is used during the takeoff and landing phases when data changes drastically; a lower frame extraction rate (1 frame / second) is used during the relatively stable level flight phase. Then, redundancy removal is performed. A pre-trained detection model is used to conduct preliminary analysis of consecutive frames, automatically removing redundant frames where instrument readings have not changed, retaining only images where the state has changed. This significantly improves the "information density" of the dataset, ultimately extracting tens of thousands of valid images from the original video.

[0040] After data collection is completed, the data is finely labeled and enhanced using a two-level labeling system.

[0041] First-level annotation (for instrument inspection): Use the LabelMe tool to annotate the bounding boxes of instruments such as PFD, ND, FCU, EFIS, and LDG in the image, such as... Figure 2 As shown, they are assigned standardized category labels according to Table 1. Table 1 is a description table of aviation flight navigation and control instrument categories and corresponding labels, including instrument categories such as PFD, ND, FCU, EFIS, LDG and their standardized category labels.

[0042] Table 1. Classification and Corresponding Labels of Aircraft Flight Navigation and Control Instruments

[0043] Second-level annotation (used for OCR recognition): such as... Figure 3As shown, within the located instrument area, the key text and numerical regions requiring identification (such as airspeed and altitude values ​​on the PFD; heading and distance on the ND) are further marked, and their character truth values ​​are provided. Standardized category labels are assigned according to Table 2. Table 2 is a description of the categories and labels for reading areas of aviation flight navigation and control instruments, used to explain the categories of key areas within the instruments requiring character identification and their corresponding labels. This project has completed the annotation of over 38,000 images.

[0044] Table 2. Classification and Labeling of Reading Areas for Aircraft Flight Navigation and Control Instruments

[0045] Meanwhile, to improve the model's generalization ability, an enhancement pipeline was constructed using the Albumentations library. For geometric transformations, a horizontal flip was performed with a probability of 0.5; random rotations within a ±10° range were also performed with a probability of 0.5. For color enhancement, one of three methods was selected with a probability of 0.7: random adjustment of brightness and contrast, random adjustment of hue, saturation and lightness, or application of contrast-limited adaptive histogram equalization. These enhancements effectively simulate lighting variations, viewpoint shifts, and equipment differences that may occur in a real cockpit, making the model more robust.

[0046] With a well-prepared dataset, a general instrument detection model was trained and optimized. YOLOv8 was used as the basic framework, and transfer learning was performed based on the aforementioned dataset. In deep learning-based computer vision tasks, the resolution of the input image directly determines the level of detail the model can perceive. At too low a resolution, the image is severely compressed, resulting in loss of detail. For aviation instrument recognition, this can lead to smaller numbers and status characters (such as mode indicators on the FCU) within the instrument being unable to be effectively recognized by the detector due to blurred pixel blocks. It can also cause difficulties in distinguishing instruments with similar shapes (such as PFD and ND at a certain angle) or tightly packed characters. Conversely, excessively high resolution enriches image details but also increases computational cost, requiring the model to process more pixels. This can lead to a significant drop in inference speed (FPS) and a surge in memory usage, making it difficult to meet real-time requirements. Furthermore, the model may become overly focused on irrelevant details and noise, thus reducing generalization ability. Therefore, finding a resolution that maintains both high accuracy and inference speed is a crucial step before model deployment.

[0047] For a two-stage inference system for flight instrument recognition (stage 1: instrument detection; stage 2: digit recognition), the impact of different input resolutions on the model's inference performance and recognition accuracy was investigated. The experiments also analyzed the effects of single-stage resolution adjustment and simultaneous two-stage adjustment, using a total of 98 Airbus A320 cockpit images as test data.

[0048] Experiment 1 The results of adjusting the input image resolution for the first stage (instrument detection) separately are shown in Table 3. Table 3 compares the detection and recognition performance at different input resolutions for the instrument detection stage, illustrating the comparison of detection time, number of characters detected, and recognition accuracy when adjusting the input resolution of the first stage (instrument detection) separately. When the resolution is increased from 640x640 to 960x960, the instrument detection time is significantly reduced, indicating that the YOLOv8 model achieves better feature extraction and fusion at this scale, and the total number of recognized characters increases significantly (2126 → 2174), while the OCR accuracy increases dramatically from 97.18% to 99.72%. This proves that the 960x960 resolution provides the model with a sufficiently clear field of view, enabling it to detect more and smaller text regions more accurately and recognize them more precisely. When the resolution is increased to 1280x1280, the total number of characters decreases instead of increasing (1456), and the total processing time increases significantly. As shown in Table 3, this indicates that excessively high resolution may interfere with the model's feature extraction.

[0049] Table 3. Comparison of detection and recognition performance at different instrument input resolutions during the detection stage.

[0050] Experiment 2 The results of adjusting the input image resolution for the second stage (numerical recognition) separately are shown in Table 4. Table 4 compares the recognition performance under different input resolutions for different character recognition stages, illustrating the comparison of recognition accuracy and processing time when adjusting the input resolution of the second stage (character recognition) separately. At an image resolution of 960x960, the OCR accuracy reached an astonishing 99.86%, the highest among all configurations. This further confirms the universal superiority of this resolution for text recognition tasks, and the total processing time only increased by about 8ms compared to the 640 resolution, which is completely acceptable considering the nearly 2.7 percentage point improvement in accuracy. Table 4. Comparison of recognition performance at different character recognition stages and input resolutions

[0051] Experiment 3 Table 5 shows the overall performance comparison under different combined resolution configurations for the detection and recognition stages. It illustrates the comparison results of character detection count, recognition accuracy, and overall processing time under different combinations of detection and recognition resolutions. By adjusting the input image resolutions for both stages, a combination of 960x960 and 960x960 was adopted. Compared to the baseline combination of 640x640 and 640x640, this resulted in an increase of nearly 50 additional characters detected and a 2.54 percentage point improvement in accuracy, at the cost of only about 4ms of additional processing time.

[0052] Table 5. Overall performance comparison under combined resolution configurations for the detection and recognition stages.

[0053] Based on the above experiments, the optimal input resolution for the model is 960x960 pixels. The training parameters are set as follows: initial learning rate lr0 = 0.01, final learning rate lrf = 0.0001, momentum = 0.937, and weight decay = 0.0005. A cosine annealing scheduler is used to adjust the learning rate, and an early stopping mechanism is introduced. Training is automatically terminated when the validation set performance no longer improves within a certain number of consecutive rounds to prevent overfitting and improve training efficiency.

[0054] The results of the first phase of training are as follows Figure 4 As shown, the train / box_loss, train / cls_loss, and train / dfl_loss all decrease rapidly with increasing training epochs, and stabilize after about 100 epochs, with smooth curves and no obvious oscillations. This indicates that the model's learning process on the training set is stable, and all loss branches (localization, classification, and distribution focus loss) can converge effectively.

[0055] In particular, the decreases in cls_loss (classification loss) and dfl_loss (distribution focus loss) are relatively smooth, indicating that the network has achieved ideal optimization results in both object classification and bounding box regression. At the same time, the "smooth" curve basically overlaps with the original "results" curve, indicating that the fluctuations during training are minimal, the gradient optimization is very smooth, and there are no oscillation problems caused by excessively large or small learning rates.

[0056] Furthermore, the metrics / precision(B) and metrics / recall(B) curves on the right are close to 1.0 almost early in the training process and remain stable. This means that the model can accurately identify the target class (high precision) and capture almost all positive samples on the training set (high recall). This simultaneous performance of high precision and high recall usually indicates that the model has sufficient capacity, clear separation of positive and negative samples, and high labeling quality.

[0057] The val / box_loss, val / cls_loss, and val / dfl_loss in the lower row also show a good decreasing trend. Compared with the training loss, the validation loss is slightly higher, but the convergence trend is consistent, indicating that the model does not exhibit obvious overfitting. The validation loss curve tends to plateau in the mid-to-late stages (after about the 100th epoch) and remains at a low level at the end of training, further demonstrating the model's strong generalization ability to unseen samples.

[0058] It's worth noting that the validation set loss fluctuates slightly more than the training set loss, which is normal because validation samples may exhibit randomness or differences in sample distribution during each evaluation round. However, the overall trend remains stable without sharp increases or fluctuations, indicating that the model did not degenerate or collapse throughout the training period.

[0059] Both the metrics / mAP50(B) and metrics / mAP50-95(B) curves rise rapidly and approach saturation in the very early stages (approximately 10-20 epochs), eventually stabilizing above 0.99. This is an extremely high level, meaning the model maintains excellent detection performance across different IoU thresholds. This performance demonstrates that the model can guarantee high accuracy when detecting various targets while maintaining stable performance under different overlap criteria. This reflects a high degree of matching between the network architecture and training strategies (such as anchor settings, loss functions, and data augmentation strategies).

[0060] The trained model exhibits excellent performance across all metrics on the validation set: precision and recall both exceed 0.98, and mAP@0.5 reaches over 0.99. This demonstrates that the model has successfully learned the common features of various instruments such as PFD, ND, FCU, EFIS, and LDG, enabling high-precision, real-time detection of all target instruments within a single model. This solves the problems of insufficient model generalization and the need for multiple models in existing technologies.

[0061] like Figure 5As shown, for each instrument area located by the detection model, the PaddleOCR framework is used for text recognition. The text detection algorithm is DB, and the recognition algorithm is SVTR_LCNet. This combination exhibits excellent recognition accuracy for instrument numbers and characters in complex backgrounds, achieving an accuracy of over 99% on the test set. Since aviation instrument recognition is a complex application scenario, a color recognition module is added in parallel with the standard PaddleOCR recognition process.

[0062] First, region extraction and transformation are performed, converting the detected text region image from the RGB color space (R represents the red component, G represents the green component, and B represents the blue component) to the HSV color space. This space is not sensitive to changes in lighting, making it more suitable for color analysis. The HSV space decomposes color into three components: hue (H), saturation (S), and lightness (V). Among them, the hue component is not sensitive to changes in lighting, making it very suitable for color classification.

[0063] The conversion formula is as follows:

[0064]

[0065]

[0066] Then, the K-Means clustering algorithm is used to cluster all pixels within the text region. The center value of the cluster with the largest number of pixels is taken as the dominant color of the text. This algorithm aims to cluster pixel sets... (in (The HSV vector of the pixel) is divided into Cluster To minimize the sum of squares within the cluster:

[0067] In the above formula, It is a cluster The center of mass, It is usually set to 2 to separate the text foreground color and the instrument background color.

[0068] After clustering, select the cluster with the most pixels. center of mass The HSV value is used as the primary color tone of the text.

[0069]

[0070] Finally, color mapping is performed, using a pre-defined color dictionary specifically for aviation instruments. The dictionary defines key colors. The range of values ​​in the HSV color space:

[0071] Color matching refers to determining the extracted primary color tone. Does it fall into the dictionary? A certain color Within the interval. Among them, and These represent the lower and upper limits of hue, respectively. and These represent the lower and upper limits of saturation, respectively. and These represent the lower and upper limits of brightness, respectively. The determination condition is a logical AND relationship:

[0072] if Simultaneously satisfying a certain color of The upper and lower bounds of the three components are then used to assign the color label of the text a value. The extracted primary color value is matched with the range in the dictionary, and the corresponding standard color label (such as "green" or "purple") is output. Finally, the OCR module not only outputs the recognized text (such as "100"), but also outputs the color information of the text (such as "green").

[0073] Finally, the general instrument detection model and the improved OCR recognition module are integrated to form an end-to-end processing pipeline. The system receives a cockpit image and outputs a structured JSON report, including: { "detection_results": [ { "bbox": [x, y, w, h], "class": "PFD", "confidence": 0.995, "ocr_results": [ { "class": "PFD_AVSI_VS", "text": "100", "color": "green", "position": [x1, y1, x2, y2] }, { "class": "PDF_FMA_C1", "text": "FL100", "color": "white", "position": [x1, y1, x2, y2] } / / ... other readings ] } / / ... Other instruments ]} The code example above illustrates the structured output of the method of this invention after completing the identification of aviation flight navigation and control instruments. This structured data organizes the detection results, reading identification results, and color semantic information of multiple types of instruments in a unified data format, and is used to carry and transmit the key information obtained by automatic identification.

[0074] Specifically, the structured results include a set of results for storing identification information for multiple types of instruments, where each element corresponds to a type of flight navigation or control instrument in the cockpit. For each instrument result, it includes the instrument's spatial location information in the input image, instrument category information, and detection confidence information. The location information is represented by a bounding box, characterizing the instrument's position and scale in the image; the instrument category identifies whether the instrument belongs to the main flight display, navigation display, flight control unit, or other types of instruments; and the detection confidence reflects the reliability of the instrument's positioning and classification results.

[0075] Each instrument result further includes a set of reading recognition results corresponding to that instrument. This set of reading recognition results stores multiple characters or numbers identified within the instrument area. Each reading result includes at least text content, color semantic information, and the text's position range within the image. The text content represents the specific reading or identifier information obtained through character or number recognition; the color information represents the display color semantics corresponding to the text, reflecting the display status of the instrument reading; and the text position identifies the specific location of the character or number within the instrument area for subsequent visualization annotation or result backtracking.

[0076] Through the above-mentioned structured data organization method, the present invention can integrate and output instrument detection results, character or numerical reading information and corresponding color semantic information in a unified manner, thereby providing a standardized and parsable result representation for data interfacing with flight status monitoring, training evaluation or other avionics systems.

[0077] Example 3 Based on the description of the aviation flight navigation and control instrument recognition technology solution of the present invention from a methodological perspective in Embodiments 1 and 2, this embodiment provides a computer vision-based aviation navigation and control instrument recognition system for implementing the above method. This system performs unified processing of aircraft cockpit image data to achieve automatic positioning, reading recognition, and color semantic information acquisition of multiple types of flight navigation and control instruments, and outputs the recognition results in a structured form, thereby supporting flight status monitoring, flight simulation training, and related avionics application scenarios.

[0078] like Figure 6 As shown, in this embodiment, the system includes an input module, a detection module, a recognition module, an analysis module, and an output module. These modules are connected sequentially and work collaboratively according to the data processing flow. The input module is used to acquire image data containing various flight navigation and control instruments within the aircraft cockpit. This image data can, for example, originate from screen recordings of flight simulation software or data collected by multiple cameras on a real or simulated flight platform. The image data may simultaneously include various instruments such as the Primary Flight Display (PFD), Navigation Display (ND), Flight Control Unit (FCU), Electronic Flight Instrument System (EFIS), and Landing Gear Status Indicator (LDG).

[0079] The detection module is connected to the input module and is used to perform visual analysis processing on the input cockpit image data. It automatically locates various flight navigation and control instruments from the complex cockpit background, determines the corresponding instrument categories, and outputs the position and category information of each instrument in the image. In one specific implementation, the detection module can be based on a deep learning object detection model, such as a general instrument detection model trained based on the YOLO series network structure. This model can uniformly detect and classify multiple heterogeneous instruments such as PFD, ND, FCU, EFIS, and LDG, thereby achieving parallel recognition of multiple instruments.

[0080] The recognition module is connected to the detection module and is used to crop the image area corresponding to each instrument based on the position information output by the detection module, and automatically recognize the character or number information within the instrument image area to output the corresponding instrument reading information. In a specific implementation, the recognition module can use optical character recognition (OCR) technology to recognize characters or numbers such as altitude, speed, heading, or mode identifiers within the instrument area. For example, it can use a deep learning-based OCR framework to detect and recognize the characters or numbers.

[0081] The analysis module, connected to the recognition module, is used to analyze and process the color information of the corresponding character or number area after character or number recognition, to determine the color semantic information corresponding to the instrument reading information. Specifically, the analysis module includes a color conversion unit, a clustering processing unit, and a main color determination unit. The color conversion unit converts the pixel color data of the image area containing the character or number from the original color representation to a color representation that includes hue, saturation, and lightness components. The clustering processing unit performs clustering grouping processing on the converted pixel color data. The main color determination unit determines the main color of the character or number area based on the clustering results. Furthermore, the analysis module also includes a color mapping unit and a semantic determination unit, used to match the main color with a pre-built aviation instrument color dictionary and output the corresponding standard color label, thereby achieving semantic differentiation of instrument display states such as green, white, and warning colors.

[0082] The output module is connected to the detection module, recognition module, and analysis module. It integrates and processes the instrument's category information, position information, reading information, and color semantic information to generate and output structured instrument recognition results. The structured recognition results can be organized in a unified data format for subsequent flight status analysis, training evaluation, or system interface integration.

[0083] As can be seen from the above embodiments, the system solution of the present invention is structurally consistent with the aforementioned method embodiments. By introducing a general instrument detection model and OCR recognition technology, it realizes the automated acquisition of information from multiple types of aviation flight navigation and control instruments, thereby further verifying the feasibility of the system of the present invention at the engineering implementation level.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0086] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A computer vision-based method for recognizing aviation navigation and control instruments, characterized in that, include: S1. Acquire image data containing various flight navigation and control instruments within the aircraft cockpit as input; S2. Perform visual analysis processing on the image data, automatically locate the flight navigation and control instrument in the image data, determine the instrument category corresponding to the located flight navigation and control instrument, and obtain the location information and instrument category information of the located flight navigation and control instrument. S3. Based on the location information, extract the instrument image area corresponding to each flight navigation and control instrument from the image data, and automatically identify the character or number information in the instrument image area to obtain the instrument reading information corresponding to each flight navigation and control instrument; S4. Based on the automatic recognition of the character or number information, analyze and process the color information of the image area where the character or number information is located to determine the color semantic information corresponding to the instrument reading information; S5. Integrate and process the location information, instrument category information, instrument reading information, and color semantic information to generate and output a structured instrument recognition result.

2. The computer vision-based method for identifying aviation navigation control instruments according to claim 1, characterized in that, S1 includes: taking screen recording data from flight simulation software and multi-camera synchronous acquisition data from the flight simulator platform as input, performing time alignment processing on the multi-camera synchronous acquisition data to obtain a synchronous video stream; taking the synchronous video stream as input, performing adaptive frame extraction processing according to the image change amplitude of different flight stages to obtain a candidate image set; Using the candidate image set as input, the changes in instrument readings of adjacent frames are compared and filtered, and redundant frames with unchanged readings are removed to obtain an image dataset for instrument recognition.

3. The computer vision-based method for identifying aviation navigation control instruments according to claim 2, characterized in that, The generation of the image dataset includes: taking the candidate image set as input, performing data augmentation processing on each image to form an enhanced image set, wherein the data augmentation processing includes at least geometric augmentation and color augmentation; the geometric augmentation includes performing horizontal flipping processing with a preset probability and performing random rotation processing within a preset angle range; the color augmentation includes randomly selecting at least one from brightness and contrast adjustment, hue saturation and brightness adjustment, and contrast-limited adaptive histogram equalization processing; and taking the enhanced image set as output to obtain consistent input data for training and inference.

4. The computer vision-based method for identifying aviation navigation control instruments according to claim 3, characterized in that, S2 includes: Using the enhanced image set as input, a general instrument detection model based on the target detection architecture is trained; Using the cockpit image to be identified as input, the general instrument detection model is invoked to detect and classify various types of flight navigation and control instruments; The detection and classification results are used as outputs to obtain the bounding box location information, instrument category information, and corresponding confidence information of each instrument.

5. The computer vision-based method for identifying aviation navigation control instruments according to claim 4, characterized in that, S3 includes: Using the bounding box position information of each instrument as input, the original cockpit image is cropped to obtain the instrument image region; Using the instrument image area as input, text region detection processing and text content recognition processing are performed sequentially to obtain character or number readings; using the character or number readings as output, reading information corresponding to each instrument category is obtained.

6. The computer vision-based method for identifying aviation navigation control instruments according to claim 5, characterized in that, S4 includes: taking the image region where the character or number is located as input, converting the pixel color data of the image region from the original color representation to a color representation that includes hue, saturation, and brightness components; taking the converted pixel color data as input, performing clustering and grouping processing on the pixel color data, dividing the pixels into a preset number of pixel clusters, wherein the affiliation relationship between pixels and pixel clusters is adjusted iteratively to reduce the sum of the differences between each pixel and the representative color of its pixel cluster; taking each pixel cluster as input, averaging the hue, saturation, and brightness components of the pixels within each pixel cluster to obtain the representative color corresponding to each pixel cluster; taking the pixel count of each pixel cluster as input, filtering the pixel cluster with the largest number of pixels, and determining the representative color corresponding to this pixel cluster as the main color output of the region where the character or number is located.

7. The computer vision-based method for identifying aviation navigation control instruments according to claim 6, characterized in that, The process of determining the color semantic information corresponding to the instrument reading information includes: taking a pre-constructed aviation instrument color dictionary as input, wherein hue range, saturation range and lightness range are preset for various standard colors respectively; taking the main color as input, performing interval comparison processing on the hue, saturation and lightness components of the main color with the hue range, saturation range and lightness range corresponding to each standard color respectively; when each component of the main color falls into the respective range corresponding to a certain standard color, determining that the main color matches the standard color, and obtaining the standard color label corresponding to the character or number as the color semantic information output.

8. A computer vision-based aviation navigation and control instrument recognition system, characterized in that, include: The input module is used to acquire image data containing various flight navigation and control instruments in the aircraft cockpit as input. The detection module, connected to the input module, is used to perform visual analysis processing on the image data, automatically locate the flight navigation and control instrument in the image data, determine the instrument category corresponding to the located flight navigation and control instrument, and obtain the location information and instrument category information of the located flight navigation and control instrument. The identification module, connected to the detection module, is used to extract the instrument image area corresponding to each flight navigation and control instrument from the image data based on the location information, and to automatically identify the character or number information in the instrument image area to obtain the instrument reading information corresponding to each flight navigation and control instrument. The analysis module, connected to the recognition module, is used to analyze and process the color information of the image area where the character or number information is located, based on the automatic recognition of the character or number information, to determine the color semantic information corresponding to the instrument reading information; The output module, connected to the detection module, recognition module, and analysis module, is used to integrate and process the location information, instrument category information, instrument reading information, and color semantic information to generate and output structured instrument recognition results.

9. The computer vision-based aviation navigation control instrument recognition system according to claim 8, characterized in that... The analysis module includes: The color conversion unit is used to convert the color representation of the pixel color data of the image area containing the character or number as input, so as to obtain pixel color data containing hue, saturation and lightness components. A clustering processing unit, connected to the color conversion unit, is used to take the converted pixel color data as input and perform clustering and grouping processing on the pixel color data, dividing the pixels into a preset number of pixel clusters. The main color determination unit, connected to the clustering processing unit, is used to perform average processing on the pixel color components in each pixel cluster to obtain the representative color of the pixel cluster, and to select the pixel cluster with the largest number of pixels based on the number of pixels, and to determine the corresponding representative color as the main color of the region where the character or number is located.

10. The computer vision-based aviation navigation control instrument recognition system according to claim 9, characterized in that, The analysis module also includes: The color mapping unit is used to take a pre-built aviation instrument color dictionary as input and the main color as input, and perform interval comparison processing on the hue, saturation and lightness components of the main color with the preset ranges corresponding to each standard color. The semantic determination unit, connected to the color mapping unit, is used to determine that the main color matches the standard color when each component of the main color falls within the respective range of a standard color, and outputs the corresponding standard color label as color semantic information.

Images