Non-intrusive aircraft cockpit virtual instrument panel recognition method

Abstract

A non-invasive virtual instrument panel recognition method for aircraft cockpits involves acquiring high-resolution color images of the cockpit, preprocessing them, calculating the second derivative of the images to filter out the instrument panel pointers, and then filtering out connected components through image binarization. Based on the connected components, the instrument panel scales are obtained and fitted to define the scale area. After filtering the circular areas containing the instrument panel scales and performing horizontal correction, the numerical portions are obtained. Finally, the scale values ​​are obtained through text recognition. This invention solves the problem of detecting instrument panel scale values ​​when the pointer position is fixed while the scale changes dynamically relative to the pointer, resulting in more accurate detection results.

Description

Technical Field

[0001] This invention relates to a technology in the field of image processing, specifically a non-invasive method for recognizing virtual instrument panels in an aircraft cockpit. Background Technology

[0002] One crucial aspect of aircraft autopilot systems is the recognition of information displayed on screens, which can generally be achieved using existing optical character recognition (OCR) methods. However, for virtual instrument panels displayed on screens, the pointer position is fixed while the scale changes dynamically relative to the pointer, making it impossible to determine the pointer's rotation angle, thus causing recognition difficulties. Therefore, it is necessary to design instrument panel recognition methods specifically tailored to the characteristics of virtual instrument panels.

[0003] The shortcomings of existing virtual dial pointer reading recognition technology include: pointerless instrument images cannot eliminate all linear interference, so the method of measuring the radius of the instrument mirror may result in scale misidentification; and it cannot handle situations where the pointer position is fixed and the virtual dial rolls, thus changing the reading. Summary of the Invention

[0004] To address the aforementioned shortcomings of existing technologies, this invention proposes a non-intrusive virtual instrument panel recognition method for aircraft cockpits. This method solves the problem of detecting instrument panel scale values ​​when the pointer position is fixed while the scale changes dynamically relative to the pointer, resulting in more accurate detection results.

[0005] This invention is achieved through the following technical solution:

[0006] This invention relates to a non-intrusive method for recognizing virtual instrument panels in an aircraft cockpit. The method involves acquiring high-definition color images of the cockpit, preprocessing them, calculating the second derivative of the images to filter out instrument panel pointers, and then filtering out connected components through image binarization. Based on the connected components, the instrument panel scales are obtained, and the scale area is fitted. After filtering the circular area containing the instrument panel scales, the numerical portion is obtained through horizontal correction. Finally, the scale values ​​are obtained through text recognition.

[0007] Preferably, when acquiring high-definition color images, the entire aircraft cockpit display screen should fill the entire image as much as possible, and the scales, numbers, and virtual pointers in the virtual instrument panel should be clear. The four vertices of the display screen are marked, and the transition matrix is ​​calculated based on the known conditions of the four vertices when viewed directly from the camera. An affine transformation is performed on the display screen image, and a quadratic interpolation method is used to fill in the pixels to obtain a new image.

[0008] The preprocessing includes: correcting and segmenting the high-definition color image by manual annotation to obtain the location of the instrument to be identified; binarizing the segmented image; and performing Gaussian smoothing on the binarized image to remove noise.

[0009] The aforementioned dial scale area is determined by calculating the second derivative of the image to select a suitable set of straight lines, which is then fitted using least squares fitting. The fitting result serves as the pointer of the virtual dashboard to be detected. Connected components are selected from the binarized image to represent the areas where the dashboard scales are located. The center points of the outer rectangles containing the scale values ​​are fitted to obtain the dial scale areas within the virtual dashboard. Specifically, this involves: sorting the connected components in ascending order of size, selecting appropriate proportions from the sorted connected components as scale areas, and then sorting the coordinates of the upper left corner of each scale area to obtain a set of scale areas; obtaining the outer rectangles of each area in the scale area set that are parallel to the image coordinate axes, and obtaining the coordinates of the center points of each outer rectangle; and using a minimum distance fitting error algorithm to minimize the distance from all center points to the circle to be fitted. The circle formed by the scale region is obtained by iteratively solving the gradient descent algorithm; the boundary of the fitted circle is expanded to construct a ring, so that the text to be detected is contained inside the ring, and the intersection of this ring region with the original image is calculated to form a fan-shaped or ring-shaped dial scale region.

[0010] The calculation of the second derivative of the image refers to: after setting high and low thresholds, if the second derivative value of a point in the image is greater than the high threshold, then the point is selected; if it is less than the low threshold, then the point is discarded; when it is between the two, the possible straight line region is set by setting parameters such as the pixel width of the line to be detected, and if a point in the image is in the possible straight line region, then the point is selected.

[0011] The pointer of the virtual dashboard to be detected is obtained by filtering all the obtained straight lines according to their lengths to get a set of straight lines with a length greater than a set length. The set of straight lines is then fitted using a weighted least squares method, and the influence of outliers is reduced by the Tukey method. The resulting unique straight line is the pointer of the virtual dashboard.

[0012] The digital portion involves calculating the connected components of the dial scale area, and based on the characteristic that the connected components of the scale values ​​are similar in size, discarding the scale portion, using the remaining connected components as the digital portion, merging adjacent digital portions, and rotating the digital portion to a horizontal orientation. Specifically, this includes:

[0013] i) Find the intersection of the annular region and the original image to obtain the possible digital regions.

[0014] ii) In the binarized image, remove multiple connected regions of similar size, select the remaining connected regions as digital regions, and sort these regions according to the position of the top-left vertex.

[0015] iii) Traverse the sorted number region and merge regions with adjacent center points into the same region.

[0016] iv) Select a digital region, calculate the angle α between the straight line connecting the center point of its outer rectangle and the circle and the fitted straight line, and rotate the digital region to the horizontal direction according to the angle α.

[0017] v) Enhance the image of the digital region.

[0018] The text recognition refers to: inputting the numerical part into the text detector to obtain the detection result of the number, and combining the detection result of the number with its deflection angle α to calculate the scale value pointed to by the current pointer.

[0019] This invention relates to a system for implementing the above-mentioned method, comprising: an image acquisition unit, an image preprocessing unit, an instrument panel pointer detection unit, an instrument panel circle fitting unit, and an instrument panel scale value detection unit, wherein: the image acquisition unit acquires high-definition aircraft cockpit images containing the instrument panel asynchronously; the image preprocessing unit extracts a sub-image containing the instrument panel based on manually labeled information of the instrument panel area, performs image correction, binarizes the image, and performs Gaussian smoothing for noise reduction to obtain the instrument panel image to be detected; the instrument panel pointer detection unit filters out all possible straight lines based on the second derivative of the image, filters out a cluster of straight lines based on the geometric features of the pointer, and fits the straight lines using a least squares fitting method to obtain the instrument panel pointer; The dashboard circular fitting unit filters possible scale regions in the image connected domain based on the geometric features of the scale, calculates the center point of the circumscribed rectangle of a single scale region, and performs circular fitting using the minimum distance fitting error algorithm and gradient descent algorithm. The fitted circle is then expanded to obtain the annular region containing the dashboard scale and numbers. The dashboard scale value detection unit filters possible number regions in the annular region containing the dashboard scale and numbers based on the geometric features of the number region. Adjacent number regions are merged, and after horizontal correction of the number regions, text recognition is performed. The angle between the line connecting the center point of the number region and the center of the fitted annular region and the fitted pointer line is calculated to obtain the dashboard reading. Technical effect

[0020] This invention reduces interference from other straight lines in the dashboard area by pre-screening the scale lines, and obtains the dashboard reading by calculating the angle between the line connecting the center point of the digital area and the center of the fitted annular area and the fitted pointer line, thus enabling dashboard readings under both pointer-fixed and scale-rolling conditions. This overcomes the shortcomings of existing technologies that cannot handle dashboard reading recognition when the pointer is fixed or the scale is rolling, and can provide dashboard readings in both pointer-rotating and scale-rolling conditions. Attached Figure Description

[0021] Figure 1This is a block diagram of the non-intrusive aircraft cockpit display virtual instrument recognition method of the present invention;

[0022] Figure 2 Images commonly used on aircraft cockpit displays;

[0023] In the figure: (a) is the image of the aircraft cockpit display screen, and (b) is the virtual instrument panel image obtained after affine transformation and image cropping;

[0024] Figure 3 To extract the pointer result image;

[0025] In the figure: (a) is the binarized image, (b) is all the straight lines selected by the second derivative of the image, and (c) is the extracted pointer;

[0026] Figure 4 This is a schematic diagram of the fitting results for the virtual dashboard scale;

[0027] In the figure: (a) the selected scale connected region, (b) the scale fitted circle, and (c) the ring containing the identified digits;

[0028] Figure 5 A schematic diagram of the digital region processing results;

[0029] In the figure: (a) is the intersection image of the annular region and the original image, (b) is the image of the digital region after filtering and merging, and (c) is the image of the digital region after horizontal correction and enhancement. Detailed Implementation

[0030] like Figure 1 As shown, this embodiment relates to a non-intrusive aircraft cockpit virtual instrument panel recognition method, which includes the following steps:

[0031] Step 1) Acquire images of the aircraft cockpit display screen. Specifically, use a high-definition industrial camera to acquire images of the aircraft cockpit display screen. During the capture, the entire aircraft cockpit display screen should fill the entire image, and the scales, numbers, and virtual pointers in the virtual instrument panel should be clearly visible. The acquired image should look like... Figure 2 As shown in (a), the four vertices of the display screen were manually marked, and the transition matrix from the captured image to the front view image was calculated based on known conditions such as the scale of the aircraft cockpit display screen. The location of the virtual instrument panel to be detected was manually marked, converted into a sub-image, and then an affine transformation was performed on the sub-image. After filling in the pixels using quadratic interpolation, the final image to be detected was obtained, as shown in (a). Figure 2 As shown in (b).

[0032] Step 2) Obtain the pointer in the virtual dashboard, specifically including:

[0033] S2.1, the image to be detected is binarized to obtain a binarized image, the result is as follows. Figure 3 As shown in (a).

[0034] S2.2 Select the binarized image and select appropriate Gaussian smoothing parameters to perform Gaussian smoothing on the image.

[0035] S2.3 After Gaussian smoothing the image, the second derivative of the image is calculated. For any pixel, a maximum and a minimum threshold are set. When the second derivative value of a point in the image is greater than the maximum threshold, that point is selected; when it is less than the minimum threshold, that point is discarded; when it is between the two, possible straight line regions can be defined by setting parameters such as the pixel width of the line to be detected. When a point in the image is within a possible straight line region, that point is selected, and the detection result is as follows: Figure 3 As shown in (b).

[0036] S2.4, construct a set of lines from the selected lines. Search based on the line's length and position to obtain a set of possible lines. Then, apply a weighted least squares fitting method to this set to obtain the final selected line. The line detection results are as follows: Figure 3 As shown in (c).

[0037] Step 3) Perform circular fitting on the virtual dashboard scale to obtain the circumcircle of the dashboard, and obtain the center of the circumcircle. Specifically, this includes:

[0038] S3.1, for the binarized dashboard area, a breadth-first search algorithm is used to obtain connected components. Since the pixel values ​​occupied by the tick marks are generally more than those of text or pointers, and their proportion in the connected components is also much larger, the connected components are sorted in ascending order, and suitable connected components are selected as possible tick regions based on the actual situation. The possible tick regions are sorted according to the coordinates of their top-left corners, as shown in the table below. Figure 4 As shown in (a).

[0039] S3.2 Draw circumscribed rectangles parallel to the image coordinate axes for each scale area, and obtain the coordinates of the center points of the circumscribed rectangles. These center points are approximated as points on the same circle.

[0040] S3.3, Fit a circle based on the coordinates of the center point, using the minimum distance fitting error algorithm to minimize the distance from all center points to the circle to be fitted, as shown in the following formula: The circle formed by the scale region is then iteratively solved using the gradient descent algorithm, with its center point coordinates (x0, y0) and radius r0. The circle fitting result is as follows: Figure 4 (b)

[0041] S3.4, Since the scale and its values ​​are generally located within a ring, to reduce the interference of other text within the circle on the detection results, the boundary of the fitted circle is expanded to construct a ring, so that the text to be detected is contained within this ring. The result is as follows: Figure 4 As shown in (c).

[0042] Step 4) Identify the virtual dashboard scale values, specifically including:

[0043] S4.1, find the intersection of the annular region and the original image, the result is as follows. Figure 5 As shown in (a).

[0044] S4.2 In the sector (or ring) region, since the general digit region is a larger region than the scale region, and after the above preprocessing, the possible digit recognition interference has been eliminated, the digit region is selected according to the size of the selected connected component, and these regions are sorted according to the position of the upper left vertex.

[0045] S4.3, traverse the sorted number region, calculate the distance between any two adjacent regions, and if the distance between two regions is less than a set threshold, merge the two regions into a single region to obtain the number region. The result is as follows. Figure 5 (b)

[0046] S4.4 Select a digital region within the merged digital region, calculate the angle α between the center point of its outer rectangle and the center of the circle, and rotate the digital region by angle α to the horizontal direction.

[0047] S4.5, for the digital region image ImageNum, set the mask size to MaS4.kWidth*MaS4.kHeight, and after mean normalization, obtain the mean image ImageMean. Set the influence factor Factor, i.e.: ImageEmp = round((ImageNum - ImageMean)*Factor) + ImageNum, to obtain the enhanced image ImageEmp. This enhances the contrast of the original image and makes the digital edges clearer. The result is as follows. Figure 5 As shown in (c).

[0048] Step 5) The processed digital region is identified using a general text recognition algorithm to obtain the recognition result of the digital region. Since the interval of the digital region is uniformly distributed, the recognition result of the digital region is combined with its deflection angle α to calculate the scale value pointed to by the pointer.

[0049] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A non-intrusive method for recognizing virtual instrument panels in an aircraft cockpit, characterized in that, By acquiring high-definition color images of the cockpit, the second derivative of the images is calculated after preprocessing to select the instrument panel pointers, and the connected components are selected by binarizing the images. The scale of the instrument panel is obtained based on the connected components, and the scale area of ​​the instrument panel is fitted. After filtering the circular area where the scale of the instrument panel is located, the numerical part is obtained by horizontal correction. Finally, the scale value is obtained by text recognition. The high-definition color image is filled by the entire aircraft cockpit display screen, and the scales, numbers and virtual pointers in the virtual instrument panel are clear. By marking the four vertices of the display screen, the corresponding positions of the four vertices under the camera's front view are obtained according to the known conditions, thereby calculating the transfer matrix, performing an affine transformation on the display screen image, and using a quadratic interpolation method to fill in the pixels before preprocessing the image. The aforementioned dial scale area is determined by calculating the second derivative of the image to select a suitable set of lines, which is then fitted using a least-squares fitting method. The fitting result is used as the pointer of the virtual dashboard to be detected. Connected components are selected from the binarized image as the areas where the dashboard scales are located. The center points of the bounding rectangles of the scale values ​​are fitted to obtain the dial scale areas where the scales are located in the virtual dashboard. Specifically, this includes: sorting the connected components in ascending order of size, selecting a suitable proportion from the sorted connected components as the scale areas, and then sorting the coordinates of the upper left corner of the scale areas to obtain a set of scale areas; obtaining the bounding rectangles of each area in the scale area set that are parallel to the image coordinate axes, and obtaining the coordinates of the center points of each bounding rectangle; and using a minimum distance fitting error algorithm to find the center points of all the bounding rectangles. To radius The distance to the circle to be fitted is minimized, specifically: The circle formed by the scale area is solved iteratively by using the gradient descent algorithm; the boundary of the fitted circle is expanded to construct a ring, so that the text to be detected is contained inside the ring; and the intersection of this ring area with the original image is calculated to form a fan-shaped or ring-shaped dial scale area. The digital portion involves calculating the connected components of the dial scale area, and based on the characteristic that the connected components of the scale values ​​are similar in size, discarding the scale portion, using the remaining connected components as the digital portion, merging adjacent digital portions, and rotating the digital portion to a horizontal orientation. Specifically, this includes: i) Find the intersection of the annular region and the original image to obtain the possible digital regions; ii) In the binarized image, remove multiple connected regions of similar size, select the remaining connected regions as digital regions, and sort these regions according to the position of the top left vertex; iii) Traverse the sorted number region and merge regions with adjacent center points into the same region; iv) Select a certain number range and calculate the angle between the line connecting the center point of its bounding rectangle and the circle and the fitted line. And according to the included angle Rotate the digital area to a horizontal position; v) Enhance the image of the digital region; The aforementioned text recognition refers to: inputting the digit portion into a text detector to obtain the digit detection result, and combining the digit detection result with its skew angle. Calculate the tick value currently pointed to by the pointer.

2. The non-intrusive aircraft cockpit virtual instrument panel recognition method according to claim 1, characterized in that, The preprocessing includes: correcting and segmenting the high-definition color image by manual annotation to obtain the location of the instrument to be identified; binarizing the segmented image; and performing Gaussian smoothing on the binarized image to remove noise.

3. The non-intrusive aircraft cockpit virtual instrument panel recognition method according to claim 1, characterized in that, The calculation of the second derivative of the image refers to: after setting high and low thresholds, if the second derivative value of a point in the image is greater than the high threshold, then the point is selected; if it is less than the low threshold, then the point is discarded; when it is between the two, the possible straight line region is defined by setting the pixel width of the line to be detected, and if a point in the image is in the possible straight line region, then the point is selected.

4. The non-intrusive aircraft cockpit virtual instrument panel recognition method according to claim 1, characterized in that, The pointer of the virtual dashboard to be detected is obtained by filtering all the obtained straight lines according to their lengths to get a set of straight lines with a length greater than a set. The set of straight lines is then fitted using weighted least squares, and the influence of outliers is reduced by using the Tukey method. The resulting unique straight line is the pointer of the virtual dashboard.

Images