Aircraft cockpit complex scene target identification and extraction method based on visual sensor

Abstract

The invention discloses an aircraft cockpit complex scene target identification and extraction method based on a visual sensor. The method has the beneficial effects that the intelligent requirement of identifying and extracting the target by the visual sensor under the influence of external irregular motion is met, and the jitter image needs to efficiently and accurately finish image stabilization processing for subsequent target region-of-interest positioning identification and extraction processing.

Description

Technical Field

[0001] This invention relates to the field of avionics technology, and in particular, to a method for target recognition and extraction in complex scenes in an aircraft cockpit based on visual sensors. Background Technology

[0002] In avionics systems, visual sensors are a crucial means of converting perceived information in complex scenarios into digital signals. They not only provide real-time feedback on environmental changes to onboard equipment for appropriate adjustments but also offer pilots real-time environmental visual information for situational adjustments. However, irregular attitudes of aircraft equipment can cause jitter in the video images acquired by visual sensors, leading to image translation, rotation, and scaling, which affects subsequent image processing. Therefore, there is an urgent need to develop target recognition and extraction technologies for complex scenes to address the challenges of target recognition and extraction based on visual sensors in complex aircraft mission scenarios, thereby improving the accuracy and reliability of target recognition. Summary of the Invention

[0003] The technical objective of this invention is to provide a method for target recognition and extraction in complex scenes of an aircraft cockpit based on a visual sensor.

[0004] To achieve the above-mentioned technical objectives, the present invention provides a technical solution: a method for target recognition and extraction in complex scenes of an aircraft cockpit based on a visual sensor, comprising:

[0005] Step S1: Set the first frame of video image acquired in real time by the vision sensor as the reference frame image to be transformed;

[0006] Step S2: For the reference frame image, extract the Faster Segmentation Test Feature (FAST) corner points and the Binary Robust Independent Basic Feature (BRIEF) descriptor to obtain the directional FAST and rotated BRIEF (ORB) feature point set of the reference frame.

[0007] Step S3: Read the current frame of the video stream from the vision sensor and extract the FAST corner points and BRIEF descriptors to obtain the ORB feature points of the current frame.

[0008] Step S4: Use a feature matching algorithm to find matching point pairs between the current frame and the reference frame, calculate the distance between matching points, and find the shortest distance between matching point pairs.

[0009] Step S5: Based on the principle of minimum Hamming distance, select matching point pairs with close Hamming distances;

[0010] Step S6: Use an optimization algorithm to remove incorrect matches and obtain correct matching point pairs to improve matching accuracy;

[0011] Step S7: Based on the mathematical relationship of image transformation between the current frame and the reference frame, the coefficients in the mathematical relationship are solved by substituting the matching point pairs. The transformed image is then calculated by obtaining the coefficients.

[0012] Step S8: Based on the pre-set target contour corner point coordinates, the target region of interest is cropped from the transformed image to achieve localization, recognition and extraction.

[0013] As a preferred method for target recognition and extraction in complex scenes of aircraft cockpits based on visual sensors, the reference frame in step S1 is the first video image with clear picture.

[0014] As a preferred method for target recognition and extraction in complex scenes of aircraft cockpits based on vision sensors, the ORB feature points in step S2 mainly consist of scale-invariant and rotation-invariant FAST corner points and BRIEF descriptors from the image pyramid. The FAST corner points are obtained by first selecting a pixel as the center point and recording its pixel value p. center Secondly, set the pixel value threshold β for judging FAST corner points; compare the pixel value p at the center point. center Given the pixel values ​​of all pixels on a circle with a radius of 3, if there exist N consecutive pixels with pixel values ​​greater than (p center +β) or less than (p) center If -β), then the center point is the FAST corner point; finally, traverse all pixels in the image to find all FAST corner points. The BRIEF descriptor randomly compares the pixel values ​​of 128 pairs of pixels around the FAST corner point according to the direction of the FAST corner point. If the previous pixel value is greater than the next pixel value, it is represented by 1, and if the previous pixel value is less than the next pixel value, it is represented by 0.

[0015] As a preferred solution for target recognition and extraction in complex scenes of aircraft cockpits based on visual sensors, the current frame of the visual sensor video stream in step S3 is a video frame other than the reference frame, and the reading method adopts loop reading.

[0016] As a preferred solution for target recognition and extraction in complex scenes of aircraft cockpits based on visual sensors, the matching algorithm in step S4 adopts brute-force matching. This algorithm calculates the distance between each descriptor in the descriptor set of ORB feature points in the current frame and the descriptor of ORB feature points in the reference frame. Then, it sorts all the distances and selects the descriptor with the smallest distance or the distance that meets the threshold requirement as the matching result.

[0017] As a preferred option for a method of target recognition and extraction in complex scenes of aircraft cockpits based on visual sensors, the minimum Hamming distance in step S5 is selected as either the shortest distance in twice the matching point pair or a pre-set maximum threshold. If twice the shortest distance is less than the pre-set maximum threshold, then the pre-set maximum threshold is used as the minimum Hamming distance.

[0018] As a preferred method for target recognition and extraction in complex scenes of aircraft cockpits based on visual sensors, the optimization algorithm in step S6 adopts the Random Sample Consensus Algorithm (RANSAC). This algorithm randomly selects four feature point pairs from the matching results and calculates the homography matrix. The reprojection coordinates of the ORB feature points of the current frame are calculated in the reference frame based on the homography matrix. The distance between the reprojection coordinates and the coordinates of the matched ORB feature points is compared. If the distance is less than a certain threshold, a correct match is made; otherwise, an incorrect match is made. After multiple iterations, all correctly matched point pairs are recorded.

[0019] As a preferred method for target recognition and extraction in complex scenes of aircraft cockpits based on visual sensors, the image transformation relationship in step S7 adopts affine transformation, and the coefficients obtained are rotation and translation matrices.

[0020] As a preferred solution for target recognition and extraction in complex scenes of aircraft cockpits based on visual sensors, step S8 uses a mask matrix to extract the region of interest in the cockpit. The region of interest is set in advance in the matrix with 0 and 1, with the value in the region of interest being 1 and the others being 0.

[0021] Compared with the prior art, the beneficial effects of the present invention are at least as follows: it meets the intelligent requirements of visual sensors to identify and extract targets under the influence of irregular external motion, and the shaking images need to be stabilized efficiently and accurately for subsequent location, identification and extraction of the target's region of interest. Attached Figure Description

[0022] Figure 1 This is a roadmap for complex scene target recognition and extraction technology based on visual sensors.

[0023] Figure 2 It is a reference frame in the video stream.

[0024] Figure 3 It is the current frame in the video stream.

[0025] Figure 4 It is the recognition and extraction of complex scene targets after affine transformation. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. It should be noted that these descriptions of embodiments are intended to aid in understanding the invention and do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0027] Please see Figure 1 The figure shows a method for target recognition and extraction in complex scenes of an aircraft cockpit based on a visual sensor.

[0028] by Figure 2-4 For example, according to the following specific implementation steps, the self-illuminating components (i.e., each LCD screen) in the aircraft cockpit in the illustration are set as regions of interest, and the target is identified and extracted by matching the current frame with the reference frame.

[0029] Step A: Set the first frame of video image acquired in real time by the vision sensor as the reference frame image to be transformed (using... Figure 2 (For example).

[0030] Step B: For the reference frame image, extract n FAST corner points and BRIEF descriptors to obtain the ORB feature point set P = {p1, p2, ..., p...} of the reference frame. n ORB feature points are mainly composed of scale-invariant and rotation-invariant FAST corner points and BRIEF descriptors from the image pyramid. The FAST corner points are determined by first selecting a pixel as the center point and recording its pixel value p. center Secondly, set the pixel value threshold β for judging FAST corner points; compare the pixel value p at the center point. center Among all pixel values ​​on a circle with radius 3, if there exist consecutive pixels, the pixel values ​​of p pixels are greater than (p... cetnter +β) or less than (p) center If -β), then the center point is the FAST corner point; finally, traverse all pixels in the image to find all FAST corner points. The BRIEF descriptor randomly compares the pixel values ​​of 128 pairs of pixels around the FAST corner point according to the direction of the FAST corner point. If the previous pixel value is greater than the next pixel value, it is represented by 1, and if the previous pixel value is less than the next pixel value, it is represented by 0.

[0031] Step C: Read the current frame of the video stream from the visual sensor (in words). Figure 3 (For example), and extract m FAST corner points and BRIEF descriptors to obtain the ORB feature points P′={p1′,p2′,...,p m The current frame of the visual sensor video stream is a video frame other than the reference frame, and the reading method is loop reading.

[0032] Step D: Use a feature matching algorithm to find matching point pairs {(p1, p1″), (p2, p2″), ..., (p...} between the current frame and the reference frame. k , p″ k )|p i Given the set {\beginning} ∈ P', calculate the distance D = {d1, d2, ..., d\beginning} between the matching points. k} and calculate the shortest distance d between matching point pairs. min The matching algorithm uses brute force matching. This algorithm calculates the distance between each descriptor in the descriptor set of ORB feature points in the current frame and the descriptor of ORB feature points in the reference frame. Then, it sorts all the distances and selects the descriptor with the smallest distance or the distance that meets the threshold requirement as the matching result.

[0033] Step E: Based on the minimum Hamming distance principle, select matching point pairs with close Hamming distances. The minimum Hamming distance is selected as the shortest distance among twice the number of matching point pairs or a pre-set maximum threshold, i.e., D′={d|d i ≤max(2d min If the shortest distance is twice 2d, then... min If the distance is less than the preset maximum threshold Max, then the threshold Max is used as the minimum Hamming distance.

[0034] Step F: An optimization algorithm is used to eliminate incorrect matches and obtain correctly matched point pairs, improving matching accuracy. The optimization algorithm uses the Random Sample Consensus Algorithm (RANSAC), which randomly selects four feature point pairs from the matching results and calculates the homography matrix. The reprojection coordinates of the ORB feature points of the current frame are calculated in the reference frame based on the homography matrix. The distance between the reprojection coordinates and the coordinates of the matched ORB feature points is compared. If the distance is less than a certain threshold, it is a correct match; otherwise, it is an incorrect match. After multiple iterations, all correctly matched point pairs are recorded.

[0035] Step G: Based on the mathematical relationship between the image transformation of the current frame and the reference frame, solve for the coefficients in this mathematical relationship by substituting the matching point pairs. By obtaining the coefficients, calculate the transformed image (e.g., ...). Figure 4 (As shown). The image transformation relationship uses affine transformation, and the coefficients are the rotation and translation matrices. The mathematical principle of affine transformation is as follows:

[0036]

[0037] In equation (1), [x′ y′ 1′] T Let [xy 1] be the homogeneous coordinates of the feature points in the reference frame. TLet M be the homogeneous coordinates of the feature points in the current frame, B be the rotation matrix, and y be the translation vector. In equation (2), x0 and y0 are the pixel coordinates of the rotation center, β is the rotation angle, and s is the scaling factor. By substituting the matching point pairs, the above M and B are solved. Finally, the affine transformed image is calculated based on the obtained M and B. According to the pre-set target contour corner coordinates, the target region of interest is extracted from the affine transformed image to achieve localization, recognition, and extraction. The region of interest is extracted using a mask matrix. The region of interest is set in advance in the matrix using 0 and 1, with the value within the region of interest being 1 and the others being 0. Figure 4 For example, the corner coordinates of each LCD screen are preset in advance, and the area of ​​the LCD screen is set as the region of interest.

[0038] The usage method is as follows: The target recognition and extraction method for complex scenes in the aircraft cockpit based on vision sensors is encapsulated in the form of software modules to complete the target perception in complex scenes under shaking vision sensors, meet the target recognition and extraction problem based on vision sensors in complex aircraft mission scenarios, and improve the accuracy and reliability of target recognition.

[0039] The above description merely illustrates embodiments of the present invention and is quite specific and detailed; however, it should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A method for target recognition and extraction of complex scene in cockpit of aircraft based on visual sensor, characterized in that, The method comprises the following steps: Step S1, setting a first frame video image collected by a visual sensor in real time as a reference frame image to be transformed; Step S2, extracting a feature point set of an oriented FAST and a rotated BRIEF (ORB) of the reference frame by extracting a feature point set of an accelerated segmentation test (FAST) corner and a binary robust independent elementary feature (BRIEF) descriptor of the reference frame image; Step S3, reading a current frame of a video stream of the visual sensor and extracting a FAST corner and a BRIEF descriptor to obtain an ORB feature point of the current frame; Step S4, finding a point pair matched between the current frame and the reference frame by using a feature matching algorithm, calculating a distance between the matched points, and finding a shortest distance between the matched point pairs; Step S5, screening out the matched point pairs with a close Hamming distance according to a minimum Hamming distance principle; Step S6, removing false matches by using an optimization algorithm to obtain correct matched point pairs and improve matching accuracy; Step S7, solving a coefficient in a mathematical relationship between image transformation of the current frame and the reference frame by substituting the matched point pairs, and calculating an image after image transformation by the coefficient; Step S8, cutting a target region of interest from the transformed image according to a pre-set target contour corner point coordinate to realize positioning recognition and extraction.

2. The method for visual sensor based target identification and extraction of complex scene in cockpit of aircraft according to claim 1, wherein, The reference frame in step S1 is a first frame video image with a clear picture.

3. The method for visual sensor based target identification and extraction of complex scene in aircraft cockpit according to claim 1, characterized in that, The ORB feature points of step S2 are composed of the FAST corner points and BRIEF descriptors with scale invariance and rotation invariance in the image pyramid; wherein the FAST corner points are obtained by selecting a pixel point as a center point and recording its pixel value p center ; secondly, setting a threshold β for judging the pixel value of the FAST corner point; comparing the pixel value p center of the center point with the pixel values of all pixels on the circumference of a circle with a radius of 3; if there are N continuous pixels with pixel values greater than (p center + β) or less than (p center - β), the center point is a FAST corner point; finally, traversing all pixel points in the image to find all FAST corner points. The BRIEF descriptor is to randomly compare the pixel values of 128 pairs of pixel points around the FAST corner point according to the direction rule of the FAST corner point, and the former pixel value is greater than the latter pixel value is represented by 1, and the former pixel value is less than the latter pixel value is represented by 0.

4. The method for visual sensor based target identification and extraction of complex scene in aircraft cockpit according to claim 1, characterized in that, The current frame of the video stream of the visual sensor in step S3 is a video frame other than the reference frame, and the reading mode is cyclic reading.

5. The vision sensor based aircraft cockpit complex scene target identification and extraction method according to claim 1, characterized in that, The matching algorithm in step S4 is a brute force matching algorithm, which calculates a distance between each descriptor in a descriptor set of the ORB feature point in the current frame and a descriptor of the ORB feature point in the reference frame, sorts all the distances, and selects a descriptor with a minimum distance or a distance meeting a threshold requirement as a matching result.

6. The vision sensor based aircraft cockpit complex scene target identification and extraction method according to claim 1, characterized in that, The minimum Hamming distance in step S5 is selected as 2 times of the shortest distance in the matched point pairs or a pre-set maximum threshold value, and if the 2 times of the shortest distance is smaller than the pre-set maximum threshold value, the pre-set maximum threshold value is used as the minimum Hamming distance.

7. The vision sensor based aircraft cockpit complex scene target identification and extraction method according to claim 1, characterized in that, The optimization algorithm in step S6 is a random sample consensus (RANSAC) algorithm, which randomly selects 4 feature point pairs from the matching result, calculates a homography matrix, calculates a re-projection coordinate of the ORB feature point in the current frame in the reference frame according to the homography matrix, compares a distance between the re-projection coordinate and the matched ORB feature point coordinate, and if the distance is smaller than a certain threshold value, the matching is correct, otherwise, the matching is false. After multiple cycles, all the correct matched point pairs are recorded.

8. The vision sensor based aircraft cockpit complex scene target identification and extraction method according to claim 1, characterized in that, The image transformation relationship in step S7 is an affine transformation, and the solved coefficient is a rotation matrix and a translation matrix.

9. The vision sensor based aircraft cockpit complex scene target identification and extraction method according to claim 1, characterized in that, The region of interest in step S8 is extracted by using a mask matrix to extract a cabin region of interest, and the region of interest is set in the matrix in advance by using 0 and 1, the value in the region of interest is 1, and the other values are 0.

Images