A visually assisted landing method and system for shipborne helicopters based on image recognition

Abstract

The present invention discloses a visually assisted landing method and system for shipborne helicopters based on image recognition. The method comprises the following steps: collecting real-time image data of the shipborne helicopter's surroundings via an onboard camera and transmitting the data to an onboard computer; extracting features from the pre-processed real-time image data using an image feature extraction method, and establishing a feature point matching relationship between adjacent image frames to obtain the relative position and motion relationship between the shipborne helicopter and the ship's flight deck. By utilizing image recognition technology, the present invention can detect and analyze the motion of the shipborne helicopter and the ship's flight deck, as well as obstacles in the area surrounding the ship, and obtain positional relationships such as the height difference and heading angle between the shipborne helicopter and the ship's flight deck, as well as the ship's motion patterns, thereby improving the landing accuracy and safety factor of the shipborne helicopter under high sea conditions and reducing the pilot's operating load.

Description

Technical Field

[0001] The present invention relates to the technical field of shipborne helicopters, and in particular to a method and system for visually assisted landing of a shipborne helicopter based on image recognition. Background Art

[0002] Helicopters have the characteristics of vertical take-off and landing and high-speed forward flight, which makes them widely used in many fields. However, as helicopters are used more and more widely, the landing safety issue of ship-borne helicopters has become increasingly prominent.

[0003] Shipborne helicopter landing accidents are frequent due to the small size of ship flight decks, the interaction between ship and helicopter during landing, and the adverse effects of sea conditions and the flight deck wake. Currently, my country's active shipborne helicopters lack the ability to land in high sea conditions. To improve the deployment capabilities of shipborne helicopters in high sea conditions, provide safe and reliable landing strategies for shipborne helicopters, and enhance their landing capabilities, research on visually assisted landing methods is particularly necessary. Summary of the Invention

[0004] In response to the problems in the related art, the present invention proposes a shipborne helicopter visually assisted landing method and system based on image recognition to overcome the above-mentioned technical problems existing in the existing related art.

[0005] This patent designs a visually assisted landing method and system for shipborne helicopters based on image recognition, which enables pilots to perceive the surrounding environment information in real time during the landing process of the shipborne helicopter, understand the relative position and speed relationship between the shipborne helicopter and the flight deck, and provide the shipborne helicopter with the best landing point, landing time and movement trajectory. It can reduce the danger of traditional visual landing, improve landing efficiency and safety factor, and also help to reduce the pilot's operating load.

[0006] To this end, the specific technical solutions adopted in the present invention are as follows:

[0007] According to one aspect of the present invention, a method for visually assisted landing of a shipborne helicopter based on image recognition is provided, the method comprising the following steps:

[0008] S1, collect real-time image data of the shipborne helicopter's surrounding environment through the onboard camera and transmit it to the onboard computer;

[0009] S2, preprocessing the collected real-time image data through the onboard computer;

[0010] S3. Extract features from the pre-processed real-time image data using an image feature extraction method, and establish a feature point matching relationship between adjacent image frames to obtain the relative position and motion relationship between the shipborne helicopter and the ship's flight deck;

[0011] S4, converting the result of feature recognition into coordinate values ​​in the actual coordinate system through airborne camera calibration;

[0012] S5. Using an artificial neural network algorithm, error correction and rationality correction are performed on the converted coordinate values, and the position, direction, and speed of each target object are determined;

[0013] S6. Use clustering algorithms to cluster the positions and directions of target objects to determine the best landing point, landing time, and movement trajectory, and issue instructions to the pilot to assist the shipborne helicopter in landing.

[0014] Furthermore, the preprocessing of the collected real-time image data by the onboard computer includes the following steps:

[0015] S21, removing noise from the collected real-time image data through a Gaussian filter;

[0016] S22, correcting brightness and color deviation of the real-time image data after noise removal;

[0017] S23. Process the real-time image data by using a histogram equalization method to obtain real-time image data with enhanced contrast and clarity.

[0018] Furthermore, the noise removal of the collected real-time image data by using a Gaussian filter includes the following steps:

[0019] S211, select the preset Gaussian kernel size and standard deviation;

[0020] S212 , calling an API function for implementing Gaussian filtering, performing a filtering operation on the real-time image data, and obtaining denoised real-time image data.

[0021] Furthermore, the histogram equalization method includes:

[0022] Convert live image data into grayscale images using OpenCV;

[0023] Get the pixel value and grayscale level of the grayscale image;

[0024] Calculate the probability value of gray level occurrence;

[0025] A discrete transformation function is used to process the pixels in the real-time image data and obtain the grayscale value of each pixel after equalization.

[0026] Furthermore, the process of extracting features from the pre-processed real-time image data using an image feature extraction method and establishing a feature point matching relationship between adjacent image frames includes the following steps:

[0027] S31, extracting feature points from real-time image data using a scale-invariant feature conversion algorithm;

[0028] S32, extracting stable feature points from the real-time image data and calculating a descriptor for each feature point to characterize image information around the feature point;

[0029] S33, matching the feature points in two adjacent images, and achieving a feature point matching relationship by calculating the distance and similarity between the two descriptors;

[0030] Furthermore, the extraction of feature points from real-time image data using a scale-invariant feature conversion algorithm includes the following steps:

[0031] S311, performing multi-scale transformation on the input image by using a Gaussian difference pyramid method to obtain a series of scale space images;

[0032] S312, comparing adjacent pixels in each scale space image and detecting feature points with extreme values;

[0033] S313, by calculating the gradient and direction of the pixels around each feature point, a histogram of the gradient direction around the feature point is calculated, and the histogram peak is found as the main direction of the feature point;

[0034] S314. Construct a local area with the feature point as the center, calculate the gradient direction and amplitude in the area, and form a vector as a descriptor of the feature point to perform feature matching.

[0035] Furthermore, the method of performing error correction and rationality correction on the coordinate values ​​and determining the position, direction and speed of each target object by using an artificial neural network algorithm includes the following steps:

[0036] S51. Using BP learning algorithm, the accurate data set is divided into a training set and a test set according to the proportion;

[0037] S52, calculating the output value of the forward propagation for the samples in the training set in sequence;

[0038] S53, updating the weights and biases according to the error back propagation rule, and obtaining a smaller value of the training error through iteration;

[0039] S54. Test the trained accurate data set on the test set and calculate the error between the predicted result and the true label.

[0040] Furthermore, the BP learning algorithm includes:

[0041] Initialize the adjustable parameters of the artificial neural network;

[0042] Input samples and obtain the output of each layer through forward calculation;

[0043] Calculate the instantaneous value of the error of each neuron in the output layer and the total square error at the output end;

[0044] Correct the connection weights between the output layer and the hidden layer;

[0045] Correct the connection weights between the hidden layer and the input layer;

[0046] Calculate the objective function after one cycle of network training;

[0047] Input new samples, make the objective function meet the preset requirements, and complete the training;

[0048] The converted coordinate values ​​are input into the trained neural network to obtain the coordinate values ​​output by the neural network, and the position, direction and speed of the final target object are obtained.

[0049] Furthermore, the method of clustering the position, direction, and speed of the target object using a clustering algorithm to determine the optimal landing point, landing time, and motion trajectory, and issuing instructions to the pilot to assist the shipborne helicopter in landing, includes the following steps:

[0050] S61, preset clustering parameter combination list;

[0051] S62, performing cluster initialization using the feature points of the position, direction and speed of the current target object to be extracted as a cluster data set;

[0052] S63, clustering the data set using a clustering algorithm to obtain a set of all clusters;

[0053] S64, performing deletion processing on the set obtained based on the statistical data, eliminating clusters in the set that do not belong to the feature area;

[0054] S65. Update the feature area using the deleted clusters in the set to determine the optimal landing point, landing time, and movement trajectory.

[0055] According to another aspect of the present invention, there is also provided a shipborne helicopter visually assisted landing system based on image recognition, the system comprising: an image acquisition module, an image preprocessing module, a feature extraction module, a feature conversion module, a coordinate correction module and a control instruction module;

[0056] The image acquisition module is used to collect real-time image data of the surrounding environment of the shipborne helicopter through an onboard camera and transmit it to the onboard computer;

[0057] The image preprocessing module is used to preprocess the collected real-time image data through the onboard computer;

[0058] The feature extraction module is used to extract features from the pre-processed real-time image data using an image feature extraction method and establish a feature point matching relationship between adjacent image frames;

[0059] The feature conversion module is used to convert the result of feature recognition into coordinate values ​​in the actual coordinate system through the airborne camera calibration;

[0060] The coordinate correction module is used to perform error correction and rationality correction on the converted coordinate values ​​through an artificial neural network algorithm, and determine the position, direction and speed of each target object;

[0061] The control instruction module is used to cluster the position, direction and speed of the target object using a clustering algorithm to determine the optimal landing point, landing time and movement trajectory, and issue instructions to the pilot to assist the shipborne helicopter in landing.

[0062] The beneficial effects of the present invention are:

[0063] 1. By utilizing image recognition technology, the present invention can detect and analyze the movement of a shipborne helicopter landing on a ship's flight deck, as well as obstacles in the area around the ship, and obtain the positional relationship between the shipborne helicopter and the flight deck, such as the height difference and heading angle, as well as the movement pattern of the ship, thereby improving landing accuracy and safety factors and reducing the pilot's operating load. In addition, by processing real-time image data through the histogram equalization method, low-contrast images can be converted into high-contrast images, which can effectively improve image quality and enhance the accuracy and reliability of image processing.

[0064] 2. By setting the Gaussian filter, it is possible to denoise Gaussian noise and random noise while effectively improving the image quality, reducing the error caused by noise, and thus improving the accuracy of subsequent image processing.

[0065] 3. By clustering the position, direction and speed of the target object through the clustering algorithm, the optimal landing point, landing time and movement trajectory can be determined, thereby providing data support for the landing of ship-borne helicopters, thereby improving the landing accuracy and safety factor of ship-borne helicopters. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0067] Figure 1The present invention is a flowchart of a method for visually assisted landing of a shipborne helicopter based on image recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0068] According to an embodiment of the present invention, a method and system for visually assisted landing of a shipborne helicopter based on image recognition are provided.

[0069] The present invention will now be further described with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, according to an embodiment of the present invention, a method for visually assisted landing of a shipborne helicopter based on image recognition includes the following steps:

[0070] S1, collect real-time image data of the shipborne helicopter's surrounding environment through the onboard camera and transmit it to the onboard computer;

[0071] S2, preprocessing the collected real-time image data through the onboard computer;

[0072] S3. Extract features from the pre-processed real-time image data using an image feature extraction method, and establish a feature point matching relationship between adjacent image frames to obtain the relative position and motion relationship between the shipborne helicopter and the ship's flight deck;

[0073] S4, converting the result of feature recognition into coordinate values ​​in the actual coordinate system through airborne camera calibration;

[0074] S5. Using an artificial neural network algorithm, error correction and rationality correction are performed on the converted coordinate values, and the position, direction, and speed of each target object are determined;

[0075] S6. Use clustering algorithms to cluster the position, direction, and speed of the target object to determine the optimal landing point, landing time, and movement trajectory, and issue instructions to the pilot to assist the shipborne helicopter in landing.

[0076] In one embodiment, the pre-processing of the collected real-time image data by the onboard computer comprises the following steps:

[0077] S21, removing noise from the collected real-time image data through a Gaussian filter;

[0078] S22, correcting brightness and color deviation of the real-time image data after noise removal;

[0079] S23. Process the real-time image data by using a histogram equalization method to obtain real-time image data with enhanced contrast and clarity.

[0080] In one embodiment, removing noise from the collected real-time image data using a Gaussian filter comprises the following steps:

[0081] S211, select the preset Gaussian kernel size and standard deviation;

[0082] S212 , calling an API function for implementing Gaussian filtering, performing a filtering operation on the real-time image data, and obtaining denoised real-time image data.

[0083] In one embodiment, the histogram equalization method includes:

[0084] Convert live image data into grayscale images using OpenCV;

[0085] Get the pixel value and grayscale level of the grayscale image;

[0086] Calculate the probability value of gray level occurrence;

[0087] A discrete transformation function is used to process the pixels in the real-time image data and obtain the grayscale value of each pixel after equalization.

[0088] In one embodiment, the process of extracting features from the pre-processed real-time image data using an image feature extraction method and establishing a feature point matching relationship between adjacent image frames includes the following steps:

[0089] S31, extracting feature points from real-time image data using a scale-invariant feature conversion algorithm;

[0090] S32, extracting stable feature points from the real-time image data and calculating a descriptor for each feature point to characterize image information around the feature point;

[0091] S33, matching the feature points in two adjacent images, and achieving a feature point matching relationship by calculating the distance and similarity between the two descriptors;

[0092] In one embodiment, extracting feature points from real-time image data using a scale-invariant feature conversion algorithm includes the following steps:

[0093] S311, performing multi-scale transformation on the input image by using a Gaussian difference pyramid method to obtain a series of scale space images;

[0094] S312, comparing adjacent pixels in each scale space image and detecting feature points with extreme values;

[0095] S313, by calculating the gradient and direction of the pixels around each feature point, a histogram of the gradient direction around the feature point is calculated, and the histogram peak is found as the main direction of the feature point;

[0096] S314. Construct a local area with the feature point as the center, calculate the gradient direction and amplitude in the area, and form a vector as a descriptor of the feature point to perform feature matching.

[0097] In one embodiment, performing error correction and rationality correction on the coordinate values ​​and determining the position, direction, and speed of each target object using an artificial neural network algorithm includes the following steps:

[0098] S51. Using BP learning algorithm, the accurate data set is divided into a training set and a test set according to the proportion;

[0099] S52, calculating the output value of the forward propagation for the samples in the training set in sequence;

[0100] S53, updating the weights and biases according to the error back propagation rule, and obtaining a smaller value of the training error through iteration;

[0101] S54. Test the trained accurate data set on the test set and calculate the error between the predicted result and the true label.

[0102] In one embodiment, the BP learning algorithm includes:

[0103] Initialize the adjustable parameters of the artificial neural network;

[0104] Input samples and obtain the output of each layer through forward calculation;

[0105] Calculate the instantaneous value of the error of each neuron in the output layer and the total square error at the output end;

[0106] Correct the connection weights between the output layer and the hidden layer;

[0107] Correct the connection weights between the hidden layer and the input layer;

[0108] Calculate the objective function after one cycle of network training;

[0109] Input new samples, make the objective function meet the preset requirements, and complete the training;

[0110] The converted coordinate values ​​are input into the trained neural network to obtain the coordinate values ​​output by the neural network, and the position, direction and speed of the final target object are obtained.

[0111] In one embodiment, clustering the position, direction, and speed of a target object using a clustering algorithm to determine an optimal landing point, landing time, and motion trajectory, and issuing instructions to a pilot to assist a shipborne helicopter in landing, includes the following steps:

[0112] S61, preset clustering parameter combination list;

[0113] S62, performing cluster initialization using the feature points of the position, direction and speed of the current target object to be extracted as a cluster data set;

[0114] S63, clustering the data set using a clustering algorithm to obtain a set of all clusters;

[0115] S64, performing deletion processing on the set obtained based on the statistical data, eliminating clusters in the set that do not belong to the feature area;

[0116] S65: Update the feature area using the deleted clusters in the set to determine the best landing point and motion trajectory.

[0117] According to another embodiment of the present invention, there is also provided a shipborne helicopter visual assisted landing system based on image recognition, the system comprising: an image acquisition module, an image preprocessing module, a feature extraction module, a feature conversion module, a coordinate correction module and a control instruction module;

[0118] The image acquisition module is used to collect real-time image data of the surrounding environment of the shipborne helicopter through an onboard camera and transmit it to the onboard computer;

[0119] The image preprocessing module is used to preprocess the collected real-time image data through the onboard computer;

[0120] The feature extraction module is used to extract features from the pre-processed real-time image data using an image feature extraction method and establish a feature point matching relationship between adjacent image frames;

[0121] The feature conversion module is used to convert the result of feature recognition into coordinate values ​​in the actual coordinate system through the airborne camera calibration;

[0122] The coordinate correction module is used to perform error correction and rationality correction on the converted coordinate values ​​through an artificial neural network algorithm, and determine the position, direction and speed of each target object;

[0123] The control instruction module is used to cluster the position, direction and speed of the target object using a clustering algorithm to determine the optimal landing point, landing time and movement trajectory, and issue instructions to the pilot to assist the shipborne helicopter in landing.

[0124] In summary, the technical solution of the present invention, by setting a Gaussian filter, can effectively denoise Gaussian and random noise while improving image quality, reducing errors caused by noise, and thus increasing the accuracy of subsequent image processing. By clustering the position, direction, and velocity of target objects using a clustering algorithm, the optimal landing point, landing time, and movement trajectory can be determined, providing data support for shipborne helicopter landings and improving the landing accuracy and safety factor of shipborne helicopters.

[0125] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A visually assisted landing method for shipborne helicopters based on image recognition, characterized in that: The method comprises the following steps: S1, collect real-time image data of the shipborne helicopter's surrounding environment through the onboard camera and transmit it to the onboard computer; S2, preprocessing the collected real-time image data through the onboard computer; S3. Extract features from the pre-processed real-time image data using an image feature extraction method, and establish a feature point matching relationship between adjacent image frames to obtain the relative position and motion relationship between the shipborne helicopter and the ship's flight deck; S4, converting the result of feature recognition into coordinate values ​​in the actual coordinate system through camera calibration; S5. Using an artificial neural network algorithm, error correction and rationality correction are performed on the converted coordinate values, and the position, direction, and speed of each target object are determined; S6. Clustering the position, direction, and speed of the target object using a clustering algorithm to determine the optimal landing point, landing time, and motion trajectory, and issuing instructions to the pilot to assist the shipborne helicopter in landing; comprising the following steps: S61, preset clustering parameter combination list; S62, performing cluster initialization using the feature points of the position, direction and speed of the current target object to be extracted as a cluster data set; S63, clustering the data set using a clustering algorithm to obtain a set of all clusters; S64, performing deletion processing on the set obtained based on the statistical data, eliminating clusters in the set that do not belong to the feature area; S65. Update the feature area using the deleted clusters in the set to determine the optimal landing point, landing time, and movement trajectory.

2. The method for visually assisted landing of a shipborne helicopter based on image recognition according to claim 1, characterized in that: The method of pre-processing the collected real-time image data by the onboard computer comprises the following steps: S21, removing noise from the collected real-time image data through a Gaussian filter; S22, correcting brightness and color deviation of the real-time image data after noise removal; S23. Process the real-time image data by using a histogram equalization method to obtain real-time image data with enhanced contrast and clarity.

3. The method for visually assisted landing of a shipborne helicopter based on image recognition according to claim 2, characterized in that: The noise removal of the collected real-time image data by using a Gaussian filter comprises the following steps: S211, select the preset Gaussian kernel size and standard deviation; S212 , calling an API function for implementing Gaussian filtering, performing a filtering operation on the real-time image data, and obtaining denoised real-time image data.

4. The method for visually assisted landing of a shipborne helicopter based on image recognition according to claim 3, characterized in that: The histogram equalization method includes: Convert live image data into grayscale images using OpenCV; Get the pixel value and grayscale level of the grayscale image; Calculate the probability value of gray level occurrence; A discrete transformation function is used to process the pixels in the real-time image data and obtain the grayscale value of each pixel after equalization.

5. The method for visually assisted landing of a shipborne helicopter based on image recognition according to claim 1, characterized in that: The method of extracting features from the pre-processed real-time image data using an image feature extraction method and establishing a feature point matching relationship between adjacent image frames includes the following steps: S31, extracting feature points from real-time image data using a scale-invariant feature conversion algorithm; S32, extracting stable feature points from the real-time image data and calculating a descriptor for each feature point to characterize image information around the feature point; S33, matching the feature points in two adjacent images, and realizing the feature point matching relationship by calculating the distance and similarity between the two descriptors.

6. The method for visually assisted landing of a shipborne helicopter based on image recognition according to claim 5, characterized in that: The extraction of feature points from real-time image data using a scale-invariant feature conversion algorithm comprises the following steps: S311, performing multi-scale transformation on the input image by using a Gaussian difference pyramid method to obtain a series of scale space images; S312, comparing adjacent pixels in each scale space image and detecting feature points with extreme values; S313, by calculating the gradient and direction of the pixels around each feature point, a histogram of the gradient direction around the feature point is calculated, and the histogram peak is found as the main direction of the feature point; S314. Construct a local area with the feature point as the center, calculate the gradient direction and amplitude in the area, and form a vector as a descriptor of the feature point to perform feature matching.

7. The method for visually assisted landing of a shipborne helicopter based on image recognition according to claim 1, characterized in that: The method of performing error correction and rationality correction on the coordinate values ​​and determining the position, direction and speed of each target object by using an artificial neural network algorithm includes the following steps: S51. Using BP learning algorithm, the accurate data set is divided into a training set and a test set according to the proportion; S52, calculating the output value of the forward propagation for the samples in the training set in sequence; S53, updating the weights and biases according to the error back propagation rule, and obtaining a smaller value of the training error through iteration; S54. Test the trained accurate data set on the test set and calculate the error between the predicted result and the true label.

8. The method for visually assisted landing of a shipborne helicopter based on image recognition according to claim 7, characterized in that: The BP learning algorithm includes: Initialize the adjustable parameters of the artificial neural network; Input samples and obtain the output of each layer through forward calculation; Calculate the instantaneous value of the error of each neuron in the output layer and the total square error at the output end; Correct the connection weights between the output layer and the hidden layer; Correct the connection weights between the hidden layer and the input layer; Calculate the objective function after one cycle of network training; Input new samples, make the objective function meet the preset requirements, and complete the training; The converted coordinate values ​​are input into the trained neural network to obtain the coordinate values ​​output by the neural network, and the position, direction and speed of the final target object are obtained.

9. A shipborne helicopter visually assisted landing system based on image recognition, used to implement the shipborne helicopter visually assisted landing method based on image recognition according to any one of claims 1 to 8, characterized in that: The system includes: image acquisition module, image preprocessing module, feature extraction module, feature conversion module, coordinate correction module and control instruction module; The image acquisition module is used to collect real-time image data of the surrounding environment of the shipborne helicopter through an onboard camera and transmit it to the onboard computer; The image preprocessing module is used to preprocess the collected real-time image data through the onboard computer; The feature extraction module is used to extract features from the pre-processed real-time image data using an image feature extraction method and establish a feature point matching relationship between adjacent image frames; The feature conversion module is used to convert the result of feature recognition into coordinate values ​​in the actual coordinate system through the airborne camera calibration; The coordinate correction module is used to perform error correction and rationality correction on the converted coordinate values ​​through an artificial neural network algorithm, and determine the position, direction and speed of each target object; The control instruction module is used to cluster the position, direction and speed of the target object using a clustering algorithm to determine the optimal landing point, landing time and movement trajectory, and issue instructions to the pilot to assist the shipborne helicopter in landing.

Images