Zoom binocular system image ship target identification method based on homography matrix mapping

Abstract

The invention relates to a zoom binocular system image ship target identification method based on homography matrix mapping, and belongs to the technical field of marine ship target image detection and identification. According to the method provided by the invention, ship targets identified through a deep learning algorithm are in two images of a binocular camera system, and four corner points and a central point of each target serve as corresponding points and are used for constructing a calculation basis of a homography matrix. Bidirectional accurate mapping between the visible light and the infrared image is realized through the constructed homography matrix, and target recognition of deep learning is supplemented. And a homography matrix group based on focal length change is constructed by adjusting the focal length of the binocular imaging system. When missing detection target identification of supplementary deep learning is carried out, the most suitable homography matrix is selected from the matrix library to be applied according to the current imaging condition and the focal length, and detection identification of missing detection or even zero detection ships obtained by a certain modal image only by using the deep learning method is realized.

Description

Technical Field

[0001] The invention relates to a zoom binocular system image ship target recognition method based on homography matrix mapping, and belongs to the technical field of marine ship target image detection and recognition. Background Art

[0002] In the field of surface ship target monitoring, multi-source data fusion analysis (such as synthetic aperture radar (SAR) images, optical remote sensing images, infrared images, and visible light video images) is the key to improving monitoring accuracy and reliability. Among them, the infrared and visible light cameras of binocular cameras play an important role in achieving this goal due to their adjustable focal length and variable field of view. However, due to the differences in the field of view and characteristics of these image sources, accurately identifying and mapping common scenes and targets in them has become a challenge. When applying the deep learning YOLO algorithm for ship target recognition in a binocular system, the following main problems are encountered:

[0003] (1) Unstable recognition effect: Although deep learning technology can identify ship targets after the image is taken, the recognition effect is not always ideal due to the differences in recognition under different spectra. For example, the temperature features in infrared images and the appearance features in visible light images may cause the recognition algorithm to perform differently, and the deep learning algorithm may fail to successfully identify the ship in one of the images. The marine environment is complex and changeable. For example, changes in weather, lighting, and sea conditions will affect image quality and thus affect recognition results.

[0004] (2) Functional differences of binocular camera devices. Existing binocular camera devices can automatically identify and select marine targets in infrared images, but lack this function in visible light images, which increases the difficulty of multimodal image matching and fusion.

[0005] To solve these problems, it is crucial to accurately identify and map the common scenes in the two images. Traditional infrared image and visible light image mapping methods usually include camera calibration and feature point matching. The camera calibration method, especially the Zhang Zhengyou plane calibration method, estimates the internal and external parameters of the camera by obtaining the corner point positions of the calibration plate, and uses the homography matrix to describe the mapping relationship between the three-dimensional world coordinate system and the two-dimensional image coordinate system. This method can provide high mapping accuracy in static scenes and fixed cameras, but its strict requirements on environmental conditions and its inadaptability to dynamic scenes limit its application in the dynamic environment of the sea surface.

[0006] The feature point matching method uses algorithms such as SIFT to detect feature points in the image and calculate the descriptors of the feature points. The feature points are matched in the two images through the descriptor matching algorithm to obtain a set of corresponding feature point pairs. Then, the homography matrix is ​​calculated using the random sampling consensus algorithm, and the accuracy of the matrix is ​​improved by iteratively selecting internal points and removing mismatched points. However, this method is highly sensitive to factors such as noise, illumination changes, resolution, and motion blur in the image, which affects the accuracy and robustness of the mapping. Summary of the invention

[0007] In view of the limitations of the above-mentioned traditional methods, deep learning is integrated with geometric mapping, and a new method for ship target recognition in zoom binocular system images based on homography matrix mapping is proposed, which can solve the defects of the above-mentioned traditional methods.

[0008] The zoom binocular system image ship target recognition method based on homography matrix mapping of the present invention is special in that it includes the following steps:

[0009] Step 1: Use the deep learning YOLOv7 target detection algorithm to achieve target detection and matching between visible light and infrared images obtained by the binocular imaging system;

[0010] Step 2: Use the coordinates and midpoints of the identified target rectangle as corresponding points to calculate the homography matrix and inverse matrix to achieve complementary target recognition and complete the bidirectional mapping between visible light and infrared images;

[0011] Step 3: Use the root mean square error (RMSE) as the geometric error to calculate the degree of change of the homography matrix on different image pairs, proving that the same homography matrix can achieve image mapping in different scenes;

[0012] Step 4: Using the applicability and effectiveness of the homography matrix, by adjusting the focal length of the binocular imaging system, a homography matrix group based on focal length change is constructed;

[0013] Step 5: Call the corresponding homography matrix in the homography matrix group generated in step 4 with known focal length for mapping, so as to achieve missed detection or even zero-detection ship detection and recognition in a certain modal image using only deep learning methods.

[0014] Preferably, the specific steps of step one are: detecting and selecting the sea ship target through deep learning YOLOv7 target detection algorithm, and obtaining the coordinate information of the corner points of the target boundary box.

[0015] Preferably, the specific steps of step 2 are: there is a point p = (x, y) in the visible light image, and there is a corresponding point p' = (x', y') in the infrared image, and the relationship between them is expressed as: p' = Hp, where H is a 3x3 homography matrix, and its elements hij (i, j = 1, 2, 3) are usually set to h33 = 1 as a normalization parameter. P and P' are points represented by homogeneous coordinates;

[0016] Assume that the homogeneous coordinates of P and P' are (x, y, 1) T and (x',y',1) T , then the formula of homography transformation is expressed as:

[0017]

[0018] In order to solve the homography matrix H, at least 4 pairs of corresponding points are required. For each pair of corresponding points (x, y) and (x', y'), two equations are obtained:

[0019]

[0020] Convert the above equation into a linear system, assuming that there are n pairs of corresponding points and (x' i ,y' i ), we get the following matrix equation:

[0021]

[0022] The matrix equation (3) is simplified to Ah = 0, where A is a 2n × 9 matrix and h is a 9-dimensional vector containing the homography matrix elements. Solving this equation by least squares method or singular value decomposition SVD yields the homography matrix h;

[0023] By solving the inverse matrix of the homography matrix, the target in the infrared image can be accurately mapped back to the visible light image. For a given homography matrix H, its inverse matrix H inv Satisfy the formula:

[0024] H×H inv =I(4)

[0025] Where I is the identity matrix, and the inverse matrix of the homography matrix is ​​H inv Corresponding to the conversion operation from the target image back to the original image, the bidirectional mapping ensures the accurate registration and spatial consistency of common objects in multimodal images.

[0026] Preferably, the specific steps of step three are:

[0027] The root mean square error (RMSE) is used as the geometric error to measure the degree of change of the calculated homography matrix on different image pairs. The smaller the RMSE value, the smaller the mapping error and the higher the reliability of the mapping result.

[0028]

[0029] Among them, H i is the homography matrix of the ith image pair, is the average of the homography matrices of all image pairs, ||·|| F Represents the Frobenius norm of the matrix;

[0030] The average mapping error (AME) is used to measure the average deviation between the position of the target point after homography matrix mapping and the actual target point. The smaller the AME value, the smaller the average offset of the mapping point, and the closer the mapping result is to the actual situation. The position error is calculated after the target is mapped by the calculated homography matrix. The calculation method is as follows:

[0031]

[0032] Among them, (x Ci ,y Ci ) is the actual position of the corresponding target, (x Di ,y Di ) is the target position obtained by mapping.

[0033] Preferably, the specific steps of step 4 are:

[0034] The focal length range of the visible light camera in the binocular imaging system is set to 50 mm to 300 mm, and the focal length range of the infrared camera is 30 mm to 100 mm. The focal length change step is 10 mm. Under the condition of fixed binocular camera focal length, the homography matrix H under the focal length is solved. By gradually changing the focal length of the binocular camera, the corresponding homography matrix under different focal length conditions is calculated, and then a homography matrix group {H1,H2,…H n}, the homography matrices of this matrix group are all used for mapping visible light images to infrared images.

[0035] Preferably, the specific steps of step five are:

[0036] When the deep learning algorithm misses a target ship or even detects zero targets during image processing at a specific focal length, the homography matrix group pre-established in step 4 is searched and the homography matrix at adjacent focal lengths is used to supplement the deficiency of the deep learning algorithm, thereby identifying and registering the missed target ships.

[0037] The method proposed in the present invention uses a deep learning algorithm to identify the ship target in the two images of the binocular camera system, and the four corner points and the center point of each target are used as corresponding points to construct the calculation basis of the homography matrix. The constructed homography matrix is ​​used to realize the bidirectional precise mapping between visible light and infrared images, supplementing the target recognition of deep learning. And by adjusting the focal length of the binocular imaging system, a homography matrix group based on focal length change is constructed. When supplementing the missed target recognition of deep learning, the most suitable homography matrix is ​​selected from the matrix library according to the current imaging conditions and focal length for application, so as to realize the detection and recognition of missed detection or even zero detection of ships in a certain modal image using only deep learning methods. The specific analysis of the beneficial effects is as follows:

[0038] (1) Under the condition of a fixed focal length of the binocular camera, the proposed method can capture different sea scenes by changing the azimuth and elevation angles. The norm error of the two homography matrices is 1.9999507297, and the root mean square error is 0.6666502432. The homography matrices show a high degree of consistency, indicating its applicability in dynamic sea environments.

[0039] (2) The proposed method achieves accurate bidirectional mapping between visible light and infrared images. By calculating the homography matrix, the target in the visible light can be accurately mapped to the infrared image. At the same time, the reverse mapping is achieved by solving the inverse matrix of the homography matrix, ensuring that the target in the infrared image can be accurately mapped back to the visible light image. It can supplement the targets that may be missed by the deep learning algorithm. The average deviation after mapping is about 1.52 pixels, which meets the requirements for target positioning accuracy in practical applications.

[0040] (3) The proposed method gradually adjusts the focal length of the binocular camera to solve and construct the corresponding homography matrix group at different focal lengths. By calling the homography matrix under the corresponding focal length conditions to map the detection target, this method can not only supplement the targets that may be missed by the deep learning algorithm, but also accurately identify and supplement the missed targets when zero target detection occurs in a certain image. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a data image; in the figure, (a) is a data infrared image, (b) is a data visible light image;

[0042] Figure 2 Figure 2 is the data 2 image; in the figure, (a) is the infrared image of data 2, and (b) is the visible light image of data 2;

[0043] Figure 3 The image mapping diagram is realized by the homography matrix; in the figure, (a) is the image mapping result of data one, and (b) is the image mapping result of data two;

[0044] Figure 4 Apply the data two mapping to the homography matrix of data one;

[0045] Figure 5 This is a comparison diagram of the target frame inverse matrix mapping result diagram;

[0046] Figure 6 Figure 3 is the data; in the figure, (a) is the infrared image of data 3, and (b) is the visible light image of data 3;

[0047] Figure 7 It is a mapping diagram of zero target image detection; in the figure, (a) is a visible light image with zero target detection, and (b) is a visible light image with missed target detection. DETAILED DESCRIPTION

[0048] In order to better understand and implement, a specific embodiment is given below to explain in detail the zoom binocular system image ship target recognition method based on homography matrix mapping of the present invention. The processing flow of the present invention is described in detail below:

[0049] Example 1

[0050] 1) Detect ship targets in visible light and infrared images obtained by binocular imaging system through deep learning YOLOv7

[0051] A total of 15,000 images were constructed using a binocular imaging system to perform deep learning target detection algorithms for targets in visible light and infrared images. The resolution of the images was 1920×1080, and the dataset was divided into three parts: training set, test set, and validation set in a ratio of 7:2:1. The deep learning YOLOv7 algorithm was used to detect and select the sea ship targets, and the coordinate information of the corner points of the target bounding box was obtained.

[0052] 2) By calculating the homography matrix, the target in the visible light can be accurately mapped to the infrared image

[0053] If there is a point p = (x, y) in the visible light image and a corresponding point p' = (x', y') in the infrared image, the relationship between them can be expressed as: p' = Hp, where H is a 3x3 homography matrix and P and P' are points represented by homogeneous coordinates.

[0054] Assume that the homogeneous coordinates of P and P' are (x, y, 1) T and (x',y',1) T , then the formula of homography transformation can be expressed as:

[0055]

[0056] In order to solve the homography matrix H, usually 4 or more pairs of corresponding points are required. For each pair of corresponding points (x, y) and (x', y'), two equations can be obtained:

[0057]

[0058] In order to convert it into a matrix form, we can use homogeneous coordinates to convert the above equation into a linear system of equations. Assume that there are n pairs of corresponding points and (x' i ,y' i ), we can get the following matrix equation:

[0059]

[0060] This matrix equation can be simplified to Ah = 0, where A is a 2n × 9 matrix and h is a 9-dimensional vector containing the homography matrix elements. This equation can be solved by the least squares method or singular value decomposition (SVD) to obtain the homography matrix h.

[0061] 3) By solving the inverse matrix of the homography matrix, the target in the infrared image can be accurately mapped back to the visible light image

[0062] For a given homography matrix H, its inverse matrix H inv Satisfy the formula: H×H inv =I(4)

[0063] Where I is the identity matrix. In practical applications, the inverse matrix H of the homography matrix inv This bidirectional mapping, corresponding to the transformation operation from the target image back to the original image, ensures accurate registration and spatial consistency of common objects in multimodal images.

[0064] 4) Error analysis of the mapped image

[0065] The root mean square error (RMSE) is used as the geometric error to measure the degree of variation of the calculated homography matrix on different image pairs. The smaller the RMSE value, the smaller the mapping error and the higher the reliability of the mapping result.

[0066]

[0067] Among them, H i is the homography matrix of the ith image pair, is the average of the homography matrices of all image pairs, ||·|| F Represents the Frobenius norm of a matrix.

[0068] The average mapping error (AME) is used to measure the average deviation between the position of the target point after homography matrix mapping and the actual target point. The smaller the AME value, the smaller the average offset of the mapped point, and the closer the mapping result is to the actual situation. The target is mapped by the calculated homography matrix and the position error is calculated as follows:

[0069]

[0070] Among them, (x Ci ,y Ci ) is the actual position of the corresponding target, (x Di ,y Di ) is the target position obtained by mapping.

[0071] 5) Construct the corresponding homography matrix group and call the corresponding homography matrix for mapping by knowing the focal length.

[0072] The focal length range of the visible light camera in the binocular imaging system of the present invention is set to 50 mm to 300 mm, and the focal length range of the infrared camera is set to 30 mm to 100 mm, and the focal length change step is 10 mm. When the focal length of the binocular camera is fixed, the homography matrix H under the focal length is solved using the method proposed in this article. By gradually changing the focal length of the binocular camera, the corresponding homography matrix under different focal length conditions is calculated, and then a homography matrix group {H1, H2, ... H n}, the homography matrices in this matrix group are used for mapping visible light images to infrared images. When the deep learning algorithm misses a target ship or even detects zero targets during image processing at a specific focal length, the pre-established homography matrix group can be found and the homography matrices at adjacent focal lengths can be used to supplement the deficiency of the deep learning algorithm, thereby detecting and registering the missed target ships.

[0073] Example 2

[0074] Using the target coordinates obtained by deep learning, the homography matrix is ​​calculated for the coordinates of the two images corresponding to the target and the center point of the target in each set of data. Figure 1 (a)(b)) and Data 2( Figure 2The homography matrices of (a)(b)) are shown in Table 1H1 and H2. H1 and H2 map the visible light images in Data 1 and 2 to infrared images respectively. The Frobenius norm error value of the two homography matrices calculated in Table 1 is: 1.9999507297, and the root mean square error (RMSE) is 0.6666502432, indicating that the average difference between the two homography matrices is relatively small. The Frobenius norm error represents the overall difference of the matrix elements. In the image with an experimental resolution of 1920×1080 pixels in this paper, the homography matrix is ​​almost unchanged. When the error matrix is ​​applied for image mapping, the point offset in the image is about 2 pixels, and it usually does not significantly affect the overall effect in a large range of regional alignment.

[0075] The mapping results of H1 and H2 are as follows Figure 3 As shown, Figure 3 The yellow rectangle in (a) is the common target recognized by the binocular system, mapped using H1, the green lines indicate the correspondence between the targets, and the red rectangle shows the mapping result of the target in the visible light image of the data in the infrared image after the homography matrix transformation. Figure 3 The yellow rectangular box in (b) is the common target recognized by the binocular system, which is mapped using H2. The green lines indicate the correspondence between the targets, and the red rectangular box shows the mapping result of the target in the visible light image of data 2 in the infrared image after the homography matrix transformation. Figure 3 (a)(b) The results show that by utilizing the method of the present invention, the homography matrix obtained by fusing deep learning with geometric mapping makes the mapping positions of the four vertices of the visible light image in the infrared highly consistent with the expected positions, verifying the accuracy of the method proposed in the present invention.

[0076] Table 1 Homography matrix value table of data 1

[0077]

[0078]

[0079] Example 3

[0080] Use the homography matrix H1 calculated from data one to transform data two Figure 2 The target box coordinates in (b) are mapped to Figure 2 (a). The actual target coordinates (i.e., the target coordinates detected by deep learning in data 2) and the target coordinates after H1 mapping are shown in Table 2. The target mapping error (average error) is calculated to be 1.52, and the pixel offset is relatively small. Figure 4In the figure, the actual position of the visible light target is marked with a green box in the left figure, the target mapping completed by the homography matrix H1 is marked with a red box in the right figure, and the actual target position is marked with a green box in the right figure. The two rectangular boxes in the infrared image on the right are almost overlapped, and the method proposed in the present invention is suitable for the requirements of target positioning accuracy in practical applications.

[0081] Table 2 Target coordinate comparison table

[0082]

[0083] Example 4

[0084] Use the homography matrix H1 calculated from data 1 to solve its inverse matrix H1 inv , as shown in Table 3.

[0085] Table 3. Table of inverse matrix values ​​of homography matrix of data 1

[0086]

[0087] Leverage H1 inv The data Figure 2 The target box coordinates in (a) are mapped to Figure 2 (b). The actual target coordinates (i.e., the target coordinates detected by deep learning in data 2) and H1 inv The mapped target coordinates are shown in Table 4. The target mapping error (average error) is calculated to be 4.82. This error value indicates that when mapping the target box from image 2(a) to Figure 2 In the process of (b), the average deviation between the vertex of the target box and the actual position is about 4.82 pixels, and the pixel offset is relatively small. Since the matrix inverse operation itself is prone to introduce errors, H1 in Example 3 inv The mapping error is larger than that of Example 2H1.

[0088] Table 4 Target coordinate comparison table

[0089]

[0090] Figure 5 In the left figure, the actual position of the infrared target is marked with a green box, using the homography matrix H1 inv The completed target mapping is marked with a red box in the right figure, and the actual target position is marked with a green box in the right figure. The two rectangular boxes of the visible light image in the right figure almost overlap. The method proposed in the present invention performs well in the target mapping task, the error is within an acceptable range, and is suitable for the requirements of target positioning accuracy in practical applications.

[0091] Example 5

[0092] The focal length of the binocular camera is set to 3 mm for the infrared phase and 15 mm for the visible light camera. Figure 6 (a) Figure 6 As shown in (b), each image contains a ship. Figure 7 As shown in (a), the ship target in the infrared image on the left is identified, and the green rectangular box is the detected target. The visible light image on the right has zero detection, and the ship is not identified. Through the focal length information of data three, the corresponding homography matrix in the constructed matrix group is called. Since the homography matrix of the matrix group is to map the visible light image to the infrared image, the mapping of the ship target detected in the infrared image to the visible light image requires solving its inverse matrix. Through the inverse matrix, the zero detection target of the visible light image is identified and selected, as shown in Figure 7 As shown in (b), the red rectangular box on the right is the target after mapping.

[0093] Experimental results show that by calling the corresponding homography matrix with known focal length, effective target recognition and selection can be performed for deep learning missed detection or even zero target detection. The proposed method effectively improves the robustness of target detection, especially in dynamic and complex marine environments, ensuring the consistency and reliability of multimodal data fusion in the zoom binocular system.

Claims

1. A method for ship target recognition in zoom binocular system images based on homography matrix mapping, characterized in that The following steps are involved: Step 1: Use the deep learning YOLOv7 target detection algorithm to achieve target detection and matching between visible light and infrared images obtained by the binocular imaging system; Step 2: Use the coordinates and midpoints of the identified target rectangle as corresponding points to calculate the homography matrix and inverse matrix to achieve complementary target recognition and complete the bidirectional mapping between visible light and infrared images; Step 3: Use the root mean square error (RMSE) as the geometric error to calculate the degree of change of the homography matrix on different image pairs, proving that the same homography matrix can achieve image mapping in different scenes; Step 4: Using the applicability and effectiveness of the homography matrix, by adjusting the focal length of the binocular imaging system, a homography matrix group based on focal length change is constructed; Step 5: Call the corresponding homography matrix in the homography matrix group generated in step 4 with known focal length for mapping, so as to achieve missed detection or even zero-detection ship detection and recognition in a certain modal image using only deep learning methods.

2. The method for ship target recognition based on homography matrix mapping in zoom binocular system image according to claim 1, characterized in that The specific steps of step one are: detecting and selecting the sea ship target through deep learning YOLOv7 target detection algorithm, and obtaining the coordinate information of the corner points of the target boundary box.

3. The method for ship target recognition based on homography matrix mapping in zoom binocular system image according to claim 1, characterized in that The specific steps of step 2 are as follows: there is a point p=(x, y) in the visible light image, and there is a corresponding point p'=(x', y') in the infrared image, and the relationship between them is expressed as: p'=Hp, where H is a 3x3 homography matrix, and its elements hij (i, j=1, 2, 3) are usually set to h33=1 as normalization parameters, and P and P' are points represented by homogeneous coordinates; Assume that the homogeneous coordinates of P and P' are (x, y, 1) T and (x',y',1) T , then the formula of homography transformation is expressed as: In order to solve the homography matrix H, at least 4 pairs of corresponding points are required. For each pair of corresponding points (x, y) and (x', y'), two equations are obtained: Convert the above equation into a linear system, assuming that there are n pairs of corresponding points and (x' i ,y' i ), we get the following matrix equation: The matrix equation (3) is simplified to Ah = 0, where A is a 2n × 9 matrix and h is a 9-dimensional vector containing the homography matrix elements. Solving this equation by least squares method or singular value decomposition SVD yields the homography matrix h; By solving the inverse matrix of the homography matrix, the target in the infrared image can be accurately mapped back to the visible light image. For a given homography matrix H, its inverse matrix H inv Satisfy the formula: H×H inv =I(4) Where I is the identity matrix, and the inverse matrix of the homography matrix is ​​H inv Corresponding to the conversion operation from the target image back to the original image, the bidirectional mapping ensures the accurate registration and spatial consistency of common objects in multimodal images.

4. The method for ship target recognition based on homography matrix mapping in zoom binocular system image according to claim 1, characterized in that The specific steps of step three are: The root mean square error (RMSE) is used as the geometric error to measure the degree of change of the calculated homography matrix on different image pairs. The smaller the RMSE value, the smaller the mapping error and the higher the reliability of the mapping result. Among them, H i is the homography matrix of the ith image pair, is the average of the homography matrices of all image pairs, ‖·‖ F Represents the Frobenius norm of the matrix; The average mapping error (AME) is used to measure the average deviation between the position of the target point after homography matrix mapping and the actual target point. The smaller the AME value, the smaller the average offset of the mapping point, and the closer the mapping result is to the actual situation. The position error is calculated after the target is mapped by the calculated homography matrix. The calculation method is as follows: Among them, (x Ci ,y Ci ) is the actual position of the corresponding target, (x Di ,y Di ) is the target position obtained by mapping.

5. The method for ship target recognition based on homography matrix mapping in zoom binocular system image according to claim 1, characterized in that The specific steps of step 4 are: The focal length range of the visible light camera in the binocular imaging system is set to 50 mm to 300 mm, and the focal length range of the infrared camera is 30 mm to 100 mm. The focal length change step is 10 mm. Under the condition of fixed binocular camera focal length, the homography matrix H under the focal length is solved. By gradually changing the focal length of the binocular camera, the corresponding homography matrix under different focal length conditions is calculated, and then a homography matrix group {H1,H2,…H n }, the homography matrices of this matrix group are all used for mapping visible light images to infrared images.

6. The method for ship target recognition based on homography matrix mapping in zoom binocular system images according to claim 1, characterized in that The specific steps of step five are: When the deep learning algorithm misses a target ship or even detects zero targets during image processing at a specific focal length, the homography matrix group pre-established in step 4 is searched and the homography matrix at adjacent focal lengths is used to supplement the deficiency of the deep learning algorithm, thereby identifying and registering the missed target ships.

Images