Target recognition method, system, device and medium based on multi-information fusion

Abstract

The present invention provides a target recognition method, system, device and medium based on multivariate information fusion, and the method includes the following steps: collecting the real-time three-dimensional geographic coordinates of the ship and converting them into a real two-dimensional image point matrix; collecting the real-time two-dimensional image of the ship to form a two-dimensional image point matrix; collecting the real-time laser three-dimensional geographic coordinates of the ship and converting them into a laser two-dimensional image point matrix; fusing the laser two-dimensional image point matrix with the two-dimensional image point matrix to obtain a fused image with a fused two-dimensional image point matrix, and calculating the overlap with the real two-dimensional image; for a fused image with an overlap threshold of 80% or more, using a convolutional neural network for iterative optimization to determine the target frame required for positioning and identifying the target in the fused image. The present invention fuses multi-sensor data to obtain more comprehensive target information, which can improve the detection accuracy of ships and expand the detection range, and can also provide more comprehensive information for safeguarding marine rights and interests.

Description

Technical Field

[0001] The present invention belongs to the technical field of target recognition, and in particular relates to a target recognition method, system, device and medium based on multi-information fusion. Background Art

[0002] The Automatic Identification System (AIS) can provide relevant information about cooperative ship targets in the monitoring area, and the identity information of the ship targets can be obtained based on the AIS data. However, AIS is a passive sensor and cannot detect non-cooperative ship targets at sea.

[0003] Synthetic aperture radar (SAR) has high resolution and strong maneuverability, and can accurately detect the location of ship targets on the ocean. SAR can also detect all ship targets in the monitoring area, but it cannot provide the identity information of the ship targets. SAR is an active microwave imaging radar that can provide high-resolution images and has all-day and all-weather detection characteristics, and can monitor targets in any weather. SAR is carried on satellites, aircraft, and drones. With the development of synthetic aperture technology, SAR can also be installed on cars. Among them, airborne SAR has fewer restrictions and can be used flexibly for real-time imaging to monitor ship targets. Airborne SAR has high spatial resolution and strong maneuverability, and the detected ship position information is relatively accurate, but airborne SAR is more easily affected by weather conditions.

[0004] The tri-spectrum camera is a long-distance monitoring device that can meet the needs of effective dynamic monitoring within a range of 3-10 kilometers and timely tracking of hotspot warnings. It has dual-channel imaging output of visible light and far-infrared thermal sensing, and can complement day and night and provide 24-hour uninterrupted monitoring.

[0005] Although the sensors used for the three types of ship target monitoring have their own characteristics, using only one target sensor will result in a single source of sampling data, which will lead to inaccurate target tracking and monitoring. Although there are some multi-sensor fusion target monitoring algorithms in the prior art, such as a multi-sensor sequential fusion tracking method based on UT-PHD disclosed in 201911041389.3 or a multi-sensor decision-level fusion intelligent ship surface target perception and recognition method disclosed in 201911125608.6, the noise is too large and the feasibility is not high. The data collected by multiple sensors are not effectively transformed and fused, and the target selection box for the final target recognition is optimized, resulting in a decrease in the accuracy of target recognition and the occurrence of defects such as missing the target that needs to be tracked and recognized. Summary of the invention

[0006] In view of the above-mentioned defects, the present invention provides a target recognition method, system, device and medium based on multi-information fusion. The present invention fuses multi-sensor data to obtain more comprehensive target information, which can improve the detection accuracy of ships and expand the detection range, and can also provide more comprehensive information for safeguarding marine rights and interests.

[0007] The present invention provides the following technical solution: a target recognition method based on multivariate information fusion, comprising the following steps;

[0008] S1: collecting the real-time three-dimensional geographic coordinates of the vessel, and the real-time real three-dimensional geographic coordinate matrix forms a real three-dimensional point cloud; converting the real three-dimensional point cloud into a real two-dimensional image point matrix;

[0009] S2: collecting a real-time two-dimensional image of the vessel to form a two-dimensional image point set, wherein the coordinates of the two-dimensional image points in the set form a two-dimensional image point matrix; simultaneously collecting the real-time laser three-dimensional geographic coordinates of the vessel, wherein the real-time laser three-dimensional geographic coordinate matrix forms a laser three-dimensional point cloud; and converting the laser three-dimensional point cloud into a laser two-dimensional image point matrix;

[0010] S3: fusing the laser two-dimensional image dot matrix with the two-dimensional image dot matrix to obtain a fused image having a fused two-dimensional image dot matrix, and calculating the degree of overlap between the obtained fused image and the two-dimensional image formed by the real two-dimensional image dot matrix obtained in step S1;

[0011] S4: Determine whether the overlap degree is greater than or equal to a threshold value of 80%. If the overlap degree reaches the threshold value, proceed to the next step, otherwise repeat steps S1-S4;

[0012] S5: Use a convolutional neural network to iteratively optimize the two-dimensional image dot matrix obtained by fusion in step S3 that meets the overlap threshold, build a minimum iterative loss model, obtain the optimized two-dimensional image dot matrix with the minimum iterative loss, and determine the target border required for positioning and identifying the target in the image after the laser two-dimensional image dot matrix and the two-dimensional image dot matrix are fused.

[0013] Furthermore, the real-time real three-dimensional geographic coordinate matrix formed by the real-time real three-dimensional geographic coordinates collected in step S1 is X real =(x real ,y real ,z real ,1) T , where x real is the actual x-axis coordinate of the ship, y real is the actual y-axis coordinate of the ship, z real is the real z-axis coordinate of the ship;

[0014] The real three-dimensional point cloud is converted into a real two-dimensional image point matrix Y using the following formula:real :

[0015] Y real =T real X real

[0016] Among them, T real is the real 3D point cloud transformation matrix, d i is the distance between the i-th point in the converted real two-dimensional image and the real point, (c x ,c y ) is the center point of the image composed of the real two-dimensional image points obtained by transformation, b i It is the baseline of the image composed of the real two-dimensional image points obtained by relative transformation.

[0017] Furthermore, the step S2 converts the laser three-dimensional point cloud formed by the real-time laser three-dimensional geographic coordinates into a laser two-dimensional image point matrix Y Lidar The formula is as follows:

[0018] Y Lidar =T Lidar RS Lidar X Lidar

[0019] The real-time laser 3D geographic coordinate matrix X Lidar =(x l ,y l ,z l ,1) T , x l is the x-axis coordinate of the ship measured by the lidar, l is the y-axis coordinate of the ship measured by the lidar, z l is the z-axis coordinate of the ship measured by the laser radar; R is the laser 3D point cloud rotation matrix, which is an orthogonal matrix with a determinant of 1; T Lidar is the laser 3D point cloud transformation matrix, d j is the distance between the jth point in the converted laser 2D image and the real point, (m x ,m y ) is the center point of the image composed of the converted laser two-dimensional image points, b j is the baseline of the image composed of the laser two-dimensional image points obtained by relative conversion; S Liadar is the transformation matrix, R Lidar is an orthogonal matrix with determinant 1.

[0020] Furthermore, the method of fusing the laser two-dimensional image point matrix with the two-dimensional image point matrix in step S3 comprises the following steps:

[0021] S31: constructing a two-dimensional image point set A obtained by splicing two-dimensional image points in the laser two-dimensional image point matrix and two-dimensional image points in the two-dimensional image point matrix;

[0022] S32: Divide the two-dimensional image point set A into N×N grids, and determine the position of the w-th two-dimensional image point in the grids.

[0023] S33: construct a maximum similarity model so that the laser two-dimensional image point matrix and the two-dimensional image point matrix Y Cam For maximum integration.

[0024] Furthermore, the maximum similarity model constructed in step S33 is as follows:

[0025]

[0026]

[0027] Where j = 1, 2, 3, 4; M is Col, Tex, Siz or Shape, is the color parameter of the fused image, is the texture parameter of the fused image, is the size parameter of the fused image in pixels, is the image shape compatibility parameter in the fused image; represent The matrix Y of the laser two-dimensional image points in the two-dimensional image point set A obtained by splicing the two-dimensional image points divided into N×N grids is Lidar The pth point of the image composed of represent The two-dimensional image point set A is obtained by splicing the two-dimensional image point matrix Y which is divided into N×N grids. Cam The qth point of .

[0028] Furthermore, the calculation formula of the color parameters of the fused image is as follows:

[0029]

[0030] The calculation formula of the texture parameters of the fused image is as follows:

[0031]

[0032] The calculation formula of the size parameter of the fused image in pixels is as follows:

[0033]

[0034] The calculation formula of the image shape compatibility parameter in the fused image is as follows:

[0035]

[0036] Indicates the two-dimensional image point matrix Y belonging to the laser in the two-dimensional image point set A in the color descriptor Lidar The kth histogram value of the image composed of Represents the two-dimensional image point set A in the color descriptor that belongs to the two-dimensional image point matrix Y Cam The kth histogram value of the composed image; Represents the matrix Y of the laser two-dimensional image points in the two-dimensional image point set A in the texture descriptor Lidar The kth histogram value of the image composed of, Represents the two-dimensional image point matrix Y in the two-dimensional image point set A in the texture descriptor Cam The kth histogram value of the image composed of; Siz(·) represents the image size calculation function in pixels.

[0037] Furthermore, the minimum iterative loss model in step S4 is as follows:

[0038]

[0039]

[0040] Among them, σ is the weight coefficient of the loss caused by the smoothness function, s is the iterative algebra, smooth(·) is the smoothness function of the image formed by multiple two-dimensional image points in the two-dimensional image point set, is the Euclidean distance between two adjacent w-th two-dimensional image points and the w-1-th two-dimensional image point.

[0041] The present invention also provides a target recognition system based on multivariate information fusion using the above method, including an AIS ship automatic identification system, a laser radar system, a tri-spectrum camera and a main control calculation module;

[0042] The AIS automatic ship identification system is used to collect the real-time three-dimensional geographic coordinates of the ship;

[0043] The tri-spectral camera is used to collect real-time two-dimensional images of the vessel;

[0044] The laser radar system is used to collect the real-time laser three-dimensional geographic coordinate matrix of the ship;

[0045] The main control calculation module is used to receive the real-time three-dimensional geographic coordinates of the ship to form a real three-dimensional point cloud, and convert it into a real two-dimensional image point matrix, receive the real-time two-dimensional image of the ship to form a two-dimensional image point set and a two-dimensional image point matrix, receive the real-time laser three-dimensional geographic coordinate matrix of the ship to form a laser three-dimensional point cloud, and convert the laser three-dimensional point cloud into a laser two-dimensional image point matrix; fuse the laser two-dimensional image point matrix with the two-dimensional image point matrix to obtain a fused image with a fused two-dimensional image point matrix, and use a convolutional neural network to iteratively optimize the two-dimensional image dot matrix that meets the overlap threshold, and determine the target frame required for positioning and identifying the target in the image after the laser two-dimensional image point matrix and the two-dimensional image point matrix are fused.

[0046] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the target recognition method based on multi-information fusion as described above when executing the computer program.

[0047] The present invention also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target recognition method based on multi-information fusion as described above are implemented.

[0048] The beneficial effects of the present invention are:

[0049] 1. The present invention fuses the three-dimensional data collected by the ship automatic identification system AIS, the three-dimensional data collected by the laser radar system and the two-dimensional data collected by the three-spectrum camera multi-sensor to obtain more comprehensive target information. Before fusion, the three-dimensional data collected by the laser radar system is converted into two-dimensional image coordinates to form a two-dimensional image and fuse it with the real-time two-dimensional image collected by the three-spectrum camera. In the fusion process, a maximum similarity model is constructed to make the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam The maximum degree of integration can effectively improve the detection accuracy of ships and expand the detection range, and can also provide more comprehensive information for safeguarding maritime rights and interests.

[0050] 2. The present invention obtains a fused two-dimensional image point matrix Y fuse Before optimizing and iterating the fused image, the degree of overlap between the fused image and the real two-dimensional image converted from the real three-dimensional geographic coordinates collected by the ship automatic identification system AIS and the real three-dimensional point cloud that forms the real three-dimensional geographic coordinate matrix is ​​calculated. By calculating the degree of overlap, it is determined whether it meets the requirements of the optimization iteration, and then the optimization iteration is performed again, which ensures the accuracy, feasibility and reliability of the identified target of the optimization iteration.

[0051] 3. The present invention converts the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam The point matrix Y with fused two-dimensional image is obtained by performing point trace association fusion fuse By using the fused image, track generation and track association, the continuous time ship target point tracks are generated into ship target tracks, which can obtain more comprehensive information about cooperative ship targets at sea and non-cooperative ship targets.

[0052] 4. In the optimization iteration process, the present invention constructs a minimum iteration loss model, takes the smoothness function of each image point in the grid in the fused image as the limiting value, minimizes the iteration loss of the iterative fusion image grid, ensures that the fusion is carried out with the minimum loss of image smoothness and authenticity, and obtains the optimal iteration result, and determines the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam The target frame required for locating and identifying the target in the fused image, within which the identified target is, effectively improves the accuracy of multi-sensor fusion in determining the identified target from another aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The present invention will be described in more detail below based on embodiments and with reference to the accompanying drawings, wherein:

[0054] Figure 1 A schematic diagram of the flow of a target recognition method based on multi-information fusion provided by the present invention;

[0055] Figure 2 This is a schematic diagram of the target recognition structure based on multi-information fusion provided by the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Example 1

[0058] like Figure 1 As shown, the target recognition method based on multivariate information fusion provided in this embodiment includes the following steps:

[0059] S1: Collect the real-time 3D geographic coordinates of the ship, and the real-time real 3D geographic coordinate matrix forms a real 3D point cloud; convert the real 3D point cloud into a real 2D image point matrix;

[0060] S2: Collect the real-time two-dimensional image of the ship to form a two-dimensional image point set, and the coordinates of the two-dimensional image points in the set form a two-dimensional image point matrix; at the same time, collect the real-time laser three-dimensional geographic coordinates of the ship, and the real-time laser three-dimensional geographic coordinate matrix forms a laser three-dimensional point cloud; convert the laser three-dimensional point cloud into a laser two-dimensional image point matrix;

[0061] S3: Fusing the laser two-dimensional image dot matrix with the two-dimensional image dot matrix to obtain a fused image having a fused two-dimensional image dot matrix, and calculating the degree of overlap between the obtained fused image and the two-dimensional image formed by the real two-dimensional image dot matrix obtained in step S1;

[0062] S4: Determine whether the overlap degree is greater than or equal to a threshold value of 80%. If the overlap degree reaches the threshold value, proceed to the next step, otherwise repeat steps S1-S4;

[0063] S5: Use convolutional neural network to iteratively optimize the two-dimensional image dot matrix obtained by fusion in step S3 that meets the overlap threshold, build a minimum iterative loss model, obtain the optimized two-dimensional image dot matrix with the minimum iterative loss, and determine the target border required for locating and identifying the target in the image after the fusion of the laser two-dimensional image dot matrix and the two-dimensional image dot matrix. The target border is the identified target.

[0064] Example 2

[0065] like Figure 1 As shown, the target recognition method based on multivariate information fusion provided in this embodiment includes the following steps:

[0066] S1: The real-time three-dimensional geographic coordinates of the ship collected by the ship automatic identification system AIS are formed by the real-time three-dimensional geographic coordinates X real =(x real ,y real ,z real ,1) T , where x real is the actual x-axis coordinate of the ship, y real is the actual y-axis coordinate of the ship, z real is the real z-axis coordinate of the ship; real-time three-dimensional geographic coordinate matrix X real Form a real three-dimensional point cloud; use the following formula to convert the real three-dimensional point cloud into a real two-dimensional image point matrix Y real :

[0067] Y real =T real X real

[0068] Among them, T realis the real 3D point cloud transformation matrix, d i is the distance between the i-th point in the converted real two-dimensional image and the real point, (c x ,c y ) is the center point of the image composed of the real two-dimensional image points obtained by transformation, b i It is the baseline of the image composed of the real two-dimensional image points obtained by relative transformation.

[0069] S2: The three-spectral camera collects real-time two-dimensional images of the ship to form a set of two-dimensional image points. The coordinates of the two-dimensional image points in the set form a two-dimensional image point matrix Y Cam ; At the same time, the laser radar system collects the real-time laser three-dimensional geographic coordinates of the ship, and the real-time laser three-dimensional geographic coordinate matrix X Lidar Form a laser three-dimensional point cloud; use the following formula to convert the real-time laser three-dimensional geographic coordinates X Lidar The formed laser 3D point cloud is converted into a laser 2D image point matrix Y Lidar The formula is as follows:

[0070] Y Lidar =T Lidar RS Lidar X Lidar

[0071] The real-time laser 3D geographic coordinate matrix X Lidar =(x l ,y l ,z l ,1) T , x l is the x-axis coordinate of the ship measured by the lidar, l is the y-axis coordinate of the ship measured by the lidar, z l is the z-axis coordinate of the ship measured by the laser radar; R is the laser 3D point cloud rotation matrix, which is an orthogonal matrix with a determinant of 1; T Lidar is the laser 3D point cloud transformation matrix, d j is the distance between the jth point in the converted laser 2D image and the real point, (m x ,m y ) is the center point of the image composed of the converted laser two-dimensional image points, b j is the baseline of the image composed of the laser two-dimensional image points obtained by relative conversion; S Liadar is the transformation matrix,

[0072] R Lidar is an orthogonal matrix with determinant 1, R Lidar Can be

[0073] t Lidar It is the translation vector matrix generated during the transformation process.

[0074]

[0075] and They are the translations generated during the conversion of the coordinates of the 3D laser radar system into a 2D plane coordinate system.

[0076] S3: The laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam Fusion is performed to obtain a fused two-dimensional image point matrix Y fuse The fused image is combined with the real two-dimensional image point matrix Y obtained in step S1 real The coincidence degree of the formed two-dimensional image is calculated;

[0077] S4: Determine whether the overlap degree is greater than or equal to a threshold value of 80%. If the overlap degree reaches the threshold value, proceed to the next step, otherwise repeat steps S1-S4;

[0078] S5: The two-dimensional image dot matrix Y obtained by iteratively optimizing the S3 step that meets the overlap threshold using a convolutional neural network fuse , construct the minimum iteration loss model, obtain the optimized two-dimensional image point matrix with the minimum iteration loss, and determine the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam The target frame required for locating and identifying the target in the fused image, and the target within the target frame is the identified target.

[0079] Example 3

[0080] like Figure 1 As shown, the target recognition method based on multivariate information fusion provided in this embodiment includes the following steps:

[0081] S1: The real-time three-dimensional geographic coordinates of the ship collected by the ship automatic identification system AIS are formed by the real-time three-dimensional geographic coordinates X real =(x real ,y real ,z real ,1) T , where x real is the actual x-axis coordinate of the ship, y real is the actual y-axis coordinate of the ship, z real is the real z-axis coordinate of the ship; real-time three-dimensional geographic coordinate matrix X realForm a real three-dimensional point cloud; use the following formula to convert the real three-dimensional point cloud into a real two-dimensional image point matrix Y real :

[0082] Y real =T real X real

[0083] Among them, T real is the real 3D point cloud transformation matrix, d i is the distance between the i-th point in the converted real two-dimensional image and the real point, (c x ,c y ) is the center point of the image composed of the real two-dimensional image points obtained by transformation, b i It is the baseline of the image composed of the real two-dimensional image points obtained by relative transformation.

[0084] S2: The three-spectral camera collects real-time two-dimensional images of the ship to form a set of two-dimensional image points. The coordinates of the two-dimensional image points in the set form a two-dimensional image point matrix Y Cam ; At the same time, the laser radar system collects the real-time laser three-dimensional geographic coordinates of the ship, and the real-time laser three-dimensional geographic coordinate matrix X Lidar Form a laser three-dimensional point cloud; use the following formula to convert the real-time laser three-dimensional geographic coordinates X Lidar The formed laser 3D point cloud is converted into a laser 2D image point matrix Y Lidar The formula is as follows:

[0085] Y Lidar =T Lidar RS Lidar X Lidar

[0086] The real-time laser 3D geographic coordinate matrix X Lidar =(x l ,y l ,z l ,1) T , x l is the x-axis coordinate of the ship measured by the lidar, l is the y-axis coordinate of the ship measured by the lidar, z l is the z-axis coordinate of the ship measured by the laser radar; R is the laser 3D point cloud rotation matrix, which is an orthogonal matrix with a determinant of 1; T Lidar is the laser 3D point cloud transformation matrix, d j is the distance between the jth point in the converted laser 2D image and the real point, (m x ,m y ) is the center point of the image composed of the converted laser two-dimensional image points, bj is the baseline of the image composed of the laser two-dimensional image points obtained by relative conversion; S Liadar is the transformation matrix,

[0087] R Lidar is an orthogonal matrix with determinant 1, R Lidar Can be

[0088] t Lidar It is the translation vector matrix generated during the transformation process.

[0089]

[0090] and They are the translations generated during the conversion of the coordinates of the 3D laser radar system into a 2D plane coordinate system.

[0091] S3: The laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam Fusion is performed to obtain a fused two-dimensional image point matrix Y fuse The fused image includes the following steps:

[0092] S31: Construct laser two-dimensional image point matrix Y Lidar The two-dimensional image points and the two-dimensional image point matrix Y Cam A set of two-dimensional image points A obtained by concatenating the two-dimensional image points in ; Two-dimensional image point matrix Y Cam =(u,v,1), where u is the three-spectrum camera The x-axis coordinate of the two-dimensional image vessel measured by the camera, v is the y-axis coordinate of the two-dimensional image vessel measured by the tri-spectral camera;

[0093] S32: Divide the two-dimensional image point set A into N×N grids, and determine the position of the w-th two-dimensional image point in the grids.

[0094] Preferably, N×N is 3×3 or 4×4;

[0095] S33: Construct the following maximum similarity model so that the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam For maximum integration.

[0096]

[0097]

[0098] Where j = 1, 2, 3, 4; M is Col, Tex, Siz or Shape, is the color parameter of the fused image, is the texture parameter of the fused image, is the size parameter of the fused image in pixels, is the image shape compatibility parameter in the fused image; represent The matrix Y of the laser two-dimensional image points in the two-dimensional image point set A obtained by splicing the two-dimensional image points divided into N×N grids is Lidar The pth point of the image composed of represent The two-dimensional image point set A is obtained by splicing the two-dimensional image point matrix Y which is divided into N×N grids. Cam The qth point of .

[0099] The calculation formula of the color parameters of the fused image is as follows:

[0100]

[0101] The calculation formula of the fused image texture parameters is as follows:

[0102]

[0103] The calculation formula of the size parameter of the fused image in pixels is as follows:

[0104]

[0105] The calculation formula of image shape compatibility parameter in the fused image is as follows:

[0106]

[0107] Indicates the two-dimensional image point matrix Y belonging to the laser in the two-dimensional image point set A in the color descriptor Lidar The kth histogram value of the image composed of Represents the two-dimensional image point set A in the color descriptor that belongs to the two-dimensional image point matrix Y Cam The kth histogram value of the composed image; Represents the matrix Y of the laser two-dimensional image points in the two-dimensional image point set A in the texture descriptor Lidar The kth histogram value of the image composed of Represents the two-dimensional image point matrix Y in the two-dimensional image point set A in the texture descriptor Cam The kth histogram value of the image composed of; Siz(·) represents the image size calculation function in pixels, The matrix Y of the laser two-dimensional image points in the two-dimensional image point set A obtained by splicing the two-dimensional image points divided into N×N grids is Lidar The image size of the p-th point of the composed image, The two-dimensional image point set A is obtained by splicing the two-dimensional image point matrix Y which is divided into N×N grids. Cam Siz(A) is the image size of the qth point of the spliced ​​two-dimensional image point set A.

[0108] The obtained fused image is combined with the real two-dimensional image point matrix Y obtained in step S1 real The overlap degree of the formed two-dimensional image is calculated. Suppose the image size of the fused image is S and the real two-dimensional image point matrix Y real The image size of the formed two-dimensional image is H, and the calculation formula of the coincidence degree is as follows:

[0109]

[0110] S4: Determine whether the overlap degree is greater than or equal to a threshold value of 80%. If the overlap degree reaches the threshold value, proceed to the next step, otherwise repeat steps S1-S4;

[0111] S5: The two-dimensional image dot matrix Y obtained by iteratively optimizing the S3 step that meets the overlap threshold using a convolutional neural network fuse , construct the minimum iteration loss model, obtain the optimized two-dimensional image point matrix with the minimum iteration loss, and determine the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam The target frame required for locating and identifying the target in the fused image, and the target within the target frame is the identified target.

[0112] Example 4

[0113] The target recognition method based on multivariate information fusion provided in this embodiment includes the following steps:

[0114] S1: The real-time three-dimensional geographic coordinates of the ship collected by the ship automatic identification system AIS are formed by the real-time three-dimensional geographic coordinates X real =(x real ,y real ,z real ,1) T , where x real is the actual x-axis coordinate of the ship, y real is the actual y-axis coordinate of the ship, z real is the real z-axis coordinate of the ship; real-time three-dimensional geographic coordinate matrix X real Form a real three-dimensional point cloud; use the following formula to convert the real three-dimensional point cloud into a real two-dimensional image point matrix Y real :

[0115] Y real =T real X real

[0116] Among them, T real is the real 3D point cloud transformation matrix, d i is the distance between the i-th point in the converted real two-dimensional image and the real point, (c x ,c y ) is the center point of the image composed of the real two-dimensional image points obtained by transformation, b i It is the baseline of the image composed of the real two-dimensional image points obtained by relative transformation.

[0117] S2: The three-spectral camera collects real-time two-dimensional images of the ship to form a set of two-dimensional image points. The coordinates of the two-dimensional image points in the set form a two-dimensional image point matrix Y Cam ; At the same time, the laser radar system collects the real-time laser three-dimensional geographic coordinates of the ship, and the real-time laser three-dimensional geographic coordinate matrix X Lidar Form a laser three-dimensional point cloud; use the following formula to convert the real-time laser three-dimensional geographic coordinates X Lidar The formed laser 3D point cloud is converted into a laser 2D image point matrix Y Lidar The formula is as follows:

[0118] Y Lidar =T Lidar RS Lidar X Lidar

[0119] The real-time laser 3D geographic coordinate matrix X Lidar =(x l ,y l ,z l ,1) T , x l is the x-axis coordinate of the ship measured by the lidar, l is the y-axis coordinate of the ship measured by the lidar, z l is the z-axis coordinate of the ship measured by the laser radar; R is the laser 3D point cloud rotation matrix, which is an orthogonal matrix with a determinant of 1; T Lidar is the laser 3D point cloud transformation matrix, d j is the distance between the jth point in the converted laser 2D image and the real point, (m x ,m y ) is the center point of the image composed of the converted laser two-dimensional image points, b j is the baseline of the image composed of the laser two-dimensional image points obtained by relative conversion; S Liadar is the transformation matrix,

[0120] R Lidaris an orthogonal matrix with determinant 1, R Lidar Can be

[0121] t Lidar It is the translation vector matrix generated during the transformation process.

[0122]

[0123] and They are the translations generated during the conversion of the coordinates of the 3D laser radar system into a 2D plane coordinate system.

[0124] S3: The laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam Fusion is performed to obtain a fused two-dimensional image point matrix Y fuse The fused image includes the following steps:

[0125] S31: Construct laser two-dimensional image point matrix Y Lidar The two-dimensional image points and the two-dimensional image point matrix Y Cam A set of two-dimensional image points A obtained by concatenating the two-dimensional image points in ; Two-dimensional image point matrix Y Cam =(u,v,1), where u is the three-spectrum camera The x-axis coordinate of the two-dimensional image vessel measured by the camera, v is the y-axis coordinate of the two-dimensional image vessel measured by the tri-spectral camera;

[0126] S32: Divide the two-dimensional image point set A into N×N grids, and determine the position of the w-th two-dimensional image point in the grids.

[0127] Preferably, N×N is 3×3 or 4×4;

[0128] S33: Construct the following maximum similarity model so that the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam For maximum integration.

[0129]

[0130]

[0131] Where j = 1, 2, 3, 4; M is Col, Tex, Siz or Shape, is the color parameter of the fused image, is the texture parameter of the fused image, is the size parameter of the fused image in pixels, is the image shape compatibility parameter in the fused image; represent The matrix Y of the laser two-dimensional image points in the two-dimensional image point set A obtained by splicing the two-dimensional image points divided into N×N grids is Lidar The pth point of the image composed of represent The two-dimensional image point set A is obtained by splicing the two-dimensional image point matrix Y which is divided into N×N grids. Cam The qth point of .

[0132] The calculation formula of the color parameters of the fused image is as follows:

[0133]

[0134] The calculation formula of the fused image texture parameters is as follows:

[0135]

[0136] The calculation formula of the size parameter of the fused image in pixels is as follows:

[0137]

[0138] The calculation formula of image shape compatibility parameter in the fused image is as follows:

[0139]

[0140] Indicates the two-dimensional image point matrix Y belonging to the laser in the two-dimensional image point set A in the color descriptor Lidar The kth histogram value of the image composed of Represents the two-dimensional image point set A in the color descriptor that belongs to the two-dimensional image point matrix Y Cam The kth histogram value of the composed image; Represents the matrix Y of the laser two-dimensional image points in the two-dimensional image point set A in the texture descriptor Lidar The kth histogram value of the image composed of Represents the two-dimensional image point matrix Y in the two-dimensional image point set A in the texture descriptor Cam The kth histogram value of the image composed of; Siz(·) represents the image size calculation function in pixels, The matrix Y of the laser two-dimensional image points in the two-dimensional image point set A obtained by splicing the two-dimensional image points divided into N×N grids is Lidar The image size of the p-th point of the composed image, The two-dimensional image point set A is obtained by splicing the two-dimensional image point matrix Y which is divided into N×N grids. Cam Siz(A) is the image size of the qth point of the spliced ​​two-dimensional image point set A.

[0141] The obtained fused image is combined with the real two-dimensional image point matrix Y obtained in step S1 real The overlap degree of the formed two-dimensional image is calculated. Suppose the image size of the fused image is S and the real two-dimensional image point matrix Y real The image size of the formed two-dimensional image is H, and the calculation formula of the coincidence degree is as follows:

[0142]

[0143] S4: Determine whether the overlap degree is greater than or equal to a threshold value of 80%. If the overlap degree reaches the threshold value, proceed to the next step, otherwise repeat steps S1-S4;

[0144] S5: The two-dimensional image dot matrix Y obtained by iteratively optimizing the S3 step that meets the overlap threshold using a convolutional neural network fuse , construct the minimum iteration loss model, obtain the optimized two-dimensional image point matrix with the minimum iteration loss, and determine the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam The target frame required for locating and identifying the target in the fused image, and the target within the target frame is the identified target.

[0145] The minimum iterative loss model constructed is as follows:

[0146]

[0147]

[0148] Among them, σ is the weight coefficient of the loss caused by the smoothness function, s is the iterative algebra, smooth(·) is the smoothness function of the image formed by multiple two-dimensional image points in the two-dimensional image point set, is the Euclidean distance between two adjacent w-th two-dimensional image points and the w-1-th two-dimensional image point.

[0149] Example 5

[0150] like Figure 2 As shown, the target recognition system based on multivariate information fusion provided by this embodiment adopts the method provided by any one of embodiments 1-4, including an AIS ship automatic identification system, a laser radar system, a tri-spectral camera and a main control computing module;

[0151] AIS Automatic Identification System, used to collect real-time three-dimensional geographic coordinates of ships;

[0152] a tri-spectral camera to capture real-time 2D images of the vessel;

[0153] LiDAR system, used to collect the real-time laser three-dimensional geographic coordinate matrix of the ship;

[0154] The main control computing module is used to receive the real-time 3D geographic coordinates of the ship to form a real 3D point cloud and convert it into a real 2D image point matrix Y real , receiving the real-time two-dimensional image of the ship, forming a two-dimensional image point set and a two-dimensional image point matrix Y Cam , receiving the real-time laser 3D geographic coordinate matrix X of the vessel Lidar Form a laser 3D point cloud and convert the laser 3D point cloud into a laser 2D image point matrix Y Lidar ;: transform the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam Fusion is performed to obtain a fused two-dimensional image point matrix Y fuse The fused image is then iterated using a convolutional neural network to optimize the two-dimensional image dot matrix Y that meets the overlap threshold. fuse , determine the laser two-dimensional image point matrix Y Lidar With the two-dimensional image point matrix Y Cam The target frame required for locating and identifying the target in the fused image, and the target within the target frame is the identified target.

[0155] Example 6

[0156] The present invention also provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the target recognition method based on multivariate information fusion according to any one of claims 1 to 7 are implemented:

[0157] Example 7

[0158] The present invention also embodies a computer storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target recognition method based on multi-information fusion provided in any one of embodiments 1-4 are implemented.

[0159] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0160] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory including computer program instructions, which can be executed by a processor of an electronic device to perform the above method.

[0161] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0162] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0163] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0164] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0165] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0166] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0167] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0168] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0169] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0170] The above has described various specific embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A target recognition method based on multivariate information fusion, characterized in that: The steps include: S1: Collect the real-time 3D geographic coordinates of the ship, and the real-time real 3D geographic coordinate matrix forms a real 3D point cloud; Converting the real three-dimensional point cloud into a real two-dimensional image point matrix; S2: collecting a real-time two-dimensional image of the vessel to form a two-dimensional image point set, wherein the coordinates of the two-dimensional image points in the set form a two-dimensional image point matrix; simultaneously collecting the real-time laser three-dimensional geographic coordinates of the vessel, wherein the real-time laser three-dimensional geographic coordinate matrix forms a laser three-dimensional point cloud; and converting the laser three-dimensional point cloud into a laser two-dimensional image point matrix; S3: Fusing the laser two-dimensional image dot matrix with the two-dimensional image dot matrix to obtain a fused image having a fused two-dimensional image dot matrix, and calculating the degree of overlap between the obtained fused image and the two-dimensional image formed by the real two-dimensional image dot matrix obtained in step S1; S4: Determine whether the overlap degree is greater than or equal to a threshold value of 80%. If the overlap degree reaches the threshold value, proceed to the next step, otherwise repeat steps S1-S4; S5: using a convolutional neural network to iteratively optimize the two-dimensional image point matrix obtained by fusion in step S3 that meets the overlap threshold, constructing a minimum iteration loss model, obtaining an optimized fused two-dimensional image point matrix with a minimum iteration loss, and determining a target frame required for positioning and identifying a target in an image obtained by fusion of the laser two-dimensional image point matrix and the two-dimensional image point matrix; The real-time real three-dimensional geographic coordinate matrix formed by the real-time real three-dimensional geographic coordinates collected in step S1 is X real =(x real ,y real ,z real ,1) T , where x real is the actual x-axis coordinate of the ship, y real is the actual y-axis coordinate of the ship, z real is the real z-axis coordinate of the ship; The real three-dimensional point cloud is converted into a real two-dimensional image point matrix Y using the following formula: real : Y real =T real X real Among them, T real is the real 3D point cloud transformation matrix, d i is the distance between the i-th point in the converted real two-dimensional image and the real point, (c x ,c y ) is the center point of the image composed of the real two-dimensional image points obtained by transformation, b i is the baseline of the image composed of the real two-dimensional image points obtained by relative conversion; in the step S2, the laser three-dimensional point cloud formed by the real-time laser three-dimensional geographic coordinates is converted into the laser two-dimensional image point matrix Y Lidar The formula is as follows: Y Lidar =T Lidar RS Lidar X Lidar The real-time laser 3D geographic coordinate matrix X Lidar =(x l ,y l ,z l ,1) T , x l is the x-axis coordinate of the ship measured by the lidar, l is the y-axis coordinate of the ship measured by the lidar, z l is the z-axis coordinate of the ship measured by the laser radar; R is the laser 3D point cloud rotation matrix, which is an orthogonal matrix with a determinant of 1; T Lidar is the laser 3D point cloud transformation matrix, d j is the distance between the jth point in the converted laser 2D image and the real point, (m x ,m y ) is the center point of the image composed of the converted laser two-dimensional image points, b j is the baseline of the image composed of the laser two-dimensional image points obtained by relative conversion; S Liadar is the transformation matrix, R Lidar is an orthogonal matrix with determinant 1; t Lidar is the translation vector matrix generated during the transformation process, and They are the translations generated during the conversion of the coordinates of the 3D laser radar system into the 2D plane coordinate system; The method for fusing the laser two-dimensional image dot matrix with the two-dimensional image dot matrix in step S3 comprises the following steps: S31: constructing a two-dimensional image point set A obtained by splicing two-dimensional image points in the laser two-dimensional image point matrix and two-dimensional image points in the two-dimensional image point matrix; S32: Divide the two-dimensional image point set A into N×N grids, and determine the position of the w-th two-dimensional image point in the grids. S33: constructing a maximum similarity model to maximize the fusion of the laser two-dimensional image point matrix and the two-dimensional image point matrix; The maximum similarity model constructed in step S33 is as follows: Where j = 1, 2, 3, 4; M is Col, Tex, Siz or Shape, is the color parameter of the fused image, is the texture parameter of the fused image, is the size parameter of the fused image in pixels, is the image shape compatibility parameter in the fused image; represent The matrix Y of the laser two-dimensional image points in the two-dimensional image point set A obtained by splicing the two-dimensional image points divided into N×N grids is Lidar The pth point of the image composed of represent The two-dimensional image point set A is obtained by splicing the two-dimensional image point matrix Y which is divided into N×N grids. Cam The qth point of .

2. The target recognition method based on multivariate information fusion according to claim 1 is characterized in that: The calculation formula of the color parameters of the fused image is as follows: The calculation formula of the texture parameters of the fused image is as follows: The calculation formula of the size parameter of the fused image in pixels is as follows: The calculation formula of the image shape compatibility parameter in the fused image is as follows: Indicates the two-dimensional image point matrix Y belonging to the laser in the two-dimensional image point set A in the color descriptor Lidar The kth histogram value of the image composed of Represents the two-dimensional image point set A in the color descriptor that belongs to the two-dimensional image point matrix Y Cam The kth histogram value of the composed image; Represents the matrix Y of the laser two-dimensional image points in the two-dimensional image point set A in the texture descriptor Lidar The kth histogram value of the image composed of, Represents the two-dimensional image point matrix Y in the two-dimensional image point set A in the texture descriptor Cam The kth histogram value of the image composed of; Siz(·) represents the image size calculation function in pixels.

3. A target recognition system based on multivariate information fusion using the method described in any one of claims 1-2, characterized in that: Includes AIS ship automatic identification system, laser radar system, tri-spectral camera and main control computing module; The AIS automatic ship identification system is used to collect the real-time three-dimensional geographic coordinates of the ship; The tri-spectral camera is used to collect real-time two-dimensional images of the vessel; The laser radar system is used to collect the real-time laser three-dimensional geographic coordinate matrix of the ship; The main control calculation module is used to receive the real-time three-dimensional geographic coordinates of the ship to form a real three-dimensional point cloud, and convert it into a real two-dimensional image point matrix, receive the real-time two-dimensional image of the ship to form a two-dimensional image point set and a two-dimensional image point matrix, receive the real-time laser three-dimensional geographic coordinate matrix of the ship to form a laser three-dimensional point cloud, and convert the laser three-dimensional point cloud into a laser two-dimensional image point matrix; fuse the laser two-dimensional image point matrix with the two-dimensional image point matrix to obtain a fused image with a fused two-dimensional image point matrix, and use a convolutional neural network to iteratively optimize the two-dimensional image dot matrix that meets the overlap threshold, and determine the target frame required for positioning and identifying the target in the image after the laser two-dimensional image point matrix and the two-dimensional image point matrix are fused.

4. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the target recognition method based on multi-information fusion as described in any one of claims 1-2 are implemented.

5. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target recognition method based on multi-information fusion as described in any one of claims 1-2 are implemented.

Images