A ship navigation state estimation method, device, equipment and storage medium

Abstract

The application discloses a ship navigation state estimation method and device, equipment and a storage medium, and relates to the field of image processing. The method comprises the following steps: collecting an original image by using a camera sensor, determining parameters of the camera sensor, and constructing a sensor coordinate system and a world coordinate system; determining rigid body transformation parameters based on the parameters of the camera sensor, and determining a homography transformation matrix by using the rigid body transformation parameters, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system by using the homography transformation matrix; pre-processing the orthographic projection image, and determining tail track area frequency domain information and water surface dispersion characteristics of the pre-processed image; and determining a ship navigation state parameter in the original image based on the tail track image frequency domain information and the water surface dispersion characteristics. In this way, the tail track image frequency domain information and the water surface dispersion characteristics of the original image can be analyzed to obtain the ship navigation state parameter.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a ship navigation state estimation method, device, equipment and storage medium. BACKGROUND

[0002] With the gradual development of visual image technology, image processing technology is widely used in various fields, but the existing visual image related technology is mainly applied to land environment, the main reason is that there is lack of suitable reference in marine environment. At the same time, the quality of sea surface optical image is easily affected by water quality and background lighting conditions, the image brightness is uneven, it is difficult to locate and analyze the ship or ship wake features on the sea surface.

[0003] And, the existing ship wake detection technology based on visible light image or infrared image is based on orthographic satellite image, but the light wave band imaging is easy to be disturbed by climate factors such as cloud layer, and it is not easy to obtain high resolution, large scale low orbit satellite image. Although the synthetic aperture radar of microwave band can obtain high resolution water surface wake image, the imaging process will be disturbed by sea wave movement and inherent coherent speckle noise of imaging system, the imaging quality is low, and special starboard or airborne radar imaging system is needed, which is difficult to popularize. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a ship navigation state estimation method, device, equipment and storage medium, which can analyze the wake image frequency domain information and water surface dispersion characteristics in the image to estimate the heading and speed of the ship in the image through any single oblique image. The specific scheme is as follows:

[0005] In the first aspect, the present application discloses a ship navigation state estimation method, comprising:

[0006] Using a camera sensor to collect images of a ship currently in navigation state to obtain an original image, determining the parameters of the camera sensor, and constructing a sensor coordinate system and a world coordinate system;

[0007] Determining the rigid transformation parameters based on the parameters of the camera sensor, and determining the homography transformation matrix using the rigid transformation parameters, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system using the homography transformation matrix;

[0008] Preprocessing the orthographic projection image, and determining the wake area frequency domain information and water surface dispersion characteristics of the preprocessed image;

[0009] Determining the navigation state parameters of the ship in the original image based on the wake image frequency domain information and the water surface dispersion characteristics.

[0010] Optionally, the image acquisition of the ship currently in the sailing state by the camera sensor to obtain the original image, the determination of the parameters of the camera sensor, and the construction of the sensor coordinate system and the world coordinate system, comprising:

[0011] Acquiring the original image by the camera sensor, and reading the camera attitude angle and the shooting height from the water surface when the camera sensor acquires the original image; the camera attitude angle includes the heading angle, the lateral angle, and the photograph rotation angle;

[0012] Setting the position of the camera sensor when acquiring the original image as the coordinate origin of the sensor coordinate system, and creating the sensor coordinate system based on the coordinate origin of the sensor coordinate system;

[0013] Determining the point on the sea level mapped by the camera sensor as the coordinate origin of the world coordinate system, and creating the world coordinate system based on the coordinate origin of the world coordinate system.

[0014] Optionally, the determination of the rigid body transformation parameter based on the parameters of the camera sensor, and the determination of the homography transformation matrix by the rigid body transformation parameter, comprising:

[0015] Determining the rigid body transformation parameter based on the camera attitude angle and the shooting height; the rigid body transformation parameter includes the rotation matrix and the translation vector;

[0016] Determining the mapping matrix from the world coordinate system to the sensor coordinate system based on the rotation matrix and the translation vector;

[0017] Taking the inverse matrix of the mapping matrix, and taking the inverse matrix of the mapping matrix as the homography transformation matrix.

[0018] Optionally, the conversion of the original image in the sensor coordinate system into the orthographic projection image based on the world coordinate system by the homography transformation matrix, comprising:

[0019] Determining the vertex coordinates of the original image in the sensor coordinate system, and determining the converted coordinates of the original image in the world coordinate system based on the vertex coordinates and the homography transformation matrix;

[0020] Determining the center coordinates of the original image in the world coordinate system by the converted coordinates;

[0021] Discretizing and normalizing the converted coordinates based on the converted coordinates, the center coordinates, and the preset sampling interval to obtain the normalized coordinates corresponding to the converted coordinates;

[0022] determine a homographic transformation matrix of the orthographic image based on the center coordinate, the preset sampling interval and the single-variant transformation matrix, and determine pixel coordinates in the original image;

[0023] process the pixel coordinates in the original image based on the homographic transformation matrix of the orthographic image and the normalized coordinates, so as to convert the original image into an orthographic projection image based on the world coordinate system.

[0024] Optionally, the preprocessing of the orthographic projection image comprises:

[0025] rotating the orthographic projection image based on a preset rotation range to obtain a rotated image; the preset rotation range is a range ensuring that the ship heading in the image is between negative ninety degrees and positive ninety degrees;

[0026] determining a wake region of the rotated image, averaging the colors of non-wake regions in the rotated image, and covering the non-wake regions in the rotated image with the obtained color average value to obtain a preprocessed image.

[0027] Optionally, the determining of the wake region frequency domain information of the preprocessed image comprises:

[0028] performing two-dimensional fast Fourier transform on the preprocessed image to obtain a frequency spectrum image of the wake region in the preprocessed image;

[0029] determining a spatial wave number corresponding to the wake region based on the frequency spectrum image of the wake region.

[0030] Optionally, the determining of the navigation state parameter of the ship in the original image based on the wake image frequency domain information and the water surface dispersion characteristics comprises:

[0031] determining a trajectory line of the wake region in the frequency spectrum image based on the spatial wave number and the water surface dispersion characteristics, so as to determine the heading and speed of the ship in the original image based on the trajectory line.

[0032] In a second aspect, the present application discloses a ship navigation state estimation device, comprising:

[0033] a coordinate system construction module, configured to collect an original image by using a camera sensor to collect an image of a ship currently in a navigation state, determine parameters of the camera sensor, and construct a sensor coordinate system and a world coordinate system;

[0034] an image conversion module configured to determine a rigid transformation parameter based on the parameters of the camera sensor, and determine a homographic transformation matrix using the rigid transformation parameter, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system using the homographic transformation matrix;

[0035] an image preprocessing module configured to preprocess the orthographic projection image, and determine tail region frequency domain information and water surface dispersion characteristics of the preprocessed image;

[0036] a ship navigation state estimation module configured to determine a navigation state parameter of the ship in the original image based on the tail image frequency domain information and the water surface dispersion characteristics.

[0037] In a third aspect, the present application discloses an electronic device, comprising:

[0038] a memory configured to save a computer program;

[0039] a processor configured to execute the computer program to implement the ship navigation state estimation method as described above.

[0040] In a fourth aspect, the present application discloses a computer readable storage medium configured to save a computer program, which is executed by a processor to implement the ship navigation state estimation method as described above.

[0041] In the present application, first, a camera sensor is used to collect images of a ship currently in a navigation state to obtain an original image, parameters of the camera sensor are determined, and a sensor coordinate system and a world coordinate system are constructed; then a rigid transformation parameter is determined based on the parameters of the camera sensor, and a homographic transformation matrix is determined using the rigid transformation parameter, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system using the homographic transformation matrix; the orthographic projection image is preprocessed, and tail region frequency domain information and water surface dispersion characteristics of the preprocessed image are determined; finally, a navigation state parameter of the ship in the original image is determined based on the tail image frequency domain information and the water surface dispersion characteristics. As can be seen, by using the ship navigation state estimation method in the present application, after an original image containing a ship is obtained, the original image collected can be converted from a sensor coordinate system constructed based on a camera sensor to a world coordinate system constructed based on a sea level to obtain an orthographic projection image, and the navigation state parameter of the ship in the original image can be determined by analyzing tail image frequency domain information and water surface dispersion characteristics in the orthographic projection image. In this way, a single image can be collected in combination with sensor parameters to convert and analyze the image, so as to complete the estimation of the navigation state of the ship in an actual water surface scene. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced. Obviously, the accompanying drawings in the following description only only the embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0043] Figure 1 A flow chart of a ship navigation state estimation method provided by the present application;

[0044] Figure 2 A flow chart of a specific ship navigation state estimation method provided by the present application;

[0045] Figure 3 A schematic diagram of a sensor coordinate system and a world coordinate system provided by the present application;

[0046] Figure 4 A flow chart of a specific ship navigation state estimation method provided by the present application;

[0047] Figure 5 A raw image collected by a camera sensor provided by the present application;

[0048] Figure 6 An orthographic projection image converted from a raw image provided by the present application;

[0049] Figure 7 A flow chart of a specific ship navigation state estimation method provided by the present application;

[0050] Figure 8 A conversion and color extraction schematic diagram of an orthographic projection image provided by the present application;

[0051] Figure 9 A wake area extraction schematic diagram provided by the present application;

[0052] Figure 10 A color overlay schematic diagram provided by the present application;

[0053] Figure 11 A frequency spectrum schematic diagram provided by the present application;

[0054] Figure 12 A device structure schematic diagram of a ship navigation state estimation method provided by the present application;

[0055] Figure 13 An electronic device structure diagram provided by the present application. DETAILED DESCRIPTION

[0056] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all the other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of the present application.

[0057] Existing ship detection technologies based on visible light images or infrared images are all based on orthographic satellite images, but light band imaging is easily disturbed by weather factors such as clouds, and it is not easy to obtain high-resolution, large-scale low-orbit satellite images. Although the synthetic aperture radar in the microwave band can obtain high-resolution water surface wake images, the imaging process is disturbed by sea wave motion and inherent coherent speckle noise of the imaging system, the imaging quality is low, and a special spaceborne or airborne radar imaging system is needed, which is difficult to popularize.

[0058] In order to overcome the above technical problems, the present application provides a ship navigation state estimation method, device, equipment and storage medium, which can analyze the wake image frequency domain information and water surface dispersion characteristics in the image to estimate the heading and speed of the ship in the image through any single oblique image.

[0059] Referring to Figure 1 The embodiment of the present application discloses a ship navigation state estimation method, which comprises:

[0060] Step S11, using a camera sensor to collect images of a ship in a navigation state to obtain an original image, determining parameters of the camera sensor, and constructing a sensor coordinate system and a world coordinate system.

[0061] In this embodiment, the image of the ship in the navigation state on the water surface is first collected by the camera sensor, and a single clear image is collected during image collection. After the image collection is completed, the parameters of the camera sensor during image collection, such as the camera pose angle and the shooting height from the water surface, are determined, so as to determine the rigid transformation parameters based on the parameters. The position of the camera sensor during image collection is determined, and the position of the camera sensor during image collection is taken as the coordinate origin, so as to construct the sensor coordinate system based on the coordinate origin. After the position of the camera sensor during image collection is determined, the position of the camera sensor during image collection is mapped to the horizontal plane, and the point mapped to the horizontal plane is taken as the coordinate origin to construct the world coordinate system. It needs to be further explained that the sensor coordinate system is the coordinate system of the collected image when the camera sensor collects the image, and the world coordinate system is the coordinate system corresponding to the actual direction and the horizontal plane in the actual world.

[0062] Step S12: Determine rigid body transformation parameters based on the parameters of the camera sensor, and use the rigid body transformation parameters to determine the homography transformation matrix, so as to use the homography transformation matrix to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system.

[0063] In this embodiment, it is necessary to utilize the determined camera sensor parameters, including the heading angle φ, side tilt angle ω, image rotation angle κ, and the camera's height h above the image. c To determine the rigid body transformation parameters, and the rigid body transformation parameters include the rotation matrix R. cw Translation vector t cw After obtaining the rotation matrix R cw Translation vector t cw Next, the mapping matrix H from the world coordinate system to the sensor coordinate system needs to be determined based on the rotation matrix and the translation vector. cw After obtaining the mapping matrix, the mapping matrix is ​​inverted to obtain the homography transformation matrix H that maps from the sensor coordinate system to the world coordinate system. wc .

[0064] After obtaining the homography transformation matrix, the original image acquired by the camera sensor needs to be transformed using the determined homography transformation matrix to obtain an orthophoto image corresponding to the original image in the sensor coordinate system. Further explanation is needed: when transforming the original image using the homography transformation matrix, the coordinates of the four vertices of the original image in the sensor coordinate system need to be determined first, and then converted to vertex coordinates in the world coordinate system to determine the transformation range of pixel coordinates in the image. After obtaining the converted vertex coordinates, the center coordinates need to be determined using the converted vertex coordinates. Then, the orthophoto homography transformation matrix is ​​determined using the center coordinates, a preset sampling interval, and the homography transformation matrix, so that the original image can be processed using the orthophoto homography transformation matrix to obtain an orthophoto image based on the world coordinate system.

[0065] Step S13 involves preprocessing the orthophoto image and determining the frequency domain information of the wake region and the dispersion characteristics of the water surface in the preprocessed image.

[0066] In the embodiment, after the orthographic projection image is obtained, the orthographic projection image needs to be preprocessed so that the preprocessed image is more conducive to two-dimensional fast Fourier transform operation. It needs to be noted that the preprocessing operation includes rotating the orthographic projection image, performing tail region extraction operation on the rotated image, extracting the color of the image of the region other than the tail region, calculating the color average value, and covering the image of the region other than the tail region with the obtained color average value to reduce the error caused by image mutation and uneven image illumination. After the preprocessed image is obtained, two-dimensional fast Fourier transform operation is performed on the preprocessed image to obtain the tail region frequency domain information and the water surface dispersion characteristics of the preprocessed image.

[0067] In step S14, the sailing state parameter of the ship in the original image is determined based on the tail image frequency domain information and the water surface dispersion characteristics.

[0068] In the embodiment, the trajectory line equation can be constructed based on the tail image frequency domain information and the water surface dispersion characteristics, and the trajectory line is determined based on the trajectory line equation. In this way, when it is necessary to estimate the sailing state of the ship, the heading and speed of the ship can be determined by taking the coordinate point on the trajectory line and combining the trajectory equation.

[0069] As can be seen, in the embodiment, first, the camera sensor is used to collect the image of the ship currently in the sailing state to obtain the original image, the parameters of the camera sensor are determined, and the sensor coordinate system and the world coordinate system are constructed. Then, the rigid transformation parameters are determined based on the parameters of the camera sensor, and the homographic transformation matrix is determined based on the rigid transformation parameters, so that the original image in the sensor coordinate system is converted into the orthographic projection image based on the world coordinate system by using the homographic transformation matrix. The orthographic projection image is preprocessed, and the tail region frequency domain information and the water surface dispersion characteristics of the preprocessed image are determined. Finally, the sailing state parameter of the ship in the original image is determined based on the tail image frequency domain information and the water surface dispersion characteristics. As can be seen, by using the ship sailing state estimation method in the application, after the original image containing the ship is obtained, the original image collected can be converted from the sensor coordinate system constructed based on the camera sensor to the world coordinate system constructed based on the sea level to obtain the orthographic projection image, and the sailing state parameter of the ship in the original image can be determined by analyzing the tail image frequency domain information and the water surface dispersion characteristics in the orthographic projection image. In this way, a single image can be collected in combination with the sensor parameters to convert and analyze the image to complete the estimation of the sailing state of the ship in the actual water surface scene.

[0070] Based on the foregoing embodiments, it is necessary to construct a sensor coordinate system and a world coordinate system in the application, and it is necessary to convert the collected original image based on the sensor coordinate system to the world coordinate system through a determined homographic transformation matrix. For this purpose, the embodiment makes a detailed description of how to construct the sensor coordinate system and the world coordinate system and how to determine the homographic transformation matrix, see Figure 2 The embodiment of the application discloses a ship navigation state estimation method, comprising:

[0071] Step S21, an original image is collected by using a camera sensor, and a camera attitude angle and a shooting height from the water surface when the original image is collected by the camera sensor are read; the camera attitude angle comprises a heading angle, a lateral angle and a photographing angle.

[0072] In the embodiment, the oblique jet wake image I cam of the ship sailing on the water surface can be collected by using the camera sensor, that is, the original image containing the ship is collected, and the camera sensor parameters when the oblique jet wake image I cam is shot are read, the camera sensor parameters specifically comprising three angle elements of the camera attitude angle, the heading angle φ, the lateral angle ω and the photographing angle κ, and the height h c from the water surface of the camera.

[0073] Step S22, the position when the original image is collected by the camera sensor is set as the coordinate origin of the sensor coordinate system, and the sensor coordinate system is created based on the coordinate origin of the sensor coordinate system.

[0074] In the embodiment, as Figure 3 shown, the position when the original image is collected by the camera sensor needs to be determined, and the optical center position when the camera is at the position for collecting the original image is set as the coordinate origin O c of the sensor coordinate system, and after the coordinate origin of the sensor coordinate system is determined, the coordinate origin O of the world coordinate system also needs to be determined, and the coordinate origin O of the world coordinate system is the point along the directly downward mapping to the water surface. c

[0075] Step S23, the point on the sea level mapped by the camera sensor is determined as the coordinate origin of the world coordinate system, and the world coordinate system is created based on the coordinate origin of the world coordinate system.

[0076] In the embodiment, as Figure 3 shown, the sensor coordinate system needs to be established based on the coordinate origin O c of the sensor coordinate system, and the Z c axis of the sensor coordinate system points to the outside of the camera along the optical axis direction, the X c axis and the Y​c The axis is based on the origin O. c Starting from point I, the direction is the same as the original I. cam The corresponding X-axis and Y-axis directions in the imaging plane. After establishing the sensor coordinates, it is also necessary to establish a world coordinate system based on the origin O of the world coordinate system, where the x-axis points east, the y-axis points north, and the z-axis points directly upward along the vertical direction.

[0077] Step S24: Determine the rigid body transformation parameters based on the camera attitude angle and the shooting height; the rigid body transformation parameters include a rotation matrix and a translation vector.

[0078] In this embodiment, it is necessary to consider the camera attitude angles from the camera sensor: the heading angle φ, the side tilt angle ω, the image rotation angle κ, and the camera's height h above the image. c Determine the rotation matrix R in the rigid body transformation parameters cw Translation vector t cw Wherein, the rotation matrix R cw The rotation matrix R can be determined using the heading angle φ, side tilt angle ω, and image rotation angle κ. Specifically, it can be constructed using trigonometric functions of the above three angles to create a matrix pair. cw The representation is determined, and the rotation matrix R is determined. cw The expression is as follows:

[0079]

[0080] And in determining the rotation matrix R cw After the expression, it can be obtained through the rotation matrix R. cw Distance from camera to height of the image h c Construct translation vector t cw The expression, and the translation vector t cw The expression is as follows:

[0081] t cw =-R cw [0 0 H c ] T

[0082] Step S25: Determine the mapping matrix from the world coordinate system to the sensor coordinate system based on the rotation matrix and the translation vector.

[0083] In this embodiment, after determining the rotation matrix R cw The expression for and translation vector t cw After obtaining the expression, the mapping matrix H between the world coordinate system based on the horizontal plane and the sensor coordinate system based on the image plane needs to be calculated using the rotation matrix and translation vector.cw That is, the two-dimensional homography transformation matrix from the world coordinate system to the sensor coordinate system where the oblique projection image is located, and the mapping matrix H cw The expression is as follows:

[0084]

[0085] Where K is the intrinsic parameter matrix of the camera sensor.

[0086] Step S26: Take the inverse of the mapping matrix and use the inverse of the mapping matrix as the homography transformation matrix.

[0087] In this embodiment, the mapping matrix H between the world coordinate system based on the horizontal plane and the sensor coordinate system based on the image plane is obtained. cw Next, the mapping matrix needs to be inverted to obtain the homography transformation matrix H, which transforms the sensor coordinate system of the obliquely projected image to the world coordinate system. wc And the homography transformation matrix H wc The expression is as follows:

[0088] H wc =(H cw ) -1

[0089] Step S27: Use the homography transformation matrix to convert the original image in the sensor coordinate system into an orthophoto image based on the world coordinate system.

[0090] Step S28: Preprocess the orthophoto image and determine the frequency domain information of the wake region and the dispersion characteristics of the water surface of the preprocessed image.

[0091] Step S29: Determine the ship's navigation state parameters in the original image based on the frequency domain information of the wake image and the water surface dispersion features.

[0092] It should be noted that for a more detailed description of steps S27, S28, and S29 in this embodiment, please refer to the foregoing embodiments, and will not be repeated here.

[0093] Therefore, this embodiment, after constructing the sensor coordinate system and the world coordinate system, determines the rigid body transformation parameters through the camera sensor parameters, and uses these rigid body transformation parameters to determine the homography transformation matrix. Furthermore, in subsequent processing, the original image acquired by the camera sensor can be processed using the determined homography transformation matrix. In this way, by constructing two coordinate systems and determining the homography transformation matrix, an effective transformation tool can be provided for subsequently transforming the original image from the sensor coordinate system to the world coordinate system, improving the efficiency of the ship navigation state estimation method described in this application.

[0094] Based on the foregoing embodiments, after the homographic transformation matrix is determined, the original image needs to be processed by using the homographic transformation matrix to convert the original image from the sensor coordinate system to the world coordinate system. Therefore, this embodiment describes in detail how to convert the original image by using the homographic transformation matrix. See Figure 4 The ship navigation state estimation method disclosed by the embodiment of the application includes:

[0095] In step S31, the camera sensor is used to collect an image of a ship currently in a navigation state to obtain an original image, parameters of the camera sensor are determined, and a sensor coordinate system and a world coordinate system are constructed.

[0096] In step S32, rigid transformation parameters are determined based on the parameters of the camera sensor, and a homographic transformation matrix is determined by using the rigid transformation parameters.

[0097] In step S33, vertex coordinates of the original image in the sensor coordinate system are determined, and converted coordinates of the original image in the world coordinate system are determined based on the vertex coordinates and the homographic transformation matrix.

[0098] In this embodiment, the original image I cam The four vertex coordinates of the image in the sensor coordinate system include a left upper vertex coordinate a right upper vertex coordinate a left lower vertex coordinate and a right lower vertex coordinate The vertex coordinates of the original image need to be converted by using the determined homographic transformation matrix to obtain the mapping of the vertex coordinates of the original image in the world coordinate system, that is, the converted coordinates. The left upper vertex coordinate corresponds to the converted coordinate in the world coordinate system The right upper vertex coordinate corresponds to the converted coordinate in the world coordinate system The left lower vertex coordinate corresponds to the converted coordinate in the world coordinate system The right lower vertex coordinate corresponds to the converted coordinate in the world coordinate system And the conversion formula is as follows:

[0099]

[0100]

[0101]

[0102]

[0103] Step S34, the center coordinates of the original image in the world coordinate system are determined through the converted coordinates.

[0104] In this embodiment, the two-dimensional coordinates M=(M x , y ) corresponding to the center point of the converted image can be determined through the converted coordinates, wherein

[0105] M x =(x max +x min ) / 2

[0106] M y =(y max +y min ) / 2

[0107] x max , x min represent the maximum and minimum values of the x coordinates of the four vertices , respectively, and y max , y min represent the maximum and minimum values of the y coordinates of the four vertices , respectively.

[0108] Step S35, the converted coordinates are discretized and normalized based on the converted coordinates, the center coordinates, and a preset sampling interval, so as to obtain normalized coordinates corresponding to the converted coordinates.

[0109] In this embodiment, the northeast coordinates of the four vertices need to be discretized and normalized again, so as to be converted into pixel coordinates in the corrected image, and the sampling interval between adjacent pixels in the corrected new image is defined as δd. It needs to be noted that the sampling interval can be set by the user as required. The normalization formula is as follows:

[0110]

[0111] wherein P represents the four vertices of the image obtained in the previous step, and p represents the point of P after discretization and normalization in the corrected image. In this way, the positions of the vertices of the original image after conversion can be determined through the converted coordinates, and the pixels of the converted image are limited to be within the range formed by the converted coordinates.

[0112] Step S36, the homographic transformation matrix of the orthographic image is determined by using the center coordinates, the preset sampling interval, and the homographic transformation matrix, and the pixel coordinates in the original image are determined.

[0113] In this embodiment, the two-dimensional coordinates M = (M_center) corresponding to the center point of the transformed image can be used. x M y ), preset sampling interval δd, homography transformation matrix H wc To determine the homography transformation matrix of the orthophoto image, and also to determine the original image I... cam pixel coordinates p cam In order to utilize the orthophoto homography transformation matrix H ortho-cam The pixel coordinates in the original image are processed. Furthermore, the orthophoto homography transformation matrix H... ortho-cam The expression is as follows:

[0114]

[0115] Step S37: Process the pixel coordinates in the original image using the orthophoto homography transformation matrix and the normalized coordinates, so as to convert the original image into an orthophoto projection image based on the world coordinate system.

[0116] In this embodiment, the orthophoto homography transformation matrix H can be determined. ortho-cam For pixel p in the original image cam The coordinates are processed to convert the pixel coordinates in the original image into the corrected image I. ortho pixel coordinates p ortho This allows for the transformation from the original image in the sensor coordinate system to an orthographically projected image in the world coordinate system. And as... Figure 5 The image shown is the original image I. cam After conversion, we can obtain the following: Figure 6 The I shown ortho .

[0117] It should be noted that for a more detailed description of steps S31 and S32 in this embodiment, please refer to the foregoing embodiments, and will not be repeated here.

[0118] It can be seen that, in the embodiment, the vertex coordinates of the original image in the sensor coordinate system are determined, and the converted coordinates of the original image in the world coordinate system are determined based on the vertex coordinates and the homographic transformation matrix. Then, the center coordinates of the original image in the world coordinate system are determined based on the converted coordinates. The converted coordinates are discretized and normalized based on the converted coordinates, the center coordinates, and the preset sampling interval, to obtain normalized coordinates corresponding to the converted coordinates. After the normalized coordinates are obtained, the orthographic image homographic transformation matrix is determined based on the center coordinates, the preset sampling interval, and the homographic transformation matrix. The pixel coordinates in the original image are determined. Finally, the pixel coordinates in the original image are processed based on the orthographic image homographic transformation matrix and the normalized coordinates, so as to convert the original image into an orthographic projection image based on the world coordinate system. In this way, the original image can be converted from the sensor coordinate system to the world coordinate system, which provides a basis for subsequent analysis of the converted orthographic projection image, and makes the ship navigation state estimation method more accurate.

[0119] Based on the foregoing embodiment, after the original image is converted to obtain the converted orthographic projection image, the orthographic projection image needs to be preprocessed, and the preprocessed image needs to be analyzed, so as to analyze the navigation state of the ship in the image. For this purpose, the embodiment details how to preprocess the ship and how to estimate the state of the ship. Referring to Figure 7 The embodiment of the application discloses a ship navigation state estimation method, which comprises the following steps:

[0120] In step S41, an original image is collected by using a camera sensor, and a camera posture angle and a shooting height from water when the original image is collected by the camera sensor are read.

[0121] In step S42, a position when the original image is collected by the camera sensor is set as a coordinate origin of a sensor coordinate system, and the sensor coordinate system is created based on the coordinate origin of the sensor coordinate system.

[0122] In step S43, the orthographic projection image is rotated based on a preset rotation range, to obtain a rotated image. The preset rotation range is a range in which a ship heading in the image is ensured to be negative ninety degrees to positive ninety degrees.

[0123] In the embodiment, the orthographic projection image is rotated based on a preset rotation range, to obtain a rotated image. That is, the obtained orthographic projection image as shown in Figure 6 is rotated to obtain an orthographic projection image as shown in Figure 8The rotated image is shown, so that the heading is towards the right side of the image, ensuring that the heading of the ship in the image is between -90° and 90°, which facilitates the arctan calculation in the subsequent steps and improves the efficiency of the ship navigation state estimation method described in the present application.

[0124] Step S44, determining the wake region of the rotated image, averaging the colors of the non-wake region of the rotated image, and covering the non-wake region of the rotated image with the obtained color average to obtain a pre-processed image.

[0125] In this embodiment, as shown in Figure 9 , it is necessary to determine the wake region of the ship in the rotated image, that is, to extract the Kelvin wake region of the ship position in the rotated image, the Kelvin wake region being a region of 39° around the ship position, and after determining the wake region, as shown in Figure 8 , it is necessary to extract the color around the wake region and average the color, and fill the non-wake region with the obtained color average to obtain a pre-processed image I Figure 10 as shown in rec .

[0126] Step S45, performing two-dimensional fast Fourier transform on the pre-processed image to obtain a frequency spectrum image of the wake region in the pre-processed image.

[0127] In this embodiment, it is necessary to perform two-dimensional fast Fourier transform on the pre-processed image to obtain the frequency spectrum S(k x ,k y ) of the image as shown in Figure 11 , where k x ,k y represent the spatial wave numbers in the x and y directions, respectively. The arc line with the "X" shape in the figure corresponds to the ship wake, the central bright region corresponds to the low frequency noise of the image, and the straight line ghost in the diagonal direction is the frequency spectrum interference noise caused by the image mutation.

[0128] Step S46, determining the spatial wave number corresponding to the wake region based on the frequency spectrum image of the wake region.

[0129] Step S47, determining the trajectory line of the wake region in the frequency spectrum image based on the spatial wave number and the water surface dispersion characteristics, so as to determine the heading and speed of the ship in the original image by using the trajectory line.

[0130] In this embodiment, the expression of the trajectory line L of the wake region in the frequency spectrum image can be determined based on the pre-set water surface dispersion characteristics and spatial wave numbers k x ,k y , and the expression of the trajectory line L is as follows:

[0131] L: [U s (k x cosθ+k y sinθ)] 2 =gk

[0132] where U s is the ship speed, g is the gravity acceleration, is the modulus of wave number, is the ship's yaw angle, and the x-axis direction is defined as 0°, and the counterclockwise direction is the positive direction.

[0133] When the ship speed needs to be estimated, any N points on the trajectory line L in the wake spectrum are needed, n = 1, 2, 3,..., N, and k n = (k xn , k yn ). The ship speed formula is calculated as follows:

[0134]

[0135] When the ship travels at a certain speed U s , the wake direction is determined as follows: is a constant, two points k1 = (k x1 , k y1 ) and k2 = (k x2 , k y2 ) on the trajectory line in the wake spectrum are selected, and k1 and k2 are not symmetric about the 0 point, and the wake direction can be solved:

[0136]

[0137] As can be seen, in the embodiment, after the orthographic projection image is preprocessed, the preprocessed image is subjected to two-dimensional fast Fourier transform to determine the spatial wave number of the wake region, and the trajectory line of the wake region in the spectrum image is determined based on the spatial wave number and the water surface dispersion characteristics, so as to determine the heading and speed of the ship in the original image by using the trajectory line. In this way, more accurate heading and speed information can be obtained by using higher precision original data.

[0138] Referring to Figure 12 , the embodiment of the present application discloses a ship navigation state estimation device, which comprises:

[0139] A coordinate system construction module 11 is configured to acquire an original image by using a camera sensor to collect an image of a ship currently in a navigation state, determine parameters of the camera sensor, and construct a sensor coordinate system and a world coordinate system.

[0140] The image conversion module 12 is configured to determine a rigid transformation parameter based on the parameters of the camera sensor, and determine a homographic transformation matrix based on the rigid transformation parameter, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system by using the homographic transformation matrix.

[0141] The image preprocessing module 13 is configured to pre-process the orthographic projection image, and determine the wake region frequency domain information and the water surface dispersion feature of the pre-processed image.

[0142] The ship navigation state estimation module 14 is configured to determine the navigation state parameter of the ship in the original image based on the wake image frequency domain information and the water surface dispersion feature.

[0143] As can be seen, in the embodiment, firstly, the camera sensor is used to collect the image of the ship in the navigation state to obtain the original image, the parameters of the camera sensor are determined, and the sensor coordinate system and the world coordinate system are constructed; then, the rigid transformation parameter is determined based on the parameters of the camera sensor, and the homographic transformation matrix is determined based on the rigid transformation parameter, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system by using the homographic transformation matrix; the orthographic projection image is pre-processed, and the wake region frequency domain information and the water surface dispersion feature of the pre-processed image are determined; finally, the navigation state parameter of the ship in the original image is determined based on the wake image frequency domain information and the water surface dispersion feature. As can be seen, by using the ship navigation state estimation method, the original image containing the ship can be converted from the sensor coordinate system constructed based on the camera sensor to the world coordinate system constructed based on the sea level to obtain the orthographic projection image, and the navigation state parameter of the ship in the original image can be determined by analyzing the wake image frequency domain information and the water surface dispersion feature in the orthographic projection image. In this way, the ship navigation state in the actual water surface scene can be estimated by collecting a single image and jointly analyzing the sensor parameters.

[0144] In some embodiments, the coordinate system construction module 11 can specifically include:

[0145] The first parameter determination unit is configured to collect the original image by using the camera sensor, and read the camera posture angle and the shooting height from the water surface when the camera sensor collects the original image; the camera posture angle includes the heading angle, the lateral angle, and the photographing angle.

[0146] The sensor coordinate system creating unit sets a position of the camera sensor when the original image is captured as an origin of a sensor coordinate system, and creates the sensor coordinate system based on the origin of the sensor coordinate system.

[0147] The world coordinate system creating unit determines a point on a sea level to which the camera sensor is mapped as an origin of a world coordinate system, and creates the world coordinate system based on the origin of the world coordinate system.

[0148] In some embodiments, the image conversion module 12 can specifically include:

[0149] The second parameter determining unit determines a rigid body transformation parameter based on the camera attitude angle and the shooting height; the rigid body transformation parameter includes a rotation matrix and a translation vector.

[0150] The first matrix determining unit determines a mapping matrix from the world coordinate system to the sensor coordinate system based on the rotation matrix and the translation vector.

[0151] The second matrix determining unit takes an inverse matrix of the mapping matrix, and uses the inverse matrix of the mapping matrix as a homographic transformation matrix.

[0152] In some embodiments, the image conversion module 12 can specifically include:

[0153] The coordinate converting unit determines a vertex coordinate of the original image in the sensor coordinate system, and determines a converted coordinate of the original image in the world coordinate system based on the vertex coordinate and the homographic transformation matrix.

[0154] The first coordinate determining unit determines a center coordinate of the original image in the world coordinate system through the converted coordinate.

[0155] The second coordinate determining unit discretizes and normalizes the converted coordinate based on the converted coordinate, the center coordinate, and a preset sampling interval, to obtain a normalized coordinate corresponding to the converted coordinate.

[0156] The third coordinate determining unit determines a homographic transformation matrix of an orthographic image using the center coordinate, the preset sampling interval, and the homographic transformation matrix, and determines a pixel coordinate in the original image.

[0157] The image converting unit processes the pixel coordinate in the original image using the homographic transformation matrix of the orthographic image and the normalized coordinate, so as to convert the original image into an orthographic projection image based on the world coordinate system.

[0158] In some embodiments, the image preprocessing module 13 can specifically include:

[0159] a first image preprocessing unit configured to rotate the orthographic projection image based on a preset rotation range to obtain a rotated image, wherein the preset rotation range is a range ensuring that the ship heading in the image is between -90 degrees and +90 degrees;

[0160] a second image preprocessing unit configured to determine a wake region of the rotated image, average the color of a non-wake region in the rotated image, and cover the non-wake region in the rotated image with the obtained color average to obtain a preprocessed image.

[0161] In some embodiments, the image preprocessing module 13 can specifically include:

[0162] a first image parameter determination unit configured to perform two-dimensional fast Fourier transform on the preprocessed image to obtain a frequency spectrum image of the wake region in the preprocessed image;

[0163] a second image parameter determination unit configured to determine a spatial wave number corresponding to the wake region based on the frequency spectrum image of the wake region.

[0164] In some embodiments, the ship navigation state estimation module 14 can specifically include:

[0165] a ship navigation state determination unit configured to determine a trajectory line of the wake region in the frequency spectrum image based on the spatial wave number and the water surface dispersion characteristics, so as to determine the heading and speed of the ship in the original image by using the trajectory line.

[0166] Further, the embodiments of the present application also disclose an electronic device, Figure 13 is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the contents in the figure cannot be considered as any limitation on the use range of the present application.

[0167] Figure 13 A structural schematic diagram of an electronic device 20 provided by the embodiments of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25 and a communication bus 26. The memory 22 is configured to store a computer program, the computer program is loaded and executed by the processor 21 to implement the related steps in the ship navigation state estimation method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the present embodiment can be an electronic computer.

[0168] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input and output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0169] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0170] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the ship navigation state estimation method executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0171] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the ship navigation state estimation method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be described here.

[0172] In the specification, each embodiment is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. For the same or similar parts between the embodiments, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the relevant part.

[0173] The skilled person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0174] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The

[0175] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply or require any such actual relationship or order. Also, the terms "comprises", "comprising", or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0176] The above detailed description of the technical solutions provided by the present application has been described in detail, and the principles and implementation modes of the present application have been described by applying specific examples; the above description of the embodiments is only for the purpose of helping to understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed, and the above description of the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for estimating the navigation state of a ship, characterized in that, include: The camera sensor is used to acquire images of the ship currently in navigation to obtain raw images, the parameters of the camera sensor are determined, and the sensor coordinate system and the world coordinate system are constructed. Rigid body transformation parameters are determined based on the parameters of the camera sensor, and a homography transformation matrix is ​​determined using the rigid body transformation parameters, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system using the homography transformation matrix. The orthophoto image is preprocessed, and the frequency domain information of the wake region and the dispersion characteristics of the water surface are determined. The navigation state parameters of the ship in the original image are determined based on the frequency domain information of the wake region and the dispersion features of the water surface. The step of converting the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system using the homography transformation matrix includes: Determine the vertex coordinates of the original image in the sensor coordinate system, and determine the transformed coordinates of the original image in the world coordinate system based on the vertex coordinates and the homography transformation matrix; The center coordinates of the original image in the world coordinate system are determined by transforming the coordinates. Based on the transformed coordinates, center coordinates, and preset sampling interval, the transformed coordinates are discretized and normalized to obtain normalized coordinates corresponding to the transformed coordinates; The orthophoto homography matrix is ​​determined using the center coordinates, the preset sampling interval, and the homography matrix, and the pixel coordinates in the original image are also determined. The pixel coordinates in the original image are processed using the orthophoto homography transformation matrix and the normalized coordinates to convert the original image into an orthophoto projection image based on the world coordinate system. The preprocessing of the orthophoto image includes: The orthophoto image is rotated based on a preset rotation range to obtain the rotated image; the preset rotation range is the range that ensures the ship's heading in the image is between -90 degrees and +90 degrees. The wake region of the rotated image is determined, the average color of the non-wake region in the rotated image is calculated, and the obtained average color is used to cover the non-wake region in the rotated image to obtain the preprocessed image.

2. The ship navigation state estimation method according to claim 1, characterized in that, The process of acquiring images of the ship currently underway using a camera sensor to obtain raw images, determining the parameters of the camera sensor, and constructing a sensor coordinate system and a world coordinate system includes: The system uses a camera sensor to acquire raw images and reads the camera attitude angle and shooting height above the water surface when the camera sensor acquires the raw images; the camera attitude angle includes heading angle, side tilt angle, and image rotation angle. The position of the camera sensor when acquiring the original image is set as the origin of the sensor coordinate system, and the sensor coordinate system is created based on the origin of the sensor coordinate system; The point mapped by the camera sensor onto the sea level is determined as the origin of the world coordinate system, and the world coordinate system is created based on the origin of the world coordinate system.

3. The ship navigation state estimation method according to claim 2, characterized in that, The process of determining rigid body transformation parameters based on the parameters of the camera sensor, and then using the rigid body transformation parameters to determine the homography transformation matrix, includes: The rigid body transformation parameters are determined based on the camera attitude angle and the shooting height; the rigid body transformation parameters include a rotation matrix and a translation vector. Based on the rotation matrix and the translation vector, a mapping matrix is ​​determined from the world coordinate system to the sensor coordinate system; Take the inverse of the mapping matrix and use it as the homography transformation matrix.

4. The ship navigation state estimation method according to claim 1, characterized in that, The determination of the frequency domain information of the wake region of the preprocessed image includes: A two-dimensional fast Fourier transform is performed on the preprocessed image to obtain the spectral image of the wake region in the preprocessed image; The spatial wavenumber corresponding to the wake region is determined based on the spectral image of the wake region.

5. The ship navigation state estimation method according to claim 4, characterized in that, The step of determining the ship's navigation state parameters in the original image based on the frequency domain information of the wake region and the water surface dispersion features includes: Based on the spatial wavenumber and the water surface dispersion characteristics, the trajectory line of the wake region in the spectral image is determined, so as to use the trajectory line to determine the ship's heading and speed in the original image.

6. A ship navigation status estimation device, characterized in that, include: The coordinate system construction module is used to acquire images of the ship currently in a navigation state using a camera sensor to obtain raw images, determine the parameters of the camera sensor, and construct the sensor coordinate system and the world coordinate system. An image conversion module is used to determine rigid body transformation parameters based on the parameters of the camera sensor, and to determine a homography transformation matrix using the rigid body transformation parameters, so as to convert the original image in the sensor coordinate system into an orthographic projection image based on the world coordinate system using the homography transformation matrix. The image preprocessing module is used to preprocess the orthophoto image and determine the frequency domain information of the wake region and the dispersion characteristics of the water surface in the preprocessed image. The ship navigation state estimation module is used to determine the ship's navigation state parameters in the original image based on the frequency domain information of the wake region and the water surface dispersion features. The image conversion module includes: A coordinate transformation unit is used to determine the vertex coordinates of the original image in the sensor coordinate system, and to determine the transformed coordinates of the original image in the world coordinate system based on the vertex coordinates and the homography transformation matrix. The first coordinate determination unit is used to determine the center coordinates of the original image in the world coordinate system by the transformed coordinates; The second coordinate determination unit is used to discretize and normalize the transformed coordinates based on the transformed coordinates, the center coordinates, and the preset sampling interval to obtain the normalized coordinates corresponding to the transformed coordinates. The third coordinate determination unit is used to determine the orthophoto homography matrix using the center coordinates, the preset sampling interval, and the homography matrix, and to determine the pixel coordinates in the original image; An image conversion unit is used to process the pixel coordinates in the original image using the orthophoto homography transformation matrix and the normalized coordinates, so as to convert the original image into an orthophoto projection image based on the world coordinate system. The image preprocessing module includes: The first image preprocessing unit is used to rotate the orthophoto image based on a preset rotation range to obtain the rotated image; the preset rotation range is the range that ensures the ship's heading in the image is between -90 degrees and +90 degrees. The second image preprocessing unit is used to determine the wake region of the rotated image, calculate the average color of the non-wake region in the rotated image, and use the obtained average color to cover the non-wake region in the rotated image to obtain the preprocessed image.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the ship navigation state estimation method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the ship navigation state estimation method as described in any one of claims 1 to 5.

Images