A ship panoramic image generation method and device, a storage medium and an electronic device

Abstract

The application discloses a ship panoramic image generation method and device, a storage medium and an electronic device. The generation method comprises the following steps: determining a conversion matrix between every two adjacent camera coordinate systems; calculating a gravity direction vector in a reference camera coordinate system; calculating a rotation matrix from the reference camera coordinate system to a world coordinate system based on a parallel relationship between the gravity direction vector in the reference camera coordinate system and a Z-axis of the world coordinate system, and determining a conversion matrix from the reference camera coordinate system to the world coordinate system based on the rotation matrix and a translation t; calculating conversion matrices from camera coordinate systems other than the reference camera to the world coordinate system; determining a single mapping matrix from each camera to a panoramic image coordinate system based on the conversion matrices from each camera coordinate system to the world coordinate system, and projecting images captured by each camera to the panoramic image coordinate system to generate a panoramic image. The application can effectively improve the quality of the panoramic image.

Description

Technical Field

[0001] This application relates to image processing technology, and in particular to a method, apparatus, storage medium, and electronic device for generating panoramic images of ships. Background Technology

[0002] With the development of the marine transportation industry and the domestic shipbuilding industry, the demand for intelligent auxiliary systems for ships is increasing. Among them, panoramic imaging systems play a significant role in early warning of ships berthing and departing from shore. A ship's panoramic image refers to an image generated from a view directly above the ship, providing a direct 360-degree view of the surroundings, often referred to as a bird's-eye view. However, the application of panoramic imaging systems in the shipbuilding field is still relatively limited. The main reasons are: firstly, due to the large size of ships and the difficulty in finding a stable calibration reference plane, the transformation matrix between the ship's camera and the world coordinate system is not easily calibrated as it is with vehicle-mounted panoramic imaging systems; secondly, ships experience pitch and roll with the waves, so the parameters from the camera to the sea level are not fixed. Therefore, using a fixed transformation matrix between the ship and the world coordinate system during offline calibration to generate panoramic images results in poor image quality. Summary of the Invention

[0003] This application provides a method, apparatus, storage medium, and electronic device for generating panoramic images of ships, which can effectively improve the quality of panoramic images.

[0004] To achieve the above objectives, this application adopts the following technical solution:

[0005] A method for generating panoramic images of a ship includes installing multiple camera modules around the ship's hull. Each camera module includes at least one camera, and at least one camera module also includes an IMU. The method includes:

[0006] Determine the transformation matrix between every two adjacent camera coordinate systems;

[0007] Gravity direction estimation based on the IMU is used to determine the gravity direction vector in the reference IMU coordinate system. Then, based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system, the gravity direction vector in the reference camera coordinate system is calculated. The reference IMU and the reference camera belong to a preset reference camera module.

[0008] Based on the gravity direction vector in the reference camera coordinate system Calculate the parallel relationship between the reference camera coordinate system and the Z-axis of the world coordinate system, and the rotation matrix from the reference camera coordinate system to the world coordinate system. The transformation matrix from the reference camera coordinate system to the world coordinate system is determined based on the rotation matrix and the translation t from the reference camera coordinate system to the world coordinate system.

[0009] Based on the transformation matrix between every two adjacent camera coordinate systems and the transformation matrix from the reference camera coordinate system to the world coordinate system, calculate the transformation matrix from the coordinate systems of other cameras besides the reference camera to the world coordinate system;

[0010] Based on the transformation matrix from each camera coordinate system to the world coordinate system, the single-image matrix from each camera to the panoramic image coordinate system is determined, and the images captured by each camera are projected onto the panoramic image coordinate system using the single-image matrix. The projection results of each camera are then stitched together to generate a panoramic image. The UV plane of the panoramic image coordinate system is parallel to the XY plane of the world coordinate system.

[0011] Preferably, the calculation of the rotation matrix The possible values ​​include:

[0012]

[0013] in, α is the preset yaw angle from the reference camera to the world coordinate system, and nx, ny, and nz are the gravity direction vectors in the reference camera coordinate system, respectively. The three components take values.

[0014] Preferably, when the reference camera is installed on the hull in the positive direction of the X-axis of the world coordinate system, the value of α is -π / 2; when the reference camera is installed on the hull in the negative direction of the X-axis of the world coordinate system, the value of α is -3π / 2; when the reference camera is installed on the hull in the positive direction of the Y-axis of the world coordinate system, the value of α is 0; and when the reference camera is installed on the hull in the negative direction of the Y-axis of the world coordinate system, the value of α is -π.

[0015] Preferably, the z-coordinate of the translation t is determined by:

[0016] a. Select two adjacent cameras to capture sea level images of the area shared by the two adjacent cameras. Identify a pair of matching points corresponding to the same target point in the two sea level images. Based on the coordinates of the matching point pair in the coordinate systems of the two adjacent cameras and the transformation matrix between the coordinate systems of the two adjacent cameras and the reference camera, calculate the coordinates p of the target point in the reference camera coordinate system. ref ;

[0017] b. Determine the initial z-coordinate as z = -(R) 31 x ref +R 32 y ref +R 33 z ref); where R 31 For the rotation matrix The element in the 3rd row and 1st column, R 32 For the rotation matrix The element in the 3rd row and 2nd column, R 33 For the rotation matrix The element in the 3rd row and 3rd column;

[0018] c. Use the initial z-coordinate as the z-coordinate of the translation amount t, or, repeat steps a and b to obtain several initial z-coordinates, perform a weighted average of all initial z-coordinates, and use the weighted average result as the z-coordinate of the translation amount t.

[0019] Preferably, determining the gravity direction vector in the reference IMU coordinate system includes:

[0020] Gravity direction estimation is performed using a reference IMU to obtain the gravity direction vector in the reference IMU coordinate system; or...

[0021] In addition to the reference camera module, at least one camera module includes an IMU, which performs gravity direction estimation using each IMU, and transforms the gravity direction vector estimated by each IMU to the reference IMU coordinate system based on the transformation matrix between every two adjacent camera coordinate systems, and determines the gravity direction vector in the reference IMU coordinate system based on the transformed gravity direction vectors.

[0022] Preferably, the method for determining the transformation matrix between every two adjacent camera coordinate systems includes:

[0023] For every two adjacent cameras, corner detection is performed based on the images acquired by the adjacent cameras for the same calibration board. The corner detection results of the adjacent cameras are then matched to obtain multiple pairs of detected corner coordinates. Among them, the The detection result of the diagonal point k of the first camera among the adjacent cameras, the The detection result of the diagonal point k of the second camera in the adjacent cameras, where k≤n, and n is the total number of detected coordinate pairs;

[0024] Based on the detected coordinate pairs of the multiple corner points, determine the relationship between the detected coordinate pairs that most closely approximates the first functional relationship. Transformation matrix As the initial transformation matrix between every two adjacent camera coordinate systems

[0025] All the initial transformation matrices are optimized as a whole using the pose graph optimization method, and the optimized initial transformation matrices are used as the transformation matrices between the corresponding two adjacent camera coordinate systems.

[0026] Preferably, determining the monoimation matrix of each camera to the panoramic image coordinate system includes:

[0027] Based on the transformation matrix from each camera coordinate system to the world coordinate system, the intrinsic parameter matrix of the camera, and the distortion coefficient, calculate the projection matrix of the distorted image of each camera onto the XY plane of the world coordinate system.

[0028] Multiply the affine transformation matrix from the world coordinate system to the panoramic image coordinate system by the inverse of the projection matrix to obtain the single-image matrix from each camera to the panoramic image coordinate system.

[0029] A ship panoramic image generation device includes multiple camera modules installed around the ship's side. Each camera module includes at least one camera, and at least one camera module also includes an IMU. The device includes: a camera extrinsic parameter calibration unit, a gravity direction estimation unit, a transformation matrix calculation unit, a single-image matrix calculation unit, and a panoramic image generation unit.

[0030] The inter-camera extrinsic calibration unit is used to determine the transformation matrix between every two adjacent camera coordinate systems;

[0031] The gravity direction estimation unit is used to determine the gravity direction vector in the reference IMU coordinate system through IMU-based gravity direction estimation, and to calculate the gravity direction vector in the reference camera coordinate system based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system. The reference IMU and the reference camera belong to a preset reference camera module;

[0032] The transformation matrix calculation unit is used to calculate the gravity direction vector in the reference camera coordinate system. Calculate the parallel relationship between the reference camera coordinate system and the Z-axis of the world coordinate system, and the rotation matrix from the reference camera coordinate system to the world coordinate system. The transformation matrix from the reference camera coordinate system to the world coordinate system is determined based on the rotation matrix and the translation t from the reference camera coordinate system to the world coordinate system. It is also used to calculate the transformation matrix from the coordinate systems of other cameras besides the reference camera to the world coordinate system based on the transformation matrix between every two adjacent camera coordinate systems and the transformation matrix from the reference camera coordinate system to the world coordinate system;

[0033] The mono-matrix calculation unit is used to determine the mono-matrix from each camera to the panoramic image coordinate system based on the transformation matrix from each camera coordinate system to the world coordinate system.

[0034] The panoramic image generation unit is used to project images captured by each camera onto the panoramic image coordinate system using the single-image matrix, and to stitch the projection results of each camera together to generate a panoramic image; wherein, the UV plane of the panoramic image coordinate system is parallel to the XY plane of the world coordinate system.

[0035] A computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, can implement the ship panoramic image generation method described above.

[0036] An electronic device, comprising at least a computer-readable storage medium and a processor;

[0037] The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the ship panoramic image generation method described above.

[0038] As can be seen from the above technical solution, in this application, camera modules are installed around the hull, each camera module including a camera, and at least one camera module including an inertial measurement unit (IMU); the transformation matrix between every two adjacent camera coordinate systems is pre-determined, thereby clarifying the relative pose relationship between each camera; then, the gravity direction vector in the reference IMU coordinate system is determined by gravity direction estimation based on the IMU, and the gravity direction vector in the reference camera coordinate system is calculated based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system. Next, based on the gravity direction vector Calculate the rotation matrix based on the parallel relationship between the axis and the world coordinate system's Z-axis. Furthermore, the transformation matrix from the reference camera coordinate system to the world coordinate system is determined. This transformation matrix can be calculated using the gravity direction estimation in the reference camera coordinate system and the property that the gravity direction is parallel to the Z-axis of the world coordinate system. Since gravity estimation can be performed in real time, the transformation matrix calculated from the gravity estimation result also reflects the real-time relationship between the reference camera coordinate system and the world coordinate system. Combined with the pose relationships between the various cameras, the transformation matrices between all cameras and the world coordinate system can be calculated. Finally, based on the transformation matrices between each camera and the world coordinate system, the single-image matrix from each camera to the panoramic image coordinate system is determined. The single-image matrix is ​​then used to project the images captured by the cameras onto the panoramic image coordinate system, and the projection results from each camera are stitched together to generate a panoramic image. In this way, on the one hand, the relative pose relationship between different cameras is determined in advance through fixed calibration, and on the other hand, the transformation matrix from the reference camera coordinate system to the world coordinate system is calculated in real time using the gravity direction measured in real time. This ensures that the transformation matrix from each camera coordinate system to the world coordinate system can accurately reflect the real-time relationship between each camera coordinate system and the world coordinate system. Then, the real-time accurate transformation matrix is ​​used to generate a single-image matrix from the camera to the sea level and is used to generate panoramic images, which can effectively ensure the quality of panoramic images of ships under various conditions. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the basic process of the ship panoramic image generation method in this application;

[0040] Figure 2 This is a schematic diagram illustrating the specific process of the ship panoramic image generation method in a particular embodiment of this application;

[0041] Figure 3 This is a schematic diagram showing the installation of the camera module around the hull.

[0042] Figure 4 This is a schematic diagram illustrating the relationship between the IMU and the camera coordinate system within the same camera module.

[0043] Figure 5 A schematic diagram of the chessboard pattern for the calibration board;

[0044] Figure 6 This is a schematic diagram of the inverse perspective transformation from the camera plane to the sea level.

[0045] Figure 7a and Figure 7b These are schematic diagrams showing the relationship between the world coordinate system, reference camera coordinate system, reference IMU coordinate system, and gravity direction under ship tilt and ship horizontal states, respectively.

[0046] Figure 8This is a schematic diagram illustrating the relationship between the pixel coordinate system and the world coordinate system of a panoramic image.

[0047] Figure 9 This is a schematic diagram of the basic structure of the ship panoramic image generation device in this application;

[0048] Figure 10 A schematic diagram of the basic structure of the electronic device provided in this application. Detailed Implementation

[0049] To make the objectives, technical means, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings.

[0050] The basic idea of ​​this application is to install an IMU module on a traditional panoramic imaging camera to estimate the direction vector of gravity in real time when the ship is sailing on the water. Since the direction of gravity is perpendicular to the ideal sea level, the single-image matrix from the camera to the sea level can be estimated in real time according to the direction of gravity. The stitching of multiple cameras is completed based on the real-time estimated single-image matrix, which effectively improves the image quality of the panoramic stitching result under various ship attitudes.

[0051] Figure 1 This is a schematic diagram illustrating the basic process of the ship panoramic image generation method in this application. Multiple camera modules are installed around the ship's hull. Each camera module includes at least one camera, and at least one camera module also includes an IMU (Instrument Module). Among all camera modules including an IMU, a certain camera module is designated as a reference camera module. The camera in this reference camera module serves as the reference camera, and the IMU in the reference camera module serves as the reference IMU. Figure 1 As shown, the method includes:

[0052] Step 101: Determine the transformation matrix between every two adjacent camera coordinate systems.

[0053] Before generating real-time panoramic images, the transformation matrix between every two adjacent camera coordinate systems is determined by pre-calibration. This allows us to obtain the relative pose relationships between the various cameras mounted on the ship, which is then used to prepare for determining the relationships between each camera and the world coordinate system and the panoramic image coordinate system.

[0054] Step 102: Determine the gravity direction vector in the reference IMU coordinate system by estimating the gravity direction based on the IMU, and calculate the gravity direction vector in the reference camera coordinate system based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system.

[0055] In this application, an IMU is installed in the camera module to measure triaxial acceleration and angular velocity, thereby enabling real-time estimation of the direction of gravity.

[0056] Gravity direction estimation is performed using an IMU to determine the gravity direction vector in the reference IMU coordinate system. The specific gravity direction estimation process can be implemented using existing methods. In practice, gravity direction estimation can be performed using only the reference IMU to obtain the gravity direction vector in the reference IMU coordinate system, or multiple IMUs can be used to estimate the gravity direction, and then the multiple estimation results can be combined to obtain the gravity direction vector in the reference IMU coordinate system.

[0057] After obtaining the gravity direction vector in the reference IMU coordinate system, the gravity direction vector in the reference IMU coordinate system can be converted to the gravity direction vector in the reference camera coordinate system using the pre-calibrated transformation matrix between the reference IMU coordinate system and the reference camera coordinate system. The reason for transforming the gravity direction vector to the reference camera coordinate system is that when generating panoramic images, it is necessary to calculate the single-image matrix from the camera to the panoramic image coordinate system. Therefore, the gravity direction vector needs to be expressed in the camera coordinate system to facilitate the subsequent calculation of the single-image matrix.

[0058] Step 103, based on the gravity direction vector in the reference camera coordinate system Calculate the rotation matrix from the reference camera coordinate system to the world coordinate system, considering the parallelism between the reference camera coordinate system and the Z-axis of the world coordinate system. Based on this rotation matrix and the translation t between the world coordinate system and the reference camera coordinate system, the transformation matrix from the reference camera coordinate system to the world coordinate system is determined.

[0059] In the generation of panoramic images of ships, the XY plane of the world coordinate system is the ideal sea level. Therefore, the gravity direction vector is perpendicular to the XY plane of the world coordinate system, that is, parallel to the Z-axis of the world coordinate system. Based on this, the rotation matrix between the reference camera coordinate system and the world coordinate system is used... Multiplying the gravity direction vector by the gravity direction vector yields the gravity direction vector in the world coordinate system, which is equal to the Z-axis direction vector in the world coordinate system. Since the two direction vectors lie in the same coordinate system, they can be directly established as equals. The rotation matrix can then be calculated using this equality. The specific value to be taken.

[0060] After obtaining the rotation matrix After determining the value of , combining it with the translation amount t yields the transformation matrix from the reference camera coordinate system to the world coordinate system.

[0061] In the above steps 102-103, the real-time estimation of gravity using the IMU can obtain the real-time estimation result of the gravity direction vector. Regardless of the current ship state, the accuracy of the real-time gravity estimation result can be guaranteed. Based on the gravity estimation result, the transformation matrix from the reference camera coordinate system to the world coordinate system is calculated. This ensures that a real-time and accurate transformation matrix can be obtained regardless of the ship state, so as to express the real-time relationship between the reference camera coordinate system and the world coordinate system.

[0062] Step 104: Based on the transformation matrix between every two adjacent camera coordinate systems and the transformation matrix from the reference camera coordinate system to the world coordinate system, calculate the transformation matrix from the other camera coordinate systems to the world coordinate system.

[0063] Since step 101 has determined the relative pose relationships between different camera coordinate systems, this relative pose relationship can be used to transform the transformation matrix from the reference camera coordinate system to the world coordinate system into transformation matrices from each of the other camera coordinate systems to the world coordinate system. Furthermore, because of the transformation matrix from the reference camera coordinate system to the world coordinate system, the transformation matrices from each of the other camera coordinate systems to the world coordinate system also accurately and in real-time express the real-time relationship between the corresponding camera coordinate system and the world coordinate system.

[0064] Step 105: Based on the transformation matrix from each camera coordinate system to the world coordinate system, determine the single-image matrix from each camera to the panoramic image coordinate system, and use the single-image matrix to project the images captured by each camera to the panoramic image coordinate system. Then, stitch the projection results of each camera together to generate a panoramic image.

[0065] When generating panoramic images, the UV plane of the panoramic image coordinate system is parallel to the XY plane of the world coordinate system. Therefore, based on the transformation matrix from each camera coordinate system to the world coordinate system, the single-image matrix from each camera to the panoramic image coordinate system can be determined. Then, the images captured by each camera are projected onto the panoramic image coordinate system using the single-image matrix, and the projection results of each camera are stitched together to generate panoramic images.

[0066] Since the transformation matrices from each camera coordinate system to the world coordinate system have been analyzed in real time and are accurate, the generated mono-mapping matrix is ​​also real time and accurate and applicable to various ship states. Therefore, the panoramic image generated by projecting the images acquired by the camera is also real time and accurate, avoiding the problem of panoramic image quality degradation caused by fixed mono-mapping matrices under different ship states.

[0067] At this point, Figure 1The basic flow of the ship panoramic image generation method shown is completed. Specific embodiments of the ship panoramic image generation method of the present application are given below to describe in detail the specific implementation of each step in the panoramic image generation method.

[0068] Figure 2 is a schematic diagram of the specific flow of the ship panoramic image generation method in the specific embodiment of the present application. In this specific embodiment, camera modules are installed around the hull, which are specifically fisheye camera modules in this embodiment. Each fisheye camera module includes a fisheye camera, at least one fisheye camera module includes an IMU, and the fisheye camera and the IMU are fixedly connected and installed through a mechanical structure. Of course, more preferably, all fisheye camera modules may be equipped with an IMU. Each fisheye camera module has a hardware synchronization interface for synchronously triggering exposure. The cameras can be connected to the acquisition system via Ethernet or coaxial cable for image acquisition.

[0069] For the generation of ship panoramic images, usually at least 4 camera modules are used, which are respectively installed at the bow, stern, port side and starboard side. The camera modules are fixedly connected to the hull. If the hull is large, several cameras can be added symmetrically and synchronously on the port and starboard sides to ensure that the distance d between the cameras is less than a certain value. This distance is not fixed, but it is necessary to ensure that there is a common view area between the cameras, and the common view area must ensure the normal development of the calibration work. According to the existing fisheye camera parameters, the camera spacing can generally be set to 10 to 20 meters. For the convenience of subsequent description, the cameras are numbered here. The camera installed at the bow is No. 0, and the camera numbers increase clockwise, denoted as C i}, i<M, where M is the total number of installed camera modules. Figure 3 shows an example diagram of the arrangement of camera modules around the hull.

[0070] As shown Figure 2 , the panoramic image generation method in the embodiment includes:

[0071] Step 201: Perform factory calibration on the camera and the IMU in the camera module, determine the camera internal parameter matrix K, the distortion coefficient D, the zero bias of the gyroscope and the zero bias of the accelerometer in the IMU, and determine the transformation matrix between the camera and the IMU in the same camera module.

[0072] The camera internal parameter matrix describes the projection relationship of 3D points to the camera normalization plane:

[0073] The distortion coefficient D describes the relationship between the distorted plane pixel coordinates and the normalized plane pixel coordinates. The calibration of camera internal parameters is very mature, and the internal parameter K and the distortion coefficient D can be obtained using the "Zhang Zhengyou" calibration scheme, which will not be repeated here.

[0074] An IMU is an inertial measurement unit that includes a triaxial angular velocity meter and a triaxial accelerometer. In this application, the measurement noise of the angular velocity meter and accelerometer is simplified to zero bias error and white noise error, and random walk error and temperature drift error are not considered for the time being. White noise is random noise with a mean of 0, which can be processed in subsequent algorithms; therefore, the main noise is zero bias noise.

[0075] The zero-bias noise of an angular velocity meter can be estimated by averaging the measurements taken when the IMU is stationary. The zero-bias noise of an accelerometer can be estimated using a turntable method, where the IMU is slowly rotated to different angles and then stationary to obtain measurements. Since the magnitude of the accelerometer measurements at different angles should be constant equal to the local gravitational acceleration, a least-squares problem can be constructed to solve for the accelerometer's zero-bias value. Both of these zero-bias noise estimation methods are mature and will not be elaborated further.

[0076] For camera modules that include both the camera and the IMU, such as Figure 4 As shown, there is also an external parameter associated with the IMU and the camera during installation. This refers to the transformation relationship between the IMU and camera coordinate systems, specifically the transformation matrix from the IMU coordinate system to the camera coordinate system. This extrinsic parameter can be calibrated using the existing Kalibr camera-IMU joint calibration tool, so it will not be elaborated further.

[0077] Step 202: Calibrate each camera mounted on the ship's hull and determine the initial transformation matrix between every two adjacent camera coordinate systems.

[0078] The camera module is arranged as follows, for example Figure 3 After being rigidly fixed to the hull as shown, the coordinate system transformation matrix between cameras (i.e., the extrinsic parameters between cameras) is calibrated.

[0079] The extrinsic parameter calibration between any two adjacent cameras can be the same as the existing stereo camera extrinsic parameter calibration scheme. Specifically, first prepare a calibration board with alternating black and white grids. The black and white grids on the calibration board should be of uniform size, and the side length L of the grid should be measurable. A coordinate system can be constructed in the upper left corner of the calibration board, with a maximum number of grids P, such as... Figure 5 As shown. The coordinates of the corner points of the chessboard grid in the calibration board can be denoted as P. w p and q are the chessboard grid numbers corresponding to the corner points:

[0080]

[0081] If a certain camera c i The entire chessboard can be observed. According to the camera's projection equation, the corner point P of the chessboard... w The ideal projected pixel coordinates are:

[0082]

[0083] Based on existing checkerboard corner detection algorithms, the pixel coordinates of any corner point of the checkerboard on the normalized imaging plane can be obtained. Based on existing chessboard corner matching algorithms, the coordinates of the detected corners can be... With the corner point P of the chessboard w By establishing a one-to-one matching relationship, and through the detection coordinates of n corner points and the corner point matching relationship, n matching pairs can be obtained. Then, together with the ideal projected pixel coordinates in equation (4), a least squares problem is formed to solve for the camera c. i Transformation matrix to calibration plate coordinate system

[0084]

[0085] Consider two adjacent cameras c i With c i+1 If the chessboard square is within the shared field of view of both cameras, then the same corner point P can be written out separately. w Ideal projection pixel coordinates of the two cameras:

[0086]

[0087] From equation (5), we can obtain: Will Split into and Substituting into equation (6) together, we get:

[0088]

[0089] Based on the chessboard corner detection algorithm, P can be obtained. w In two cameras c i and c i+1 The coordinates in the pixel coordinate system are respectively and Corner point P w Corresponding matching pairs (Also known as detecting coordinate pairs), this is done in the same way for n corner points. Corner detection is performed based on two camera images to obtain corner points. Corresponding matching pairs k≤n.

[0090] Equation (7) above expresses the relationship between two ideal coordinates of the same corner point in two adjacent camera coordinate systems. The actual measured detection coordinate matching pair... This relationship should also be satisfied; based on this, matching pairs corresponding to n corner points can be used. Determine the matching pair The transformation matrix that most closely approximates the functional relationship shown in equation (7) is the one with the closest relationship. This transformation matrix can also be viewed as a transformation matrix between two adjacent camera coordinate systems.

[0091] Specifically, a least squares problem can be constructed to solve it.

[0092]

[0093] Here, Qi is the information matrix of the optimization problem, used to characterize the weight of pixel reprojection error in the optimization problem; specifically, it can be a 2*2 diagonal matrix. The above least-squares optimization problem can be solved using mature methods such as the Gauss-Newton method or the LM algorithm, which will not be elaborated here. This yields the transformation matrix between two adjacent camera coordinate systems. For every two adjacent cameras, the corresponding transformation matrix is ​​determined in the same way, thus obtaining the transformation matrix between every two adjacent camera coordinate systems.

[0094] In this embodiment, the transformation matrix obtained above can be regarded as the initial transformation matrix. To further improve the accuracy of the transformation matrix, the initial transformation matrix can be globally optimized through the processing in step 203 below. Of course, in practical applications, the processing in step 203 can be omitted according to actual needs, and the transformation matrix obtained in step 202 can be directly used for subsequent processing.

[0095] Step 203: Perform global optimization on the transformation matrix between every two adjacent camera coordinate systems.

[0096] The transformation matrix between each pair of adjacent camera coordinate systems can be independently determined through the processing in step 202 above. However, the calibration of the above transformation matrix has a certain error. When the transformation matrix is ​​passed between multiple cameras, the accumulated error will cause a large deviation in the final result. Therefore, after the transformation matrix between the last camera and the first camera coordinate system is estimated by equations (8) and (9), a closed-loop constraint can be constructed, and the transformation matrix between cameras can be optimized as a whole by using pose graph optimization.

[0097] The pose graph optimization scheme assumes that the coordinate system of camera c0 (position 0) is used as the reference coordinate system. This is achieved by using the transformation matrices of a series of adjacent cameras. When the data is retransmitted to c0, the positions and orientations of both cameras should be completely identical, meaning the pose error is zero. Therefore, the error of the closed-loop edge can be defined as the product of the transformation matrices between cameras:

[0098]

[0099] Here, % represents the modulo operation.

[0100] In the pose graph optimization scheme, the constraints of adjacent edges are the initial transformation matrix between each pair of cameras estimated by equation (9), which is the same as the final transformation matrix. To distinguish them, the initial transformation matrix is ​​represented here as Therefore, the residual between adjacent edges can be defined as:

[0101]

[0102] Therefore, the overall optimization problem can be defined as:

[0103]

[0104] in, and Q loop Both are information matrices for the optimization problem, representing the weights of position and attitude errors in the optimization process. Specifically, they can be 6x6 diagonal matrices. This least-squares optimization problem can be solved using mature methods such as the Gauss-Newton method or the LM algorithm, yielding the optimized transformation matrix. The specific solution method will not be elaborated here.

[0105] This completes the calibration of the transformation matrices between the cameras installed around the ship's hull. This calibration process can be performed offline beforehand, without requiring real-time calculations. In summary, the camera module installation and camera transformation matrix calibration process can include:

[0106] 1) Determine the number of cameras based on the length of the ship and fix the camera modules to the ship's hull.

[0107] 2) Prepare a checkerboard calibration board of known size. Position the checkerboard calibration board facing the camera's field of view and move the calibration board to the common field of view of the cameras, acquiring images for several corresponding cameras; repeat step 2) to ensure that images are acquired for the common field of view of all cameras;

[0108] 3) The acquired images are deformed according to the camera's intrinsic parameters and distortion parameters;

[0109] 4) Detect the coordinates of the checkerboard corner points in the distorted image and match the checkerboard corner point detection results of adjacent cameras to obtain several matching pairs;

[0110] 5) Based on the optimization problem described by equation (9), the initial transformation matrix between every two adjacent camera coordinate systems is obtained.

[0111] 6) Based on the pose graph optimization problem described by equation (12), the globally optimal transformation matrix between cameras is obtained.

[0112] In generating panoramic images of ships, besides needing to know the transformation matrices between cameras, it is also necessary to solve for the single-image matrix from each camera to the sea-level coordinate system in order to perform inverse perspective transformation in order to stitch the panoramic images together. A schematic diagram of the inverse perspective transformation from the camera plane to the sea plane is shown below. Figure 6 As shown. Next, we will introduce the calculation process of the single-image matrix from the camera to the ideal sea-level coordinate system.

[0113] Step 204: Estimate the gravity direction based on the IMU and determine the unit vector of the gravity direction in the reference IMU coordinate system.

[0114] At least one camera module of this application is equipped with an IMU sensor that can measure three-axis acceleration and three-axis angular velocity, and can achieve gravity direction estimation.

[0115] In this embodiment, it is assumed that multiple camera modules on the ship are equipped with IMUs. One of the camera modules equipped with IMUs is selected as a reference camera module. The camera of the reference camera module is called the reference camera, and the IMU of the reference camera module is called the reference IMU.

[0116] For each camera module equipped with an IMU, gravity direction estimation is performed using the IMU to obtain a unit vector of gravity direction in the corresponding IMU coordinate system. The gravity direction estimation using the IMU can be performed using various existing methods.

[0117] More specifically, since the IMU in this application is installed on a ship, when the ship is traveling at a constant speed on still water with no waves or is stationary on a calm horizontal surface, the three-axis acceleration of the IMU can be regarded as the components of gravitational acceleration g→ along the three axes of the IMU. However, on the sea surface, the ship hull is constantly affected by wind and waves, and coupled with the ship's own power, this leads to significant noise in the accelerometer measurements, making them unusable directly as components of gravitational acceleration. Therefore, in this embodiment, preferably, the existing AHRS algorithm is used to fuse the accelerometer and angular velocity, which can greatly reduce the influence of accelerometer noise and finally obtain the pitch angle θ and roll angle σ of a single IMU. From this, the unit direction vector of gravity in this IMU system can be obtained:

[0118]

[0119] Of course, in addition to the AHRS algorithm, methods such as VSLAM and VIO can also be used to estimate the unit direction vector of gravity in the IMU frame. Here, no constraints are placed on the estimation scheme of the gravity direction.

[0120] Following the above method, gravity direction estimation can be performed using each IMU to obtain the gravity unit direction vector in the corresponding IMU coordinate system. Then, combined with the transformation matrix between adjacent cameras obtained in steps 202 or 203 above, the gravity unit direction vectors can be unified to the reference IMU coordinate system (such as the IMU coordinate system of camera module 0).

[0121]

[0122] Subsequently, the gravity unit direction vector in the reference IMU coordinate system can be synthesized based on the various gravity unit direction vectors unified to the reference IMU coordinate system. The specific synthesis method can be set as needed. A scheme that uses random sampling consistency can be used to filter out outliers in the direction vectors and average the remaining valid direction vectors to obtain a more stable gravity unit direction vector.

[0123]

[0124] Step 205: Based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system, calculate the gravity direction vector in the reference camera coordinate system.

[0125] Obtain the unit gravity direction vector in the reference IMU coordinate system. Subsequently, since the transformation matrix from the IMU within the camera module to the camera coordinate system has already been obtained through the aforementioned step 201, in, The rotation matrix from the reference IMU to the reference camera coordinate system allows us to calculate the unit direction vector of gravity in the reference camera frame:

[0126]

[0127] Step 206: Calculate the rotation matrix from the reference camera coordinate system to the world coordinate system.

[0128] In the generation of panoramic images of ships, the XY plane of the world coordinate system is usually the ideal sea level, with the origin being the projection of the ship's center onto the sea level. Considering that the direction of gravity points towards the Earth's center and is perpendicular to the ideal sea level, the opposite direction of gravity is the Z-axis direction of the world coordinate system. In this embodiment, the X-axis of the world coordinate system points towards the bow, the Y-axis points towards the left side of the ship, and the Z-axis points towards the sky. It is understandable that, due to the rigid connection between the ship and the camera module, the swaying of the ship will cause the direction of gravity in the reference camera coordinate system to constantly change; that is, the unit direction vector of gravity in the reference camera coordinate system... The value of will change continuously, such as Figure 7a and Figure 7bThe diagram shows the relationship between the world coordinate system, the reference camera coordinate system, the reference IMU coordinate system, and the direction of gravity under the ship's tilted and horizontal states. Therefore, the rotation of the world coordinate system relative to the reference camera coordinate system is constantly changing, that is, the rotation matrix from the reference camera coordinate system to the world coordinate system is constantly changing.

[0129] Let the transformation matrix from the reference camera coordinate system to the world coordinate system be: Let t be the rotation matrix from the reference camera coordinate system to the world coordinate system, and t be the translation amount from the reference camera coordinate system to the world coordinate system.

[0130] Based on rotation matrix The unit gravity direction vector in the reference camera coordinate system can be used. Convert to gravity unit direction vector in world coordinate system Because this direction is parallel to the z-axis of the world coordinate system, the following relationship can be obtained:

[0131]

[0132] Will Decomposed into Euler angles expressed in terms of yaw angle (α), pitch angle (β), and roll angle (θ), i.e.:

[0133]

[0134] Based on equation (18), multiply both sides of equation (17) by R(α) simultaneously. -1 We can obtain:

[0135]

[0136] Solving the above equation yields:

[0137]

[0138] According to the above formula, it can be seen that the solution of equation (19) is independent of the yaw angle. In this case, α is the preset yaw angle from the reference camera to the world coordinate system. Specifically, to ensure full utilization of the camera's field of view during installation, the camera's optical axis is basically perpendicular to the hull at the installation location. Therefore, in this embodiment... Figure 3 The coordinate system definition and camera module example shown can be used to list the yaw angles of different cameras to the world coordinate system.

[0139]

[0140] Where i is the camera number given above. Based on the location of the reference camera, the corresponding α is selected, and then, combined with equations (20) and (18), the rotation matrix is ​​calculated.

[0141] Furthermore, the above equation (21) is based on Figure 3 The coordinate system definition and the number of camera modules are used to determine the α value for each camera with a different number. In fact, the optical axis of the camera is basically perpendicular to the hull of the ship at the installation position. Therefore, the α value for different cameras is actually determined based on the position of the camera and the direction of the X and Y axes of the world coordinate system. Specifically, when the reference camera is installed on the hull in the positive direction of the X axis of the world coordinate system, the value of α is -π / 2. When the reference camera is installed on the hull in the opposite direction of the X axis of the world coordinate system, the value of α is -3π / 2. When the reference camera is installed on the hull in the positive direction of the Y axis of the world coordinate system, the value of α is 0. When the reference camera is installed on the hull in the opposite direction of the Y axis of the world coordinate system, the value of α is -π.

[0142] Step 207: Based on the rotation matrix from the reference camera coordinate system to the world coordinate system and the translation amount t between the reference camera coordinate system and the world coordinate system, determine the transformation matrix from the reference camera coordinate system to the world coordinate system.

[0143] According to the definition of the world coordinate system mentioned above, the x and y coordinates in t can be obtained by measuring the translation components of the reference camera to the world coordinate system, and are not sensitive to measurement deviations. However, the z coordinate, i.e. the camera installation height, has a more serious impact on the final image stitching. Similar to the x and y coordinates, the z coordinate can also be obtained by measuring the vertical distance from the center of the reference camera installation position to the target plane (i.e., sea level) of the top view projection using measuring devices such as tape measures.

[0144] Furthermore, preferably, considering that different ship loads lead to different drafts, the distance z from the camera to the sea level is actually a time-varying variable. Based on this, this embodiment provides a dynamic measurement scheme for the z-coordinate:

[0145] To find the target plane (i.e., sea level) projected from the top view within the shared field of view of two camera images, two cameras capable of capturing the shared field of view acquire images of the sea level. In these two images, feature extraction and matching schemes are used to determine a pair of matching points corresponding to the same target point. Specifically, one pixel is selected from each image, corresponding to the same target point on the actual sea level. The coordinates of these two selected pixels are called the coordinates of a matching point pair in their respective coordinate systems. Combined with the transformation matrix from the two cameras to the reference camera coordinate system, triangulation can be used to obtain the coordinates p of the target point in the reference camera coordinate system. ref =[x ref y ref z ref ] T Among them, p was calculated using the triangulation method. refThe processing can be achieved using existing methods, which will not be elaborated here;

[0146] Next, we will use the transformation matrix from the reference camera coordinate system to the world coordinate system. Transform the target point coordinates to coordinates p in the world coordinate system. w Meanwhile, the target point is located on sea level, so its z-coordinate in the world coordinate system is 0. Therefore, we can conclude that:

[0147]

[0148] Then we can obtain:

[0149] z = -(R) 31 x ref +R 32 y ref +R 33 z ref ) (twenty three)

[0150] Among them, R 31 For the rotation matrix The element in the 3rd row and 1st column, R 32 For the rotation matrix The element in the 3rd row and 2nd column, R 33 For the rotation matrix The element in the 3rd row and 3rd column.

[0151] If there are multiple matching point pairs corresponding to multiple target points, the z values ​​obtained from the multiple matching point pairs according to equation (23) can be weighted and averaged to obtain the z coordinate of the translation amount t.

[0152] Thus far, the transformation matrix from the reference camera coordinate system to the world coordinate system has been obtained. Once the solution is complete, the transformation matrix between adjacent camera coordinate systems obtained in steps 202 or 203 above is used. This allows us to obtain the transformation matrices from the coordinate systems of all cameras except the reference camera to the world coordinate system.

[0153] Step 208: Calculate the affine transformation matrix from the world coordinate system to the panoramic image coordinate system.

[0154] This step can be performed using existing methods. Specifically, the panoramic image parameters can be set according to the ship's size and the required field of view as follows:

[0155] 1) Target panoramic image field of view: Based on the physical dimensions in the X and Y directions in the world coordinate system, such as H×W meters;

[0156] 2) Resolution: pixels per meter, such as pixels / m;

[0157] 3) Target panoramic image size: Width W of the panoramic image pix =W·s, the height H of the panoramic image pix =H·s.

[0158] like Figure 8 As shown, the UV plane of the panoramic image coordinate system is parallel to the XY plane of the world coordinate system. Therefore, the affine transformation matrix from the world coordinate system to the panoramic image pixel coordinate system can be obtained:

[0159]

[0160] Step 209: Determine the monoimation matrix of each camera to the panoramic image coordinate system.

[0161] This step can also be implemented using existing methods.

[0162] Specifically, based on the principle of single-image matrix calculation, given the camera's intrinsic parameter matrix K and the transformation matrix from the camera to the world coordinate system... Given the coplanar equations to be projected (Z=0), the projection matrices of each camera's distorted image onto the ship's XY plane in the world coordinate system can be directly calculated:

[0163]

[0164] in, and Camera c i The first and second columns of the rotation matrix of the coordinate system relative to the world coordinate system, t i For camera c i The translation of the coordinate system relative to the world coordinate system.

[0165] The affine transformation matrix M from the world coordinate system to the panoramic image coordinate system and the projection matrix H cam i Multiplying the inverse matrices yields the single-image matrix from each camera coordinate system to the panoramic image coordinate system:

[0166]

[0167] Step 210: Project the images captured by each camera onto the panoramic image coordinate system using a single-image matrix, and stitch the projection results of each camera together to generate a panoramic image.

[0168] This step can be implemented using existing methods. Specifically, after obtaining the monoimation matrix, the images from each camera can be projected onto the target panoramic image coordinate system. The mapping relationship between the pixel coordinates of the panoramic image and the corresponding camera pixel coordinates can be solved using equation (27), p i For camera ci Pixel coordinates, p pano H represents the pixel coordinates of the target panoramic image. total i For this camera c i The mapping matrix is ​​obtained. After obtaining the mapping relationship, the colors of the camera image can be copied or interpolated into the pixels of the corresponding panoramic image to obtain the panoramic image to be fused. The central area of ​​the panoramic image is the ship itself, which cannot be observed by the camera module, so the ship's icon can be used to fill it.

[0169] p pano =∑H total i p i (27)

[0170] In the panoramic images to be fused, there are problems such as uneven brightness and color in the overlapping areas of the fields of view of adjacent cameras. Various existing methods such as weighted averaging, brightness equalization, color equalization, multi-band fusion or alpha fusion can be used to fuse the stitching seam area, reduce the impact of the stitching seam and improve the stitching effect. These methods will not be elaborated here.

[0171] At this point, Figure 2 The specific process of the ship panoramic image generation method in the illustrated embodiment is now complete. Using the ship panoramic image generation method described above, only the transformation matrix between each camera needs to be calibrated during offline calibration. During online stitching, the direction vector of gravity is estimated in real time based on the IMU installed in the camera module, and a world coordinate system is constructed in real time based on the gravity direction. The single-image matrix from each camera to the world coordinate system is then calculated, and finally, panoramic stitching can be achieved based on this single-image matrix.

[0172] Compared with existing panoramic image generation methods, the method in this application does not require finding a coplanar region to set up a calibration board or a high-precision positioning system during the offline calibration stage, thus solving the problem that it is difficult for large ships to find a stable calibration plane on the sea surface. At the same time, the single-image matrix from the camera to the sea surface can be adaptively estimated in real time, solving the problem of panoramic image stitching anomalies caused by the inconsistency between the ship's attitude and the calibration time due to wind, waves or uneven load after offline calibration in existing panoramic image generation schemes. It has stronger robustness on sea surfaces with large winds and waves.

[0173] The above describes the specific implementation of the ship panoramic image generation method in this application. This application also provides a ship panoramic image generation apparatus, which can be used to implement the above panoramic image generation method. Figure 9 This is a schematic diagram of the basic structure of the ship panoramic image generation device in this application. Multiple camera modules are installed around the ship's hull. Each camera module includes at least a camera, and at least one camera module also includes an IMU, such as... Figure 9As shown, the device includes: an inter-camera extrinsic parameter calibration unit, a gravity direction estimation unit, a transformation matrix calculation unit, a single-image matrix calculation unit, and a panoramic image generation unit;

[0174] Among them, the camera extrinsic calibration unit is used to determine the transformation matrix between every two adjacent camera coordinate systems;

[0175] The gravity direction estimation unit is used to determine the gravity direction vector in the reference IMU coordinate system through IMU-based gravity direction estimation, and to calculate the gravity direction vector in the reference camera coordinate system based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system. Among them, the reference IMU and the reference camera belong to the preset reference camera module;

[0176] The transformation matrix calculation unit is used to calculate the gravity direction vector based on the reference camera coordinate system. Calculate the rotation matrix from the reference camera coordinate system to the world coordinate system, considering the parallelism between the reference camera coordinate system and the Z-axis of the world coordinate system. The transformation matrix from the reference camera coordinate system to the world coordinate system is determined based on the rotation matrix and the translation t from the reference camera coordinate system to the world coordinate system. It is also used to calculate the transformation matrix from the coordinate system of other cameras (excluding the reference camera) to the world coordinate system based on the transformation matrix between every two adjacent camera coordinate systems and the transformation matrix from the reference camera coordinate system to the world coordinate system;

[0177] The mono-matrix calculation unit is used to determine the mono-matrix from each camera to the panoramic image coordinate system based on the transformation matrix from each camera coordinate system to the world coordinate system.

[0178] The panoramic image generation unit is used to project images captured by each camera onto the panoramic image coordinate system using a single-image matrix, and to stitch the projection results of each camera together to generate a panoramic image; wherein, the UV plane of the panoramic image coordinate system is parallel to the XY plane of the world coordinate system.

[0179] Optionally, in the rotation matrix calculation unit, the rotation matrix is ​​calculated. The ways in which the value can be obtained can specifically include:

[0180]

[0181] in, α is the preset yaw angle from the reference camera to the world coordinate system, and nx, ny, and nz are the gravity direction vectors in the reference camera coordinate system, respectively. The three components take values.

[0182] Optionally, when the reference camera is installed on the hull in the positive direction of the X-axis of the world coordinate system, the value of α can be -π / 2; when the reference camera is installed on the hull in the negative direction of the X-axis of the world coordinate system, the value of α is -3π / 2; when the reference camera is installed on the hull in the positive direction of the Y-axis of the world coordinate system, the value of α is 0; and when the reference camera is installed on the hull in the negative direction of the Y-axis of the world coordinate system, the value of α is -π.

[0183] Optionally, in the transformation matrix calculation unit, the method for determining the z-coordinate of the translation amount t may specifically include:

[0184] a. Select two adjacent cameras to capture sea level images of the area they share a common view of. Identify a pair of matching points corresponding to the same target point in the two sea level images. Based on the coordinates of the matching point pair in the coordinate systems of the two adjacent cameras and the transformation matrix between the coordinate systems of the two adjacent cameras and the reference camera, calculate the coordinates p of the target point in the reference camera coordinate system. ref ;

[0185] b. Determine the initial z-coordinate as z = -(R) 31 x ref +R 32 y ref +R 33 z ref ); where R 31 Rotation matrix The element in the 3rd row and 1st column, R 32 Rotation matrix The element in the 3rd row and 2nd column, R 33 Rotation matrix The element in the 3rd row and 3rd column;

[0186] c. Use the initial z-coordinate as the z-coordinate of the translation amount t, or repeat steps a and b to obtain several initial z-coordinates, take a weighted average of all the initial z-coordinates, and use the weighted average result as the z-coordinate of the translation amount t.

[0187] Optionally, the process of determining the gravity direction vector in the reference IMU coordinate system within the gravity direction estimation unit may specifically include:

[0188] Gravity direction estimation is performed using a reference IMU to obtain the gravity direction vector in the reference IMU coordinate system; or...

[0189] In addition to the reference camera module, at least one camera module includes an IMU. Gravity direction is estimated using each IMU. Based on the transformation matrix between every two adjacent camera coordinate systems, the gravity direction vector estimated by each IMU is transformed to the reference IMU coordinate system. The gravity direction vector in the reference IMU coordinate system is determined based on the transformed gravity direction vectors.

[0190] Optionally, the method for determining the transformation matrix between every two adjacent camera coordinate systems in the inter-camera extrinsic calibration unit may specifically include:

[0191] For every two adjacent cameras, corner detection is performed based on the images acquired by the adjacent cameras for the same calibration board. The corner detection results of the adjacent cameras are matched to obtain multiple pairs of detected corner coordinates. in, This represents the detection result of the diagonal point k of the first camera among adjacent cameras. The detection result of the diagonal point k of the second camera in the adjacent cameras, where k≤n, and n is the total number of detected coordinate pairs;

[0192] Based on the detection coordinate pairs of multiple corner points, determine the relationship between the detection coordinate pairs that best approximates the first functional relationship. Transformation matrix As the initial transformation matrix between every two adjacent camera coordinate systems

[0193] All initial transformation matrices are optimized as a whole using a pose graph optimization method, and the optimized initial transformation matrices are used as the transformation matrices between two adjacent camera coordinate systems.

[0194] Optionally, the process of determining the single-image matrix of each camera to the panoramic image coordinate system in the single-image matrix calculation unit may specifically include:

[0195] Calculate the projection matrix of the distorted image of each camera onto the XY plane of the world coordinate system based on the transformation matrix from each camera coordinate system to the world coordinate system, the intrinsic parameter matrix of the camera, and the distortion coefficient.

[0196] Multiply the inverse of the affine transformation matrix from the world coordinate system to the panoramic image coordinate system with the projection matrix to obtain the single-image matrix from each camera to the panoramic image coordinate system.

[0197] This application also provides a computer-readable storage medium that stores instructions, which, when executed by a processor, can perform the steps in the ship panoramic image generation method described above. In practical applications, the computer-readable medium may be included in the devices / apparatus / systems of the above embodiments, or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium stores instructions, which, when executed by a processor, can perform the steps in the ship panoramic image generation method described above.

[0198] According to the embodiments disclosed in this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof, but not intended to limit the scope of protection of this application. In the embodiments disclosed in this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0199] Figure 10 An electronic device is also provided for this application. For example... Figure 10 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0200] The electronic device may include a processor 1001 with one or more processing cores, a memory 1002 with one or more computer-readable storage media, and a computer program stored in the memory and executable on the processor. When the program in the memory 1002 is executed, a method for generating panoramic images of a ship can be implemented.

[0201] Specifically, in practical applications, this electronic device may also include components such as a power supply 1003 and an input / output unit 1004. Those skilled in the art will understand that... Figure 10 The structure of the electronic device shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0202] The processor 1001 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 1002, and calling data stored in the memory 1002, it performs various functions of the server and processes data, thereby monitoring the electronic device as a whole.

[0203] The memory 1002 can be used to store software programs and modules, i.e., the aforementioned computer-readable storage medium. The processor 1001 executes various functional applications and data processing by running the software programs and modules stored in the memory 1002. The memory 1002 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function, etc.; the data storage area may store data created based on server usage, etc. Furthermore, the memory 1002 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 1002 may also include a memory controller to provide the processor 1001 with access to the memory 1002.

[0204] The electronic device also includes a power supply 1003 that supplies power to the various components. This power supply can be logically connected to the processor 1001 via a power management system, thereby enabling functions such as charging, discharging, and power consumption management. The power supply 1003 may also include one or more DC or AC power supplies, a recharging system, a power fault detection circuit, a power converter or inverter, a power status indicator, or any other components.

[0205] The electronic device may also include an input / output unit 1004, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, and optical signal inputs related to user settings and function control. The input unit output 1004 can also be used to display information input by the user or information provided to the user, as well as various graphical user interfaces, which can be composed of graphics, text, icons, video, and any combination thereof.

[0206] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for generating panoramic images of ships, characterized in that, Multiple camera modules are installed around the hull, each camera module including at least a camera, and at least one camera module also including an IMU. The method includes: Determine the transformation matrix between every two adjacent camera coordinate systems; Gravity direction estimation based on the IMU is used to determine the gravity direction vector in the reference IMU coordinate system. Then, based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system, the gravity direction vector in the reference camera coordinate system is calculated. The reference IMU and the reference camera belong to a preset reference camera module; Based on the gravity direction vector in the reference camera coordinate system Calculate the parallel relationship between the reference camera coordinate system and the Z-axis of the world coordinate system, and the rotation matrix from the reference camera coordinate system to the world coordinate system. The transformation matrix from the reference camera coordinate system to the world coordinate system is determined based on the rotation matrix and the translation t from the reference camera coordinate system to the world coordinate system. Based on the transformation matrix between every two adjacent camera coordinate systems and the transformation matrix from the reference camera coordinate system to the world coordinate system, calculate the transformation matrix from the coordinate systems of other cameras besides the reference camera to the world coordinate system; Based on the transformation matrix from each camera coordinate system to the world coordinate system, the single-image matrix from each camera to the panoramic image coordinate system is determined, and the images captured by each camera are projected onto the panoramic image coordinate system using the single-image matrix. The projection results of each camera are then stitched together to generate a panoramic image. The UV plane of the panoramic image coordinate system is parallel to the XY plane of the world coordinate system.

2. The method according to claim 1, characterized in that, The calculation of the rotation matrix The possible values ​​include: in, α is the preset yaw angle from the reference camera to the world coordinate system, and nx, ny, and nz are the gravity direction vectors in the reference camera coordinate system, respectively. The three components take values.

3. The method according to claim 2, characterized in that, When the reference camera is installed on the hull in the positive direction of the X-axis of the world coordinate system, the value of α is -π / 2; when the reference camera is installed on the hull in the opposite direction of the X-axis of the world coordinate system, the value of α is -3π / 2; when the reference camera is installed on the hull in the positive direction of the Y-axis of the world coordinate system, the value of α is 0; and when the reference camera is installed on the hull in the opposite direction of the Y-axis of the world coordinate system, the value of α is -π.

4. The method according to any one of claims 1 to 3, characterized in that, The methods for determining the z-coordinate of the translation amount t include: a. Select two adjacent cameras to capture sea level images of the area shared by the two adjacent cameras. Identify a pair of matching points corresponding to the same target point in the two sea level images. Based on the coordinates of the matching point pair in the coordinate systems of the two adjacent cameras and the transformation matrix between the coordinate systems of the two adjacent cameras and the reference camera, calculate the coordinates p of the target point in the reference camera coordinate system. ref ; b. Determine the initial z-coordinate as z = -(R) 31 x ref +R 32 y ref +R 33 z ref ); where R 31 For the rotation matrix The element in the 3rd row and 1st column, R 32 For the rotation matrix The element in the 3rd row and 2nd column, R 33 For the rotation matrix The element in the 3rd row and 3rd column; c. Use the initial z-coordinate as the z-coordinate of the translation amount t, or, repeat steps a and b to obtain several initial z-coordinates, perform a weighted average of all initial z-coordinates, and use the weighted average result as the z-coordinate of the translation amount t.

5. The method according to claim 1, characterized in that, Determining the gravity direction vector in the reference IMU coordinate system includes: Gravity direction estimation is performed using a reference IMU to obtain the gravity direction vector in the reference IMU coordinate system; or... In addition to the reference camera module, at least one camera module includes an IMU, which performs gravity direction estimation using each IMU, and transforms the gravity direction vector estimated by each IMU to the reference IMU coordinate system based on the transformation matrix between every two adjacent camera coordinate systems, and determines the gravity direction vector in the reference IMU coordinate system based on the transformed gravity direction vectors.

6. The method according to claim 1, characterized in that, The methods for determining the transformation matrix between every two adjacent camera coordinate systems include: For every two adjacent cameras, corner detection is performed based on the images acquired by the adjacent cameras for the same calibration board. The corner detection results of the adjacent cameras are then matched to obtain multiple pairs of detected corner coordinates. Among them, the The detection result of the diagonal point k of the first camera among the adjacent cameras, the The detection result of the diagonal point k of the second camera in the adjacent cameras, where k≤n, and n is the total number of detected coordinate pairs; Based on the detected coordinate pairs of the multiple corner points, determine the relationship between the detected coordinate pairs that most closely approximates the first functional relationship. Transformation matrix As the initial transformation matrix between every two adjacent camera coordinate systems All the initial transformation matrices are optimized as a whole using the pose graph optimization method, and the optimized initial transformation matrices are used as the transformation matrices between the corresponding two adjacent camera coordinate systems.

7. The method according to claim 1, characterized in that, Determining the single-image matrix of each camera to the panoramic image coordinate system includes: Based on the transformation matrix from each camera coordinate system to the world coordinate system, the intrinsic parameter matrix of the camera, and the distortion coefficient, calculate the projection matrix of the distorted image of each camera onto the XY plane of the world coordinate system. Multiply the affine transformation matrix from the world coordinate system to the panoramic image coordinate system by the inverse of the projection matrix to obtain the single-image matrix from each camera to the panoramic image coordinate system.

8. A device for generating panoramic images of ships, characterized in that, Multiple camera modules are installed around the ship's hull. Each camera module includes at least one camera, and at least one camera module also includes an IMU. The device includes: a camera extrinsic parameter calibration unit, a gravity direction estimation unit, a transformation matrix calculation unit, a single-image matrix calculation unit, and a panoramic image generation unit. The inter-camera extrinsic calibration unit is used to determine the transformation matrix between every two adjacent camera coordinate systems; The gravity direction estimation unit is used to determine the gravity direction vector in the reference IMU coordinate system through IMU-based gravity direction estimation, and to calculate the gravity direction vector in the reference camera coordinate system based on the transformation matrix from the reference IMU coordinate system to the reference camera coordinate system. The reference IMU and the reference camera belong to a preset reference camera module; The transformation matrix calculation unit is used to calculate the gravity direction vector in the reference camera coordinate system. Calculate the parallel relationship between the reference camera coordinate system and the Z-axis of the world coordinate system, and the rotation matrix from the reference camera coordinate system to the world coordinate system. The transformation matrix from the reference camera coordinate system to the world coordinate system is determined based on the rotation matrix and the translation t from the reference camera coordinate system to the world coordinate system. It is also used to calculate the transformation matrix from the coordinate systems of other cameras besides the reference camera to the world coordinate system based on the transformation matrix between every two adjacent camera coordinate systems and the transformation matrix from the reference camera coordinate system to the world coordinate system; The mono-matrix calculation unit is used to determine the mono-matrix from each camera to the panoramic image coordinate system based on the transformation matrix from each camera coordinate system to the world coordinate system. The panoramic image generation unit is used to project images captured by each camera onto the panoramic image coordinate system using the single-image matrix, and to stitch the projection results of each camera together to generate a panoramic image; wherein, the UV plane of the panoramic image coordinate system is parallel to the XY plane of the world coordinate system.

9. A computer-readable storage medium storing computer instructions thereon, characterized in that, When the instruction is executed by the processor, it can realize the method for generating panoramic images of ships according to any one of claims 1 to 7.

10. An electronic device, characterized in that, The electronic device includes at least a computer-readable storage medium and a processor; The processor is configured to read executable instructions from the computer-readable storage medium and execute the instructions to implement the ship panoramic image generation method according to any one of claims 1 to 7.

Images