Platform situation awareness sensor group online calibration method based on virtual-real fusion

Abstract

The invention discloses an online calibration method for a platform situation awareness sensor group based on virtual-real fusion, and relates to the technical field of digital twinning, and the method comprises the steps: constructing an equipment model and a sensor model of an ocean platform with a calibration requirement in a virtual space, and assembling the models according to the actual installation position of the platform; off-line calibration is carried out on a sensor group of the ocean platform to obtain a spatial transformation matrix from the sensors to the platform and between the sensors, and the spatial transformation matrix is applied to a model of a virtual space; for a certain known obstacle or target, real sensing data of the sensor group of the ocean platform on the obstacle or target is compared with virtual sensing data of the sensor model of the virtual space on the obstacle or target; and correcting the spatial transformation matrix based on a comparison result, and applying to an ocean platform. According to the method, whether the sensor in physical reality has position change and performance degradation or not can be judged, so that high-precision operation or combat requirements are met.

Description

Technical Field

[0001] The present invention relates to the field of digital twin technology, and is aimed at the space and performance calibration requirements of intelligent and autonomous marine equipment, and in particular to an online calibration method for a platform situation awareness sensor group based on virtual-reality fusion. Background Art

[0002] In order to cope with the complex and changing external environment and obtain stable and reliable situational awareness capabilities, some new marine equipment, such as USV, UUV, etc., as well as some traditional marine platforms that are undergoing intelligent transformation, usually install various types of situational awareness sensors, such as radar, lidar, infrared optoelectronics, forward-looking sonar, etc. These sensors are based on different working principles and have different advantages and disadvantages, so they need to be used in an integrated manner to maximize their strengths and avoid their weaknesses.

[0003] However, in terms of maritime applications, not only will harsh environmental factors such as high temperature, high salt, and high humidity cause the performance of the sensor itself to degrade, but also vibration, slamming, and large-scale shaking of the ship will cause changes in the sensor's installation position and angle. The above two reasons will lead to perception errors, which in turn affect the accuracy of target positioning and affect the performance of maritime missions or operations. Summary of the invention

[0004] In view of the above problems and technical requirements, the inventors have proposed an online calibration method for a platform situation awareness sensor group based on virtual-real fusion, which can avoid the perception error caused by performance degradation or installation position change of sensors and sensor groups during application. The technical solution of the present invention is as follows:

[0005] An online calibration method for platform situation awareness sensor groups based on virtual-reality fusion is aimed at the calibration needs of intelligent or autonomous marine equipment situation awareness sensor groups. Based on the interaction between physical reality and virtual space, it realizes the online dynamic spatial calibration of the sensor group of the marine platform in physical reality, supporting high-precision operation or combat needs, such as target positioning. The method includes the following steps:

[0006] For offshore platforms that require calibration, build the equipment model and sensor model of the platform in virtual space, and assemble the model according to the actual installation location of the platform;

[0007] Offline calibration of the sensor group of the ocean platform is performed to obtain the spatial transformation matrix from sensor to platform and between sensors, and applied to the model of virtual space;

[0008] For a known obstacle or target, the real perception data of the obstacle or target by the sensor group of the ocean platform is compared with the virtual perception data of the obstacle or target by the sensor model of the virtual space;

[0009] Modify the spatial transformation matrix based on the comparison result and apply it to the offshore platform.

[0010] A further technical solution thereof is to perform offline calibration on the sensor group of the offshore platform to obtain the spatial transformation matrices between the sensors and the platform and between the sensors, including:

[0011] Before the application of the offshore platform, perform offline calibration on the external parameters of the platform sensors;

[0012] Taking a point on the platform as the measurement point, knowing the actual installation positions of individual sensors and the relative positions between the sensors, and calculating the spatial transformation matrices between the sensors and the platform and between the sensors in combination with the external parameters of the sensors.

[0013] A further technical solution thereof is that assuming the sensor group includes a camera and a radar, then performing offline calibration on the external parameters of the platform sensors includes:

[0014] Place a calibration object within the monitoring ranges of the camera and the radar, and adjust the camera and the radar to have a good co-visual field;

[0015] Collect camera images containing the calibration object and radar point cloud data with the same time stamp, and obtain the coordinate transformation matrix between the camera and the radar through 2D-3D non-linear coordinate transformation, including translation and rotation matrices.

[0016] A further technical solution thereof is that placing a calibration object within the monitoring ranges of the camera and the radar and adjusting the camera and the radar to have a good co-visual field includes:

[0017] Adjust the camera field of view to a suitable position in the specified environment, adjust the image zoom to ensure a good co-visual field with the radar, and ensure that the area occupied by the calibration checkerboard calibration board in the camera image is greater than the specified threshold.

[0018] A further technical solution thereof is that the method for the sensor model in the virtual space to obtain virtual perception data of an obstacle or a target includes:

[0019] During the application of the offshore platform, for a certain obstacle or target, if its longitude and latitude coordinates are known, then in combination with the real perception data of the obstacle or target by the sensor group of the offshore platform, construct a virtual obstacle or target with the same characteristics in the virtual space and configure it in the virtual environment according to the longitude and latitude coordinates, so as to obtain the virtual perception data of the obstacle or target by the sensor model in the virtual space.

[0020] A further technical solution thereof is that modifying the spatial transformation matrix based on the comparison result includes:

[0021] Utilize the characteristic that the physical properties of the equipment in the virtual space do not change over time, compare the real perception data and virtual perception data of a single sensor. If the data deviation exceeds the specified threshold, it is determined that the spatial transformation matrix from the sensor to the platform on the platform has changed;

[0022] Adjust the spatial position of the sensor in the virtual space according to the real perception data, recalculate the spatial transformation matrix from the sensor to the platform, and apply it to the offshore platform.

[0023] A further technical solution thereof is that, based on the comparison result to correct the spatial transformation matrix, it further includes:

[0024] If the deviations between the real perception data and virtual perception data of all sensors exceed the specified threshold and the spatial transformation matrix between the sensors remains unchanged, it is considered that the navigation obstacle or target has changed in the physical world;

[0025] Re-select the navigation obstacle or target, and for this navigation obstacle or target, obtain the corresponding real perception data and virtual perception data, and re-perform the steps of comparison and correction.

[0026] A further technical solution thereof is that a sensor model of the platform is constructed in the virtual space, including:

[0027] Construct a 3D visualization model and physical mechanism model for each sensor, where:

[0028] For the 3D visualization model of the sensor, it is required to be installed at the same position as the corresponding physical sensor on the platform to provide the positional relationship between the sensor and the navigation obstacle or target;

[0029] The physical mechanism model of the sensor has the same functions and parameters as the corresponding physical sensor on the platform, and provides virtual perception data under the relative spatial position.

[0030] The beneficial technical effects of the present invention are:

[0031] Based on the one-to-one correspondence between the virtual space and the physical platform, and the characteristic that the models constructed in the virtual space do not undergo relative position migration due to aging over time, this method can perform online, dynamic, and on-demand calibration on the physical sensors and between sensor groups during the actual application process of the offshore platform. By comparing the virtual and real perception results of the sensors in the physical reality and the virtual space, it is possible to determine whether there are position changes, performance degradation, etc. in the sensors in the physical reality, and correct the calibrated spatial transformation matrix according to the comparison situation. The offshore platform equipped with this method does not need to go to a designated location or be disassembled and sent to a designated place for calibration during the mission. On the one hand, it improves the combat and operation accuracy, on the other hand, it reduces the calibration cost and improves the equipment utilization rate. Description of the Drawings

[0032] Figure 1 It is a flowchart of an online calibration method for a platform situation awareness sensor group based on virtual-real fusion provided by this application. Specific implementation manners

[0033] The following further describes the specific implementation manners of the present invention with reference to the accompanying drawings.

[0034] An embodiment of this application provides an online calibration method for a platform situation awareness sensor group based on virtual-real fusion. Please refer to Figure 1 As shown, this method specifically includes the following contents:

[0035] Step 1: For an ocean platform with a calibration requirement, construct an equipment model and a sensor model of the platform in the virtual space. The specific implementation method of this step includes:

[0036] According to the ocean platform with a calibration requirement and the physical sensor group thereon, construct a typical application environment of the platform in the virtual space, and load the constructed high-precision models in this environment, including 3D visualization models of the platform equipment and each sensor, and physical mechanism models of each sensor. The physical mechanism model of the sensor has the same functions and parameters as the corresponding physical sensor of the platform, and can provide virtual perception data under the relative spatial position. Taking a radar as an example, its physical mechanism model should be able to implement radar transmit-receive signals in the virtual electromagnetic space, and consider factors such as the radar cross section RCS of the target (affected by target material, geometric size, radar viewing angle, and radar operating wavelength), target movement, and antenna pattern, and simulate and generate original echoes and radar images under ideal weather conditions.

[0037] Step 2: According to the actual installation positions of the platform sensors, complete the configuration of the 3D visualization models of each sensor on the platform equipment model, including that the 3D visualization model of the sensor should be installed at the same position as the corresponding physical sensor of the platform, and have the same functions, parameters, etc.

[0038] Step 3: Perform offline calibration on the sensor group of the ocean platform. The specific implementation method of this step includes:

[0039] First, before applying to the offshore platform, the extrinsic parameters of the platform sensors are calibrated offline. Assuming the sensor group includes a camera and a radar, the extrinsic parameters of the sensors include the correspondence between camera pixels and the physical space, the correspondence between radar point clouds and the physical space, and the correspondence between camera pixels and radar point clouds. Taking the joint calibration of the camera and the radar as an example, obtaining the correspondence between the radar point cloud and the physical space includes: (1) Placing calibration objects within the monitoring range of the camera and the radar, and adjusting the camera and the radar to have a good common view. Specifically, in an environment with good lighting and less occlusion, adjust the camera's field of view to an appropriate position, and adjust the image zoom to ensure a good common view with the radar and ensure that the calibration checkerboard calibration board occupies a sufficiently large area in the camera image. (2) Collecting camera images and radar point cloud data containing calibration objects with the same timestamp, and obtaining the coordinate transformation matrix between the camera and the radar through 2D-3D non-linear coordinate transformation, including translation and rotation matrices. (3) Setting the perception parameters of the physical mechanism model of the virtual sensor according to the obtained extrinsic parameters of the sensor to obtain a perception effect matching that of the real sensor.

[0040] Secondly, taking a point on the platform as the measurement point, given the actual installation positions of individual sensors and the relative positions between sensors, the spatial transformation matrices between the sensors and the platform and between the sensors are calculated in combination with the extrinsic parameters of the sensors and applied to the sensor model in the virtual space.

[0041] Step Four: For a known navigation obstacle or target, obtain the virtual perception data of the navigation obstacle or target using the sensor model in the virtual space. The specific implementation method of this step includes:

[0042] During the actual application process of the offshore platform, for a certain navigation obstacle or target, if its longitude and latitude coordinates are known (obtained through electronic nautical charts or AIS, etc.), then combining the real perception data of the navigation obstacle or target by the sensor group of the offshore platform, a virtual navigation obstacle or target with the same characteristics is constructed in the virtual space and configured in the virtual environment according to the longitude and latitude coordinates, so as to obtain the virtual perception data of the navigation obstacle or target by the sensor model in the virtual space. Among them, the 3D visualization model of the sensor provides the positional relationship between the sensor and the navigation obstacle or target, and the physical mechanism model of the sensor realizes the perception data under the relative spatial position. For example, the perception data of the radar is an echo image, and the perception data of the camera is a visual image.

[0043] Step Five: Compare the real perception data of the sensor group of the offshore platform for the selected navigation obstacle or target with the virtual perception data of the sensor model in the virtual space for the same navigation obstacle or target. Based on the comparison results, correct the spatial transformation matrix and apply it to the offshore platform, so as to correct the perception errors of the sensors in the real physical world caused by installation positions, angles or environmental factors.

[0044] In this embodiment, the correction principle is to use the characteristic that the physical properties of equipment in the virtual space do not change over time. First, the real perception data and virtual perception data of a single sensor are compared. If the selected obstacle or target has not changed its position in the physical world, the measurement results of its position by each sensor should be consistent with the measurement results of its corresponding virtual sensor in the virtual space; if the data deviation exceeds the specified threshold, it is determined that the spatial transformation matrix from the sensor to the platform on the physical platform has changed. Then, the spatial position of the sensor in the virtual space is adjusted according to the real perception data, for example, the deviated virtual sensor is translated a certain distance in the virtual space, or rotated a certain angle, etc., and the spatial transformation matrix from the sensor to the platform is recalculated and applied to the marine platform.

[0045] In addition, if the deviations between the real perception data and the virtual perception data of all sensors exceed the specified threshold and the spatial transformation matrix between the sensors remains unchanged, it is considered that the selected obstacle or target has changed in the physical world, and it is necessary to reselect the obstacle or target, and obtain the corresponding real perception data and virtual perception data for this obstacle or target, and re-perform the comparison and correction steps, that is, repeat the above steps four and five to complete the online calibration of the sensor group.

[0046] Step 6: Apply the corrected spatial transformation matrix to sensor data processing to ensure the reliability of the perception results. Subsequently, the platform sensor group can be calibrated online based on a given time period and in combination with certain working conditions to ensure the reliability of the perception data at all times.

[0047] The above is only a preferred embodiment of the present application, and the present invention is not limited to the above embodiments. It is understood that other improvements and changes directly derived or associated by those skilled in the art without departing from the spirit and concept of the present invention should be considered to be included in the protection scope of the present invention.

Claims

1. An online calibration method for a platform situation awareness sensor group based on virtual-real fusion, characterized in that: The method comprises: For offshore platforms that require calibration, build the equipment model and sensor model of the platform in virtual space, and assemble the model according to the actual installation location of the platform; Offline calibration of the sensor group of the ocean platform is performed to obtain the spatial transformation matrix from sensor to platform and between sensors, and applied to the model of virtual space; For a known obstacle or target, comparing the real perception data of the obstacle or target by the sensor group of the marine platform with the virtual perception data of the obstacle or target by the sensor model of the virtual space; The spatial transformation matrix is ​​modified based on the comparison result and applied to the marine platform.

2. The online calibration method of the platform situation awareness sensor group based on virtual-real fusion according to claim 1 is characterized in that: The sensor group of the ocean platform is calibrated offline to obtain a spatial transformation matrix from the sensor to the platform and between the sensors, including: Before using on offshore platforms, the external parameters of platform sensors are calibrated offline. Taking a point on the platform as the measurement point, the actual installation position of a single sensor and the relative position between sensors are known. The spatial transformation matrix from sensor to platform and between sensors is calculated by combining the sensor external parameters.

3. The online calibration method of the platform situation awareness sensor group based on virtual-real fusion according to claim 2 is characterized in that: Assuming that the sensor group includes a camera and a radar, the offline calibration of the external parameters of the platform sensor includes: Place calibration objects within the monitoring range of the camera and radar, and adjust the camera and radar to have a good common view; The camera image and radar point cloud data containing the calibration object are collected at the same timestamp, and the coordinate transformation matrix of the camera and radar is obtained through 2D-3D nonlinear coordinate transformation, including translation and rotation matrices.

4. The online calibration method of the platform situation awareness sensor group based on virtual-real fusion according to claim 3 is characterized in that: The process of placing a calibration object within the monitoring range of the camera and radar and adjusting the camera and radar to have a good common view includes: Under the specified environment, adjust the camera field of view to a suitable position, adjust the image magnification to ensure a good common field of view with the radar, and ensure that the area occupied by the calibration checkerboard calibration plate in the camera image is larger than the specified threshold.

5. The online calibration method of platform situation awareness sensor group based on virtual-real fusion according to claim 1 is characterized in that: The method for the sensor model of the virtual space to obtain the virtual perception data of the obstacle or target includes: In the application process of the marine platform, for a certain obstacle or target, if its longitude and latitude coordinates are known, the real perception data of the obstacle or target by the sensor group of the marine platform is combined to construct a virtual obstacle or target with the same characteristics in the virtual space, and configure it in the virtual environment according to the longitude and latitude coordinates, so as to obtain the virtual perception data of the obstacle or target by the sensor model in the virtual space.

6. The online calibration method of platform situation awareness sensor group based on virtual-real fusion according to claim 1 is characterized in that: The modifying the spatial transformation matrix based on the comparison result includes: By using the fact that the physical properties of equipment in virtual space do not change over time, the real perception data and virtual perception data of a single sensor are compared. If the data deviation exceeds a specified threshold, it is determined that the spatial transformation matrix from the sensor to the platform on the platform has changed. The spatial position of the sensor in the virtual space is adjusted according to the real perception data, and the spatial transformation matrix from the sensor to the platform is recalculated and applied to the marine platform.

7. The online calibration method of platform situation awareness sensor group based on virtual-real fusion according to claim 6 is characterized in that: The step of correcting the spatial transformation matrix based on the comparison result further includes: If the deviations between the real perception data and the virtual perception data of all sensors exceed the specified threshold, and the spatial transformation matrix between the sensors remains unchanged, it is considered that the obstacle or target has changed in the physical world; Reselect the obstacle or target, obtain the corresponding real perception data and virtual perception data for the obstacle or target, and re-perform the comparison and correction steps.

8. The online calibration method for platform situation awareness sensor group based on virtual-real fusion according to claim 1 is characterized in that: Construct the sensor model of the platform in the virtual space, including: Construct a 3D visualization model and physical mechanism model for each sensor, including: The 3D visualization model of the sensor is required to be installed at the same position as the corresponding physical sensor of the platform to provide the positional relationship between the sensor and the obstacle or target; The physical mechanism model of the sensor has the same functions and parameters as the corresponding physical sensor of the platform, and provides virtual perception data in relative spatial position.

Images