External parameter calibration method, device, equipment and medium

Abstract

The invention provides an external parameter calibration method, device, equipment and medium, and the method comprises the steps: extracting the two-dimensional position measurement data of a plurality of target feature points of a 360-degree panoramic image shot by a panoramic camera, and enabling the distortion degree of the image data of each target feature point to be smaller than a preset distortion degree; and determining external parameters between the panorama camera and the IMU based on the two-dimensional position measurement data of the plurality of target feature points, the three-dimensional position data of the plurality of target feature points, the internal parameters of the panorama camera and the pose of the IMU. According to the method, the calibration accuracy is high, and the 360-degree VIO precision based on the 360-degree panoramic image can be improved.

Description

Technical Field

[0001] The present invention relates to the field of panoramic camera computer vision technology, and in particular to an extrinsic parameter calibration method, device, equipment and medium. Background Art

[0002] An inertial measurement unit (IMU) can be a device that measures the three-axis attitude angle (or angular rate) and acceleration of an object. It can be combined with a panoramic camera to form a panoramic camera and IMU fusion system, which can be used to obtain rich visual and motion information.

[0003] In the related technology, it is necessary to use the external parameters between the panoramic camera and the IMU to fuse the visual information and motion information. However, there is currently a lack of external parameter calibration methods for panoramic cameras and IMUs. Therefore, there is an urgent need for an external parameter calibration technology for panoramic cameras and IMUs. Summary of the invention

[0004] According to one aspect of the present disclosure, a method for calibrating an external parameter is provided, comprising:

[0005] Extracting two-dimensional position measurement data of a plurality of target feature points of a 360° panoramic image captured by a panoramic camera, wherein the degree of distortion of the image data of each of the target feature points is less than a preset degree of distortion;

[0006] Based on the two-dimensional position measurement data of the plurality of target feature points, the three-dimensional position data of the plurality of target feature points, the intrinsic parameters of the panoramic camera and the position and posture of the IMU, the extrinsic parameters between the panoramic camera and the IMU are determined.

[0007] According to another aspect of the present disclosure, there is provided an external parameter calibration device, comprising:

[0008] An extraction module, used to extract two-dimensional position measurement data of multiple target feature points of a 360° panoramic image taken by a panoramic camera, wherein the degree of distortion of the image data of each target feature point is less than a preset degree of distortion;

[0009] The determination module is used to determine the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of the multiple target feature points, the three-dimensional position data of the multiple target feature points, the internal parameters of the panoramic camera and the position and posture of the IMU.

[0010] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0011] processor; and,

[0012] Memory for storing programs,

[0013] The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to the exemplary embodiment of the present disclosure.

[0014] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the method according to the exemplary embodiments of the present disclosure.

[0015] One or more technical solutions provided in the exemplary embodiments of the present disclosure can extract the two-dimensional position measurement data of multiple target feature points of the 360° panoramic image taken by the panoramic camera to control the degree of distortion of the image data of each target feature point to be less than a preset degree of distortion, and then determine the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of the multiple target feature points, the three-dimensional position data of the multiple target feature points, the internal parameters of the panoramic camera, and the position and posture of the IMU. This calibration method has high calibration accuracy and can improve the accuracy of 360-VIO based on 360° panoramic images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0017] Figure 1 A schematic diagram showing a flow chart of an external parameter calibration method of an exemplary embodiment of the present disclosure is shown;

[0018] Figure 2 A schematic diagram of an external parameter determination process between a panoramic camera and an IMU according to an exemplary embodiment of the present disclosure is shown;

[0019] Figure 3 A schematic diagram showing the principle of two-dimensional position correction of target feature points of an exemplary embodiment of the present disclosure is shown;

[0020] Figure 4 A schematic diagram of a process for extracting two-dimensional position measurement data of target feature points according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 5 A schematic diagram of a feature point extraction model architecture for an exemplary implementation of the present disclosure is shown;

[0022] Figure 6 A schematic block diagram of functional modules of an external parameter calibration device according to an exemplary embodiment of the present disclosure is shown;

[0023] Figure 7 A schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown;

[0024] Figure 8A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] Before introducing the embodiments of the present disclosure, the following definitions are given for the terms involved in the embodiments of the present disclosure:

[0030] The visual-inertial system (VINS), also known as visual-inertial odometry (VIO), integrates camera and IMU data to implement the Simultaneous Localization and Mapping (SLAM) algorithm.

[0031] Sub-pixel convolution is a clever method to improve the resolution of images and feature maps, also known as pixel shuffle.

[0032] Equidistant cylindrical projection is a simple map projection method. In this projection method, it is assumed that the sphere and the cylinder are tangent at the equator, and the longitude and latitude lines on the sphere are projected onto the cylindrical surface, and then unfolded into a plane along a generatrix of the cylindrical surface.

[0033] An exemplary embodiment of the present disclosure provides an extrinsic parameter calibration method, which can be used to calibrate the extrinsic parameters between a panoramic camera and an IMU to improve the accuracy of the extrinsic parameter calibration between the panoramic camera and the IMU, and improve the 360-VIO accuracy based on a 360° panoramic image.

[0034] In practical applications, the method of the exemplary embodiments of the present disclosure can be applied to an electronic device or a chip in an electronic device, and the electronic device can be various electronic devices with computing functions, such as a server or a terminal device.

[0035] The mobile station of the exemplary embodiment of the present disclosure may include a terminal device, which may be an electronic device with a display function, such as a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and a wearable device based on augmented reality (AR) and / or virtual reality (VR) technology.

[0036] Exemplarily, when the terminal device is a wearable device, the wearable device can also be a general term for wearable devices that are intelligently designed and developed using wearable technology for daily wear, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothes or accessories. Wearable devices are not just hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are fully functional, large in size, and can achieve complete or partial functions without relying on smartphones, such as smart watches or smart glasses, as well as those that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0037] Figure 1 FIG. 2 is a flow chart showing an external parameter calibration method according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the external parameter calibration method of the exemplary embodiment of the present disclosure may include:

[0038] Step 101: extracting two-dimensional position measurement data of multiple target feature points of a 360° panoramic image taken by a panoramic camera, wherein the distortion degree of the image data of each target feature point is less than a preset distortion degree.

[0039] The exemplary embodiments of the present disclosure can extract two-dimensional position measurement data of multiple target feature points from a 360° panoramic image through various possible feature point extraction algorithms. These feature point extraction algorithms can be color-based feature extraction algorithms, texture-based feature extraction algorithms, or image processing based on deep neural networks.

[0040] Taking into account the large distortion of 360° panoramic images, the exemplary embodiment of the present disclosure can constrain the degree of distortion of the image data of the extracted target feature points, ensure that the degree of distortion of the image data of multiple target feature points obtained is less than the preset degree of distortion, and eliminate the influence of the target feature points with large distortion on the accuracy of the external parameter correction between the panoramic camera and the IMU.

[0041] Step 102: Determine the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of the multiple target feature points, the three-dimensional position data of the multiple target feature points, the internal parameters of the panoramic camera, and the position and posture of the IMU.

[0042] In the exemplary embodiment of the present disclosure, the correspondence between the three-dimensional position measurement data and the two-dimensional position measurement data of the same target feature point may be known or unknown. When the correspondence between the three-dimensional position measurement data and the two-dimensional position measurement data of the same target feature point is unknown, but the three-dimensional position measurement data and the two-dimensional position measurement data of the target feature point are known, the correspondence between the three-dimensional position measurement data and the two-dimensional position measurement data of the same target feature point may be solved by a PnP algorithm, an SVD solution, or the like.

[0043] In practical applications, when the exemplary embodiment of the present disclosure determines the external parameters between the panoramic camera and the IMU, the reprojection method can be used for solution. Considering the trigger delay and transmission delay between the panoramic camera and the IMU, which causes the acquisition timestamp of the panoramic camera and the sampling time of the IMU to not match, resulting in a data time difference between the camera and the IMU, the external parameters between the panoramic camera and the IMU in the exemplary embodiment of the present disclosure include not only the posture transfer relationship between the panoramic camera and the IMU, but also the time difference between the panoramic camera and the IMU.

[0044] Based on this, Figure 2 FIG. 2 shows a schematic diagram of an external parameter determination process between a panoramic camera and an IMU according to an exemplary embodiment of the present disclosure. Figure 2As shown, the exemplary embodiment of the present disclosure determines the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of multiple target feature points, the three-dimensional position data of multiple target feature points, the internal parameters of the panoramic camera, and the position and posture of the IMU, which may include:

[0045] Step 201: Determine two-dimensional position correction data of the target feature point based on the two-dimensional position measurement data of the target feature point, the moving speed of the panoramic camera, and the time offset between the panoramic camera and the IMU.

[0046] In practical applications, the data obtained by the IMU at the same time and the 360° panoramic image taken by the panoramic camera do not correspond to each other due to the time difference between the panoramic camera and the IMU. Based on this, when reprojecting, the two-dimensional position measurement data of the target feature point can be corrected based on the time difference between the panoramic camera and the IMU.

[0047] Figure 3 FIG. 2 shows a schematic diagram of the principle of two-dimensional position correction of target feature points of an exemplary embodiment of the present disclosure. Figure 3 As shown, the target feature point of the exemplary embodiment of the present disclosure in the global coordinate system is P l , the panoramic camera can be moved from the first position C to k by Move to the second position C at a speed of k+1 , and the target feature point is P l The projection into the image coordinate system can also be obtained from the first image position by Move to the second image position at a speed of

[0048] For example, considering that the frequency of taking 360° panoramic images by a panoramic camera is relatively low, while the sampling frequency of the IMU is relatively high, therefore, for the target feature point P l For example, its target feature point P l Two-dimensional position correction data

[0049] Step 202: Based on the three-dimensional position data of each target feature point and the position and posture of the IMU, determine the position data of each target feature point in the IMU.

[0050] Step 203: Determine the external parameters between the panoramic camera and the IMU based on the position data of the multiple target feature points in the IMU, the two-dimensional position correction data of the multiple target feature points, and the internal parameters of the panoramic camera.

[0051] Exemplarily, when the intrinsic parameters of the panoramic camera and the extrinsic parameters of the IMU are known, the present disclosure may exemplarily determine a fusion model of the panoramic camera and the IMU, and the fusion model of the panoramic camera and the IMU may satisfy Formula 1:

[0052]

[0053] Among them, when the dedistortion equation is used to dedistort the target feature points, f(c) represents the dedistortion model, and when the target feature points that meet the distortion requirements are extracted through the deep extraction network, f(c) represents the feature point extraction model. π(f(c) can be expressed as the two-dimensional position calculation data of the target feature points, which can be expressed as Formula 2:

[0054] π(P l )=T cb •T wb -1 •K -1 •P l Formula 2

[0055] Among them, T cb Represents the pose transfer matrix between the panoramic camera and IMU, T wb represents the position and posture of the IMU, and K represents the internal parameters of the panoramic camera. Here, the internal parameters of the panoramic camera may include the principal point parameters and focal length parameters of the 360° panoramic image.

[0056] When the exemplary embodiment of the present disclosure uses the camera intrinsic parameters of the front fisheye camera in the panoramic camera to calibrate the external parameters between the panoramic camera and the IMU, the inventors found that because the foreground fisheye camera is composed of a dozen different lenses, distortion is introduced in the imaging process. By compressing the diameter space, the limitation of the imaging angle of view is broken through, thereby achieving wide-angle imaging. Therefore, the introduction of distortion also has an adverse effect on the accuracy of the external parameter calibration between the panoramic camera and the IMU.

[0057] In view of the problem of using the camera intrinsic parameters of the front fisheye camera to calibrate the external parameters between the panoramic camera and the IMU, the exemplary embodiment of the present disclosure can define the 360° panoramic image as a ring image, unfold it through the principle of equidistant cylindrical projection, and the size of the 360° panoramic image in the unfolded state is defined as the unfolded size of the 360° panoramic image, and the internal parameters of the panoramic camera are set to be related to the unfolded size of the 360° panoramic image. Since the unfolded size of the panoramic image is fixed, the external parameters between the panoramic camera and the IMU are calibrated based on the intrinsic parameters of the panoramic camera and the position and posture of the IMU, which can ensure the calibration accuracy.

[0058] Moreover, the distortion degree of multiple target feature points of the exemplary embodiment of the present disclosure is relatively low. Even if they are unevenly distributed in the 360° panoramic image or only distributed in a local area of ​​the 360° panoramic image, the invariance of the intrinsic parameters of the panoramic camera and the position and posture of the IMU can be used to accurately calibrate the extrinsic parameters between the panoramic camera and the IMU, thereby eliminating the adverse effect of the distortion introduced by the foreground fisheye camera on the calibration accuracy.

[0059] Exemplarily, the principal point parameters of the panoramic camera of the exemplary embodiment of the present disclosure are determined by the unfolded size of the 360° panoramic image, for example: the principal point parameters include the horizontal coordinate of the principal point in the pixel coordinate system and the vertical coordinate of the principal point in the pixel coordinate system, the horizontal coordinate of the principal point in the pixel coordinate system can be equal to half of the width of the 360° panoramic image, and the horizontal coordinate of the principal point in the pixel coordinate system can be equal to the vertical coordinate of the principal point can be equal to half of the width of the 360° panoramic image.

[0060] Exemplarily, the focal length parameter of the exemplary embodiment of the present disclosure may be related to the unfolded size of the 360° panoramic image and the spherical coordinate system parameter range of the panoramic camera.

[0061] For example, when the focal length parameter includes the horizontal coordinate of the focal length in the pixel coordinate system and the vertical coordinate of the focal length in the pixel coordinate system, considering that the panoramic image is a ring image and its size is constant when it is unfolded, the horizontal coordinate of the focal length in the pixel coordinate system can be related to the unfolded width W of the 360° panoramic image and the longitude coordinate range of the spherical coordinate system of the panoramic camera -π<θ<π. The vertical coordinate of the focal length in the pixel coordinate system can be related to the unfolded height of the 360° panoramic image and the latitude coordinate range of the spherical coordinate system of the panoramic camera related.

[0062] For example, the intrinsic parameter K of the panoramic camera can be expressed as Formula 3, and the projection equation based on the intrinsic parameter K of the panoramic camera can be expressed as Formula 4.

[0063]

[0064]

[0065] in, represents the pixel coordinates of the target feature point, u represents the horizontal coordinate of the target feature point in the pixel coordinate system, and v represents the horizontal coordinate of the target feature point in the pixel coordinate system. represents the coordinates of the target feature point in the panoramic camera coordinate system, θ represents the longitude of the target feature point in the panoramic camera coordinate system, and its value range is -π<θ<π. Indicates the latitude of the target feature point in the panoramic camera coordinate system, and its value range is Here, the panoramic camera coordinate system is assumed to be a spherical coordinate system, and the spherical coordinate system is idealized into a cylindrical coordinate system, and then expanded to form a pixel coordinate system.

[0066] In an optional manner, the exemplary embodiment of the present disclosure obtains two-dimensional position measurement data of multiple target feature points of a 360° panoramic image taken by a panoramic camera, including: extracting two-dimensional position measurement data of multiple initial feature points of the 360° panoramic image, and obtaining two-dimensional position measurement data of multiple target feature points from the two-dimensional position measurement data of the multiple initial feature points.

[0067] Exemplarily, the exemplary embodiment of the present disclosure can obtain the degree of distortion of the image data of the initial feature point based on the two-dimensional position measurement data of the initial feature point. If the degree of distortion of the image data of the initial feature point is less than a preset degree of distortion, the two-dimensional position measurement data of the initial feature point is determined to be the two-dimensional position measurement data of the target feature point.

[0068] For example, the distortion degree of the image data of the initial feature points can be obtained by using the equidistant projection method based on the two-dimensional position measurement data of the initial feature points, and the known distortion degree distribution feature map of the 360° panoramic image can be obtained, and then the distortion degree of the image data of the initial feature points can be obtained according to the two-dimensional position measurement data of the initial feature points. It can be seen that the exemplary embodiment of the present disclosure can ensure the accuracy of the external parameters between the panoramic camera and the IMU that are finally calibrated by filtering out the distorted feature points.

[0069] In an optional manner, the exemplary embodiment of the present disclosure may also use a deep extraction network to extract target feature points that meet the requirements. Figure 4 FIG. 2 shows a schematic diagram of a process for extracting two-dimensional position measurement data of target feature points according to an exemplary embodiment of the present disclosure. Figure 4 As shown, the exemplary embodiment of the present disclosure extracts two-dimensional position measurement data of multiple target feature points of a 360° panoramic image captured by a panoramic camera, which may include:

[0070] Step 401: extracting a feature map of the 360° panoramic image. The scale of the feature map of the 360° panoramic image extracted here can be smaller than the scale of the 360° panoramic image. In other words, step 401 can not only realize feature map extraction, but also realize image scale reduction. For example, the feature map of the 360° panoramic image can be extracted by various deep neural networks such as VGG and ResNet.

[0071] Step 402: Obtain multiple target feature points based on the feature map of the 360° panoramic image. Here, the feature point probability of the feature map can be determined based on the feature map of the 360° panoramic image, and then the two-dimensional position measurement data of the multiple target feature points can be determined based on the feature point probability of the feature map.

[0072] When determining multiple target feature points based on the feature point probability map of the feature map of the 360° panoramic image, the feature point probability of the feature map can be processed by sub-pixel convolution to improve the image resolution, so as to obtain the resolution improvement result of the feature point probability, and then determine the two-dimensional position measurement data of the multiple target feature points based on the resolution improvement result of the feature point probability.

[0073] The image resolution improvement processing method of the exemplary embodiment of the present disclosure may include upsampling, bilinear difference, deconvolution, pixel shuffling, etc. When the feature map resolution of the 360° panoramic image is improved by pixel shuffling, the influence of artificial factors on the resolution improvement result can be reduced.

[0074] Exemplary Embodiments of the Disclosure Figure 4 The 2D position measurement data extraction process shown can also be done by Figure 5 The feature point extraction model architecture is shown in Figure 1. Figure 5 As shown, the feature point extraction model 500 of the exemplary embodiment of the present disclosure may include an encoder 501 and a decoder 502. The encoder 501 may include a plurality of convolution structures connected in series, each of which may further extract features from the feature map output by the previous convolution structure while reducing the scale of the feature map. The decoder 502 may include a feature point detection network 5021, and may also include a descriptor detection network 5022 on this basis.

[0075] like Figure 5 As shown, the feature point detection network 5021 can determine the feature point probability of the feature map based on the feature map of the 360° panoramic image, and determine the two-dimensional position measurement data of multiple target feature points based on the feature point probability of the feature map. The descriptor detection network 5022 can output the descriptor of the target feature point based on the feature map of the 360° panoramic image. The feature point extraction model has relatively good generalization ability and robustness, and can ensure that the two-dimensional position measurement data of the extracted target feature points has a relatively low degree of distortion.

[0076] When training the feature point extraction model, simulated panoramic images with obvious edge features and corner features can be randomly generated. The simulated panoramic images include 9 types of simulated panoramic images, including checkerboard, cube, ellipse, non-intersecting lines, intersecting lines, multiple polygons, single polygon, stripes and Gaussian noise. Since the simulated panoramic images are computer-generated, the true coordinate values ​​(pseudo labels) of the feature points on the image are attached. The Base Detector, that is, MagicPoint, is trained through these simulated panoramic images to extract the two-dimensional position measurement data of the target feature points of the 360° panoramic image samples.

[0077] The 360° panoramic image samples are geometrically transformed to obtain the transformed 360° panoramic image samples, thereby forming a pair of image samples with a known posture relationship. The image sample pairs are input into Figure 5 The feature point extraction model shown in the figure extracts the two-dimensional position measurement data of the target feature point and the descriptor of the target feature point. In this process, the loss function can be constructed with the pose error of the 360° panoramic image sample and the transformed 360° panoramic image sample, so as to achieve the purpose of training the feature point extraction model.

[0078] One or more technical solutions provided in the exemplary embodiments of the present disclosure can extract the two-dimensional position measurement data of multiple target feature points of the 360° panoramic image taken by the panoramic camera to control the degree of distortion of the image data of each target feature point to be less than a preset degree of distortion, and then determine the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of the multiple target feature points, the three-dimensional position data of the multiple target feature points, the internal parameters of the panoramic camera, and the position and posture of the IMU. This calibration method has high calibration accuracy and can improve the accuracy of 360-VIO based on 360° panoramic images.

[0079] The above mainly introduces the solution provided by the embodiment of the present disclosure from the perspective of an electronic device. It is understandable that, in order to realize the above functions, the electronic device includes a hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0080] The disclosed embodiment can divide the electronic device into functional units according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the disclosed embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0081] In the case of dividing each functional module according to each function, an exemplary embodiment of the present disclosure provides an external parameter calibration device, which may be an electronic device or a chip applied to the electronic device. Figure 6FIG. 2 shows a schematic block diagram of functional modules of an external parameter calibration device according to an exemplary embodiment of the present disclosure. Figure 6 As shown, the external parameter calibration device 600 includes:

[0082] An extraction module 601 is used to extract two-dimensional position measurement data of multiple target feature points of a 360° panoramic image taken by a panoramic camera, wherein the distortion degree of the image data of each target feature point is less than a preset distortion degree;

[0083] The determination module 602 is used to determine the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of the multiple target feature points, the three-dimensional position data of the multiple target feature points, the internal parameters of the panoramic camera and the posture of the IMU.

[0084] In a possible implementation, the extraction module 601 is used to extract two-dimensional position measurement data of multiple initial feature points of the 360° panoramic image, and obtain two-dimensional position measurement data of multiple target feature points from the two-dimensional position measurement data of the multiple initial feature points.

[0085] In a possible implementation, the extraction module 601 is used to obtain the degree of distortion of the image data of the initial feature point based on the two-dimensional position measurement data of the initial feature point. If the degree of distortion of the image data of the initial feature point is less than a preset distortion degree, the two-dimensional position measurement data of the initial feature point is determined to be the two-dimensional position measurement data of the target feature point.

[0086] In a possible implementation, the extraction module 601 is used to extract a feature map of the 360° panoramic image, and obtain two-dimensional position measurement data of multiple target feature points based on the feature map of the 360° panoramic image, wherein the scale of the feature map of the 360° panoramic image is smaller than the scale of the 360° panoramic image.

[0087] In a possible implementation, the extraction module 601 is used to determine the probability of feature points of the feature map based on the feature map of the 360° panoramic image, and determine the two-dimensional position measurement data of a plurality of the target feature points based on the probability of feature points of the feature map.

[0088] In a possible implementation, the extraction module 601 is used to perform image resolution improvement processing on the feature point probability of the feature map to obtain a resolution improvement result of the feature point probability, and determine two-dimensional position measurement data of multiple target feature points based on the resolution improvement result of the feature point probability.

[0089] In one possible implementation, the determination module 602 is used to determine the two-dimensional position correction data of the target feature point based on the two-dimensional position measurement data of the target feature point, the moving speed of the panoramic camera, and the time offset between the panoramic camera and the IMU, determine the position data of each target feature point in the IMU based on the three-dimensional position data of each target feature point and the posture of the IMU, and determine the external parameters between the panoramic camera and the IMU based on the position data of multiple target feature points in the IMU, the two-dimensional position correction data of multiple target feature points, and the internal parameters of the panoramic camera.

[0090] In one possible implementation, the determination module 602 is used to determine the reprojection error relationship based on the position data of multiple target feature points in the IMU, the two-dimensional position correction data of multiple target feature points, and the intrinsic parameters of the panoramic camera, and determine the external parameters between the panoramic camera and the IMU based on the reprojection error relationship.

[0091] In a possible implementation, the internal parameters of the panoramic camera include a principal point parameter and a focal length parameter of the 360° panoramic image, wherein the principal point parameter is determined by the unfolded size of the 360° panoramic image, and the focal length parameter is related to the unfolded size of the 360° panoramic image and a spherical coordinate system parameter range of the 360° panoramic image.

[0092] Figure 7 Schematic block diagram of a chip according to an exemplary embodiment of the present disclosure is shown. Figure 7 As shown, the chip 700 includes one or more (including two) processors 701 and a communication interface 702. The communication interface 702 can support the electronic device to perform the data sending and receiving steps in the above method, and the processor 701 can support the electronic device to perform the data processing steps in the above method.

[0093] Optional, such as Figure 7 As shown, the chip 700 also includes a memory 703, which may include a read-only memory and a random access memory, and provides operation instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory (NVRAM).

[0094] In some embodiments, Figure 7As shown, the processor 701 performs corresponding operations by calling the operation instructions stored in the memory (the operation instructions may be stored in the operating system). The processor 701 controls the processing operations of any one of the terminal devices, and the processor may also be called a central processing unit (CPU). The memory 703 may include a read-only memory and a random access memory, and provides instructions and data to the processor 701. A portion of the memory 703 may also include NVRAM. For example, in an application, the memory, the communication interface, and the memory are coupled together through a bus system, wherein the bus system may include a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, in Figure 7 Various buses are labeled as bus system 704 .

[0095] The method disclosed in the above-mentioned embodiment of the present disclosure can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above-mentioned processor may be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an ASIC, a field-programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present disclosure can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present disclosure can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in conjunction with its hardware.

[0096] The exemplary embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to cause the electronic device to perform the method according to the embodiment of the present disclosure when executed by the at least one processor.

[0097] The exemplary embodiments of the present disclosure also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform the method according to the embodiments of the present disclosure.

[0098] The exemplary embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform the method according to the embodiments of the present disclosure.

[0099] refer to Figure 8 , a block diagram of an electronic device 800 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0100] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0101] A plurality of components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 804 may include, but is not limited to, a disk, an optical disk. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0102] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above. For example, in some embodiments, the method of the exemplary embodiment of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via the ROM 802 and / or the communication unit 809. In some embodiments, the computing unit 801 may be configured to perform the method of the exemplary embodiment of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0103] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0104] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0106] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0107] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0108] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is executed in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a terminal, a user device or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer program or instruction may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, for example, a floppy disk, a hard disk, a tape; it may also be an optical medium, for example, a digital video disc (DVD); it may also be a semiconductor medium, for example, a solid state drive (SSD).

[0109] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for calibrating an external parameter, characterized in that: include: Extracting two-dimensional position measurement data of a plurality of target feature points of a 360° panoramic image captured by a panoramic camera, wherein the degree of distortion of the image data of each of the target feature points is less than a preset degree of distortion; Based on the two-dimensional position measurement data of the plurality of target feature points, the three-dimensional position data of the plurality of target feature points, the intrinsic parameters of the panoramic camera and the position and posture of the IMU, the extrinsic parameters between the panoramic camera and the IMU are determined.

2. The method according to claim 1, characterized in that The step of extracting two-dimensional position measurement data of a plurality of target feature points of a 360° panoramic image captured by a panoramic camera comprises: Extracting two-dimensional position measurement data of a plurality of initial feature points of the 360° panoramic image; The two-dimensional position measurement data of the plurality of target feature points are acquired from the two-dimensional position measurement data of the plurality of initial feature points.

3. The method according to claim 2, characterized in that The step of acquiring the two-dimensional position measurement data of the plurality of target feature points from the two-dimensional position measurement data of the plurality of initial feature points comprises: Acquire the degree of image data distortion of the initial feature point based on the two-dimensional position measurement data of the initial feature point; If the degree of distortion of the image data of the initial feature point is less than a preset degree of distortion, the two-dimensional position measurement data of the initial feature point is determined as the two-dimensional position measurement data of the target feature point.

4. The method according to claim 1, characterized in that: The step of extracting two-dimensional position measurement data of a plurality of target feature points of a 360° panoramic image captured by a panoramic camera comprises: Extracting a feature map of the 360° panoramic image, wherein a scale of the feature map of the 360° panoramic image is smaller than a scale of the 360° panoramic image; The two-dimensional position measurement data of the plurality of target feature points are obtained based on the feature map of the 360° panoramic image.

5. The method according to claim 4, characterized in that The step of obtaining two-dimensional position measurement data of a plurality of target feature points based on the feature map of the 360° panoramic image comprises: Determine a probability of feature points of the feature map based on the feature map of the 360° panoramic image; Based on the feature point probabilities of the feature graph, two-dimensional position measurement data of a plurality of the target feature points are determined.

6. The method according to claim 5, characterized in that The determining of the two-dimensional position measurement data of the plurality of target feature points based on the feature point probability map of the feature map of the 360° panoramic image comprises: Performing image resolution improvement processing on the feature point probability of the feature graph to obtain a resolution improvement result of the feature point probability; Based on the pixel lifting result of the feature point probability, two-dimensional position measurement data of a plurality of the target feature points are determined.

7. The method according to claim 1, characterized in that The internal parameters of the panoramic camera include the principal point parameter and the focal length parameter of the 360° panoramic image, wherein the principal point parameter is determined by the unfolded size of the 360° panoramic image, and the focal length parameter is related to the unfolded size of the 360° panoramic image and the spherical coordinate system parameter range of the 360° panoramic image.

8. The method according to any one of claims 1 to 7, characterized in that: The method of determining the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of the plurality of target feature points, the three-dimensional position data of the plurality of target feature points, the internal parameters of the panoramic camera and the position and posture of the IMU includes: Determine the two-dimensional position correction data of the target feature point based on the two-dimensional position measurement data of the target feature point, the moving speed of the panoramic camera and the time offset between the panoramic camera and the IMU; Based on the three-dimensional position data of each target feature point and the position and posture of the IMU, determine the position data of each target feature point in the IMU; Based on the position data of the plurality of target feature points in the IMU, the two-dimensional position correction data of the plurality of target feature points and the intrinsic parameters of the panoramic camera, the extrinsic parameters between the panoramic camera and the IMU are determined.

9. The method according to claim 8, characterized in that The determining of the external parameters between the panoramic camera and the IMU based on the position data of the plurality of target feature points in the IMU, the two-dimensional position correction data of the plurality of target feature points and the internal parameters of the panoramic camera comprises: Determine a reprojection error relationship based on position data of a plurality of target feature points in the IMU, two-dimensional position correction data of a plurality of target feature points, and an internal parameter of the panoramic camera; The external parameters between the panoramic camera and the IMU are determined based on the reprojection error relationship.

10. An external parameter calibration device, characterized in that: include: An extraction module, used to extract two-dimensional position measurement data of multiple target feature points of a 360° panoramic image taken by a panoramic camera, wherein the degree of distortion of the image data of each target feature point is less than a preset degree of distortion; The determination module is used to determine the external parameters between the panoramic camera and the IMU based on the two-dimensional position measurement data of the multiple target feature points, the three-dimensional position data of the multiple target feature points, the internal parameters of the panoramic camera and the position and posture of the IMU.

11. An electronic device, characterized in that: include: processor; as well as, A memory for storing programs; The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.

Images