Calibration method and device and storage medium

Abstract

The embodiment of the invention discloses a calibration method and device and a storage medium, and the method comprises the steps: obtaining an internal reference initial value and visual odometer data of an image collection device, the visual odometer data being generated based on target images shot at a plurality of continuous moments in the movement process of the image collection device; the visual odometer data comprises image feature point data of the target image and initial pose estimation data of the image acquisition equipment when the target image is shot, and the image feature point data comprises a three-dimensional coordinate estimation value of an image feature point under a world coordinate system and a two-dimensional coordinate observation value of the image feature point on the target image; screening out visual odometer data in a non-critical motion state from the visual odometer data based on initial pose estimation data of target images shot at adjacent moments; and solving an internal reference estimation value of the image acquisition equipment at least based on an internal reference initial value of the image acquisition equipment and the screened visual odometer data. The calibration precision of the internal reference of the image acquisition equipment can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of visual technology, and in particular to a calibration method, device and storage medium. Background Art

[0002] The vision-based electronic map construction solution needs to rely on accurate internal parameters of image acquisition devices (such as cameras). However, due to production errors, differences in installed vehicles, etc., the actual internal parameters of the image acquisition device will change compared to when it leaves the factory. If the internal parameters of the image acquisition device are not calibrated, the accuracy of visual positioning and mapping will be affected, ultimately resulting in inaccurate electronic maps constructed based on vision.

[0003] In the existing offline calibration scheme, operators are required to carry calibration equipment (such as chessboard, QR code calibration plate, etc.) to perform field calibration on each image acquisition device. However, the offline calibration method is difficult to be widely used in terms of cost and application mode.

[0004] Therefore, it is necessary to propose a more applicable calibration scheme that can not only reduce the intrinsic calibration cost of image acquisition equipment, but also ensure accuracy and improve efficiency. Summary of the invention

[0005] Embodiments of the present disclosure provide a calibration method, device, and storage medium.

[0006] In a first aspect, an embodiment of the present disclosure provides a calibration method, which includes:

[0007] Acquire the initial value of the internal parameter and the visual odometer data of the image acquisition device, wherein the visual odometer data is generated based on the target images captured at multiple consecutive moments during the movement of the image acquisition device; the visual odometer data includes the image feature point data of the target image and the initial pose estimation data of the image acquisition device when capturing the target image, wherein the image feature point data includes: the three-dimensional coordinate estimation value of the image feature point in the world coordinate system and the two-dimensional coordinate observation value on the target image;

[0008] Based on the initial pose estimation data of the target images taken at adjacent moments, the visual odometer data of the non-critical motion state is screened out from the visual odometer data;

[0009] An estimated value of the intrinsic parameter of the image acquisition device is solved based on at least an initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data.

[0010] In a second aspect, an embodiment of the present invention provides a calibration device, which includes:

[0011] A first acquisition module is configured to acquire an initial value of an internal parameter of an image acquisition device and visual odometer data, wherein the visual odometer data is generated based on target images captured at multiple consecutive moments during the motion of the image acquisition device; the visual odometer data includes image feature point data of the target image and initial pose estimation data of the image acquisition device when capturing the target image, wherein the image feature point data includes: a three-dimensional coordinate estimation value of the image feature point in a world coordinate system and a two-dimensional coordinate observation value on the target image;

[0012] A first screening module is configured to screen out visual odometer data of a non-critical motion state from the visual odometer data based on initial pose estimation data of target images captured at adjacent moments;

[0013] The solution module is configured to solve the estimated value of the intrinsic parameter of the image acquisition device based on at least the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data.

[0014] The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the functions.

[0015] In one possible design, the structure of the above-mentioned device includes a memory and a processor, the memory is used to store one or more computer instructions that support the above-mentioned device to execute the above-mentioned corresponding method, and the processor is configured to execute the computer instructions stored in the memory. The above-mentioned device may also include a communication interface for the above-mentioned device to communicate with other devices or communication networks.

[0016] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the method described in any one of the above aspects.

[0017] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium for storing computer instructions used by any of the above-mentioned devices, and when the computer instructions are executed by a processor, they are used to implement the method described in any of the above-mentioned aspects.

[0018] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, which includes computer instructions, and when the computer instructions are executed by a processor, they are used to implement the method described in any of the above aspects.

[0019] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0020] The disclosed embodiment proposes a method for automatically calibrating the internal parameters of an image acquisition device mounted on a vehicle, wherein the method obtains visual odometer data generated by the image acquisition device as the vehicle moves, and screens out the visual odometer data of the image acquisition device in a non-critical motion state, thereby optimizing and solving the estimated internal parameters of the image acquisition device based on the initial value of the internal parameters of the image acquisition device and the screened visual odometer data. In this way, since no human intervention is required, the automatic calibration of the internal parameters of the vehicle-mounted image acquisition device can be achieved, and the cost of the internal parameter calibration of the image acquisition device can be reduced. In addition, since the disclosed embodiment analyzes the motion characteristics of the image acquisition device during the calibration process and screens out the visual odometer data of the image acquisition device in a non-critical motion state, the calibration accuracy of the internal parameters of the image acquisition device is improved, and the calibration efficiency is improved.

[0021] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0023] Figure 1 A flow chart of a calibration method according to an embodiment of the present disclosure is shown.

[0024] Figure 2 A schematic diagram of an implementation flow of camera intrinsic parameter calibration according to one embodiment of the present disclosure is shown.

[0025] Figure 3 A structural block diagram of a calibration device according to an embodiment of the present disclosure is shown.

[0026] Figure 4 It is a structural schematic diagram of an electronic device suitable for implementing a calibration method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0028] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the specification, and do not exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0029] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0030] The user information (including but not limited to user device information such as location information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] The details of the embodiments of the present disclosure are described in detail below through specific examples.

[0032] Figure 1 FIG. 1 is a flow chart showing a calibration method according to an embodiment of the present disclosure. Figure 1 As shown, the calibration method includes the following steps:

[0033] In step S101, the initial value of the internal parameter and the visual odometer data of the image acquisition device are obtained, wherein the visual odometer data is generated based on the target images captured at multiple consecutive moments during the movement of the image acquisition device; the visual odometer data includes image feature point data of the target image and initial pose estimation data of the image acquisition device when the target image is captured, wherein the image feature point data includes: three-dimensional coordinate estimation values ​​of the image feature points in the world coordinate system and two-dimensional coordinate observation values ​​on the target image;

[0034] In step S102, based on the initial pose estimation data of the target images taken at adjacent moments, the visual odometer data of the non-critical motion state is screened out from the visual odometer data;

[0035] In step S103, an estimated value of the intrinsic parameter of the image acquisition device is solved based on at least the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data.

[0036] In this embodiment, the calibration method can be executed on a server or a terminal. The image acquisition device can be a monocular RGB camera mounted on a vehicle. The vehicle can be a data acquisition vehicle or a crowdsourcing vehicle for collecting map data production materials.

[0037] The initial value of the internal parameter of the image acquisition device can be recorded as in It is composed of the horizontal focal length, vertical focal length, optical center position and distortion coefficient of the image acquisition device. is the distortion parameter of the image acquisition device, which is related to the specific distortion model selected. In some embodiments, the distortion model can be Radtan, Equidistant, etc.

[0038] In some embodiments, the initial value of the intrinsic parameter of the image acquisition device can be determined by factory parameters, and / or calibration results of the same model of image acquisition device. Generally, the error of the initial value of the intrinsic parameter of the image acquisition device is large, and it is difficult to support high-precision visual tasks. Therefore, the embodiment of the present disclosure proposes a calibration method, which optimizes and estimates the initial value of the intrinsic parameter of the image acquisition device and the motion-related information generated by the image acquisition device during the motion process, and then obtains a more accurate estimate of the intrinsic parameter of the image acquisition device, which is used to calibrate the intrinsic parameter of the image acquisition device.

[0039] In some embodiments, the motion-related information generated by the image acquisition device during the motion process may include but is not limited to the visual odometer data of the image acquisition device. The visual odometer data may be generated based on the target images captured at multiple consecutive moments during the motion of the image acquisition device. The visual odometer data may include but is not limited to the image feature point data of the target image and the initial pose estimation data of the image acquisition device when capturing the target image; the image feature point data may include but is not limited to the three-dimensional coordinate estimation values ​​of multiple image feature points in the world coordinate system and the two-dimensional coordinate observation values ​​of the multiple image feature points on the target image, respectively. The target image is an image in which multiple image feature points can be observed in the images captured by the image acquisition device at multiple consecutive moments.

[0040] Visual odometry is a process of determining the position and direction of an image acquisition device by analyzing images acquired by the image acquisition device at multiple consecutive moments. It can be understood that in the embodiments of the present disclosure, the image acquisition device is mounted on a vehicle, and the movement process of the image acquisition device can be understood as the movement process of the vehicle.

[0041] Through the visual odometry system, the initial pose estimation data of the image acquisition device at each moment can be determined based on the images acquired by the image acquisition device at multiple consecutive moments. The three-dimensional coordinate estimation values ​​of the image feature points on the images acquired at multiple consecutive moments in the world coordinate system and the two-dimensional coordinate observation values ​​of the multiple image feature points on the images acquired at different moments can also be determined. It can be understood that the two-dimensional coordinate observation values ​​are the image coordinates on the images corresponding to each moment, and the two-dimensional coordinate observation values ​​of the same image feature point in the image acquisition environment are different on different images.

[0042] In the embodiment of the present disclosure, the initial pose estimation data of the image acquisition device can be obtained by using the visual odometer. Initial value of 3D coordinate estimation of image feature points And the matching results of image feature points in the world coordinate system and image coordinate system That is, the matching result between the estimated 3D coordinates of the same image feature point and the 2D coordinates observed in each image. Represents the mapping from the image feature point i to the j-th set of two-dimensional image points in the t-th frame image of all the two-dimensional coordinates of the image feature point that can be observed, that is, is the two-dimensional coordinate observation value in the t-th frame image.

[0043] In some embodiments, the visual odometer can be implemented using existing technologies, such as ORB-SLAM3, SVO, etc., and this disclosure does not impose any limitations.

[0044] In the disclosed embodiment, in order to prevent the motion trajectory of the image acquisition device at multiple consecutive moments from causing critical motion sequences (CMS), the visual odometer data corresponding to some moments that meet the preset conditions can be screened out from the visual odometer data corresponding to the target images acquired at multiple consecutive moments. The visual odometer data corresponding to some moments that meet the preset conditions can be understood as the visual odometer data corresponding to the moment when the image acquisition device is in a non-critical motion state. Under critical motion, the initial pose estimation data of the image acquisition device output by the visual odometer cannot be uniquely optimized to solve the intrinsic parameters of the image acquisition device, or the solved intrinsic parameter estimation value of the image acquisition device is greatly affected by noise and has poor accuracy.

[0045] In some embodiments, the motion trajectory of the image acquisition device at the selected partial moment does not constitute critical motion, that is, it is not in a critical motion state. Therefore, based on the initial pose estimation data of the image acquisition device at multiple consecutive moments output by the visual odometer, the visual odometer data corresponding to the partial moments that do not constitute a critical motion state can be screened out from the visual odometer data corresponding to the multiple consecutive moments. In some embodiments, the partial moments that meet the preset conditions can be understood as the camera intrinsic parameters of the image acquisition device with higher accuracy can be solved based on the initial pose estimation data of the image acquisition device at the partial moment, and the motion trajectory of the image acquisition device at the partial moment does not constitute a critical motion. It should be noted that the partial moment can be a part of the continuous moments of the above-mentioned multiple consecutive moments, that is, the partial moment can include multiple moments. In some embodiments, when determining whether the image acquisition device is in a critical motion state at the current moment, it can be determined based on the initial position estimation data of the previous moment of the adjacent moment and the current moment.

[0046] After filtering out the line-of-sight odometer data corresponding to some moments that meet the preset conditions, the estimated values ​​of the intrinsic parameters of the image acquisition device can be optimized based on the line-of-sight odometer data corresponding to the some moments and the initial values ​​of the intrinsic parameters of the image acquisition device. The solved estimated values ​​of the intrinsic parameters of the image acquisition device can be used to calibrate the internal parameters of the image acquisition device.

[0047] In this embodiment, after the visual odometer data corresponding to the motion trajectory of the image acquisition device in a non-critical motion state is screened, an optimization equation can be established based on the screened visual odometer data and the initial value of the intrinsic parameter of the image acquisition device, so that the estimated value of the intrinsic parameter of the image acquisition device with the smallest error is obtained through continuous iterative optimization. In the above process, the motion characteristics of the image acquisition device are analyzed, and the visual odometer data of the image acquisition device in a non-critical motion state are screened. Therefore, based on the screened visual odometer data, after the optimization equation is established, the internal parameters of the image acquisition device are optimized on the basis of the initial value of the internal parameters of the image acquisition device, and finally the estimated value of the internal parameter of the image acquisition device with higher accuracy can be obtained, and the estimated value can be used as the calibration value of the internal parameter of the image acquisition device.

[0048] In some embodiments, the objective function of the optimization equation can be established by referring to the existing technology, and no specific limitation is made here. For example, the estimated value of the intrinsic parameter of the image acquisition device can be obtained by establishing a factor graph and optimizing the maximum a posteriori estimation method of the intrinsic parameter of the image acquisition device.

[0049] In some embodiments, when optimizing and solving the intrinsic parameter estimation value of the image acquisition device, the two-dimensional coordinate observation values ​​of the image feature points in the visual odometer data corresponding to the screened out partial moments are used as the observation data in the optimization objective function, and based on the projection equation of the three-dimensional coordinates of the image feature points in the world coordinate system to the two-dimensional image space corresponding to the partial moments, the three-dimensional coordinate estimation values ​​of the image feature points in the visual odometer data can be projected to the two-dimensional image space to obtain the two-dimensional coordinate estimation values ​​of the image feature points at these partial moments. These two-dimensional coordinate estimation values ​​are used as the estimation data in the objective function, and then the minimization of the error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of the same image feature point is used as the optimization goal, and finally the intrinsic parameter estimation value of the image acquisition device can be optimized and solved.

[0050] The disclosed embodiment proposes a method for automatically calibrating the internal parameters of an image acquisition device mounted on a vehicle, wherein the method obtains visual odometer data generated by the image acquisition device as the vehicle moves, and screens out the visual odometer data of the image acquisition device in a non-critical motion state, thereby optimizing and solving the estimated internal parameters of the image acquisition device based on the initial value of the internal parameters of the image acquisition device and the screened visual odometer data. In this way, since no human intervention is required, the automatic calibration of the internal parameters of the vehicle-mounted image acquisition device can be achieved, and the cost of the internal parameter calibration of the image acquisition device can be reduced. In addition, since the disclosed embodiment analyzes the motion characteristics of the image acquisition device during the calibration process and screens out the visual odometer data of the image acquisition device in a non-critical motion state, the calibration accuracy of the internal parameters of the image acquisition device is improved, and the calibration efficiency is improved.

[0051] In an optional implementation of this embodiment, if the image acquisition device is further fixedly connected to a non-image sensor, the method further includes the following steps:

[0052] Acquire non-image observation data output by the non-image sensor during the movement of the image acquisition device;

[0053] Screening non-image observation data of non-critical motion state from the non-image observation data;

[0054] Step S103, i.e., the step of solving the estimated value of the intrinsic parameter of the image acquisition device based on at least the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data, is specifically implemented as follows:

[0055] Based on the initial value of the intrinsic parameter of the image acquisition device, the screened visual odometer data and the non-image observation data, the estimated value of the intrinsic parameter of the image acquisition device is solved.

[0056] In this optional implementation, the image acquisition device may be further fixedly connected to a non-image sensor, and obtain non-image observation data output by the non-image sensor fixedly connected to the image acquisition device during the movement of the image acquisition device. It can be understood that since the non-image sensor is fixedly connected to the image acquisition device, the non-image observation data output by the non-image sensor is also the non-image observation data of the image acquisition device. In some embodiments, a non-image sensor such as a GPS, an inertial measurement unit, etc. may be fixedly mounted on the vehicle together with the image acquisition device. In some embodiments, the image acquisition device and the non-image sensor may be fixedly connected to the image acquisition device and installed on the same vehicle. During the movement of the vehicle, the GPS may generate the geographic coordinates of the motion trajectory points of the image acquisition device, and the inertial measurement unit may measure the three-axis attitude angle and acceleration information of the image acquisition device during the movement. The non-image observation data may include but is not limited to the geographic coordinates, three-axis attitude angle, acceleration information, etc. of the image acquisition device. When optimizing and solving the estimated intrinsic parameters of the image acquisition device, the visual odometry data can be used as the state data to establish the observation equation, so that the non-image observation data of the image acquisition device estimated based on the visual odometry data can be used as a factor in the optimization objective function, so that the optimization objective further includes minimizing the error between the non-image observation data measured by non-image sensors such as GPS, inertial measurement unit, etc. and the estimated value of the observation data estimated based on the visual odometry data.

[0057] That is to say, in other embodiments, non-image observation data of non-image sensors connected to the image acquisition device can also be added to the process of optimizing the intrinsic parameter estimation value of the image acquisition device, thereby improving the calibration accuracy of the intrinsic parameters of the image acquisition device. It is understandable that in the process of optimizing the intrinsic parameter estimation value of the image acquisition device, the addition of non-image observation data of non-image sensors is not necessary, but it can improve the accuracy of the final optimization result.

[0058] In an optional implementation of this embodiment, step S103, i.e., the step of solving the estimated value of the intrinsic parameter of the image acquisition device based at least on the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data, further includes the following steps:

[0059] Based on the screened visual odometer data and the initial value of the internal parameter of the image acquisition device, the three-dimensional coordinate estimation values ​​of the multiple image feature points are mapped to the target image corresponding to the visual odometer data to obtain the two-dimensional coordinate estimation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data;

[0060] By optimizing the errors between the two-dimensional coordinate estimation values ​​and the two-dimensional coordinate observation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data, the maximum a posteriori estimation value of the intrinsic parameter of the image acquisition device is solved as the intrinsic parameter estimation value of the image acquisition device.

[0061] In this optional implementation, after the visual odometer data corresponding to the partial moments that meet the preset conditions are screened out, an optimization equation can be established to solve the internal parameter estimation value of the image acquisition device. In some embodiments, the optimization solution process can be represented by an objective function, and the optimization goal of the objective function is to minimize the error between the observed data and the estimated data at each moment in the partial moments corresponding to the screened visual odometer data.

[0062] In the disclosed embodiment, the observation data at each moment includes the two-dimensional coordinate observation value of each image feature point in the target image corresponding to the visual odometer data corresponding to the moment, and the estimated data at each moment is the two-dimensional coordinate estimation value obtained after mapping the three-dimensional coordinate estimation value of each image feature point to the target image acquired at the moment based on the visual odometer data and the initial value of the internal parameter of the image acquisition device. Optimizing the internal parameter estimation value of the image acquisition device can be understood as continuously optimizing and adjusting the initial pose estimation data, the three-dimensional coordinate estimation value of the image feature point in the world coordinate system, and the initial value of the internal parameter of the image acquisition device, so as to minimize the error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point. That is, the internal parameter of the image acquisition device corresponding to the minimum error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point is the final internal parameter estimation value.

[0063] When the internal parameters of the image acquisition device are known, the three-dimensional space point can be mapped to the two-dimensional image space through the following projection equation, that is, the three-dimensional coordinate estimation value of the same image feature point and the two-dimensional coordinate estimation value on a certain target image can be expressed by the following projection equation:

[0064]

[0065] Among them, T t Represents the initial pose estimation data of the image acquisition device at time t, which is output by the visual odometer, including the rotation matrix R between the world coordinate system and the image coordinate system t and translation vectors represents the i-th 3D coordinate point in the world coordinate system, which corresponds to the 3D coordinate estimate of the i-th image feature point output by the visual odometer. It represents the projection position of the three-dimensional coordinate point i in the image coordinate system at time t, that is, the estimated two-dimensional coordinate value of the i-th image feature point on the target image collected at time t.

[0066] In an optional implementation of this embodiment, the step of solving the estimated value of the intrinsic parameter of the image acquisition device based on the initial value of the intrinsic parameter of the image acquisition device, the screened visual odometer data and the non-image observation data further includes the following steps:

[0067] Based on the screened visual odometer data and the initial value of the internal parameter of the image acquisition device, the three-dimensional coordinate estimation values ​​of the multiple image feature points are mapped to the target image corresponding to the visual odometer data to obtain the two-dimensional coordinate estimation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data;

[0068] Based on the selected visual odometer data and the initial value of the internal parameter of the image acquisition device, the acquisition time of the target image corresponding to the visual odometer data is expressed by an observation equation, and the non-image sensor estimation value of the non-image sensor of the image acquisition device is fixedly connected;

[0069] By optimizing the first error between the two-dimensional coordinate estimation value and the two-dimensional coordinate observation value of the multiple image feature points in the target image corresponding to the visual odometer data, and the second error between the non-image observation data of the non-image sensor fixedly connected to the image acquisition device at the acquisition time and the non-image sensor estimation value, the maximum a posteriori estimation value of the intrinsic parameter of the image acquisition device is solved as the intrinsic parameter estimation value.

[0070] In this optional implementation, after the visual odometer data corresponding to the partial moments that meet the preset conditions are screened out, an optimization equation can be established to solve the internal parameter estimation value of the image acquisition device. In some embodiments, the optimization solution process can be represented by an objective function, and the optimization goal of the objective function is to minimize the error between the observed data and the estimated data at each moment in the partial moments.

[0071] In the disclosed embodiment, the observation data at each moment includes the two-dimensional coordinate observation value of each image feature point in the target image corresponding to the visual odometer data corresponding to the moment, and the non-image observation data at each moment output by the non-image sensor fixedly connected to the image acquisition device and installed on the vehicle; and the estimated data at each moment includes the two-dimensional coordinate estimation value obtained by mapping the three-dimensional coordinate estimation value of each image feature point to the target image acquired at the moment based on the visual odometer data and the initial value of the internal parameter of the image acquisition device, and the non-image sensor estimation value of the non-image sensor fixedly connected to the image acquisition device expressed by the observation equation based on the visual odometer data and the initial value of the internal parameter of the image acquisition device, which is also the non-image sensor estimation value of the image acquisition device. The non-image sensor estimation value may include but is not limited to the estimation value of geographic coordinates, three-axis attitude angle, acceleration information, etc.

[0072] Optimizing the internal parameter estimation value of the image acquisition device can be understood as continuously optimizing and adjusting the internal parameter initial value and initial pose estimation data of the image acquisition device, and the three-dimensional coordinate estimation value of the image feature point in the world coordinate system, so as to minimize the sum of the first error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point, and the second error between the non-image observation data output by the non-image sensor at each moment in some moments and the non-image sensor estimation value expressed by the observation equation. That is, the sum of the first error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point at each moment in some moments, and the second error between the non-image observation data and the non-image sensor estimation value at each moment in some moments is minimized, and the corresponding internal parameter of the image acquisition device is the final internal parameter estimation value.

[0073] The following takes the construction of factor graph as an example to explain the objective function of solving the maximum a posteriori estimation of the intrinsic parameters of the image acquisition device:

[0074]

[0075] Among them, t is the partial time set At a certain moment in k is the non-image observation data output by the kth non-image sensor, Z k corresponds to z k The observation equation is used to solve the non-image sensor estimation value of the non-image sensor connected to the image acquisition device. K and d are the internal parameters of the image acquisition device. K is composed of the horizontal focal length, vertical focal length, optical center position and distortion coefficient of the image acquisition device. d is the distortion parameter. {T t} is the initial pose estimation data of the image acquisition device at time t, {x i} is the estimated three-dimensional coordinate value of the i-th image feature point, j is x i The j-th two-dimensional point in the corresponding target image, For x i The 2D coordinate observations in the target image acquired at time t.

[0076] In some embodiments, the above objective function can be solved by an unconstrained nonlinear optimization problem, such as using the Levenberg-Marquardt algorithm. During the iterative optimization process, the initial value of the internal parameter of the image acquisition device is And the initial values ​​of the pose are the initial pose estimation data output by the visual odometer And the estimated three-dimensional coordinates of the image feature points Finally, the final solution K can be obtained by optimization. * ,d* , complete the online self-calibration of the internal parameters of the image acquisition device.

[0077] In an optional implementation of this embodiment, the adjacent moments include moment t and moment t-1, and step S102, i.e., the step of screening out visual odometer data of a non-critical motion state from the visual odometer data based on the initial pose estimation data of the target image captured at the adjacent moments, further includes the following steps:

[0078] Calculate the angular velocity estimate of the image acquisition device at time t based on the initial pose estimation data of the target images captured at time t and time t-1;

[0079] Based on the angular velocity estimation value, visual odometer data of a non-critical motion state is filtered out.

[0080] In this optional implementation, the purpose of filtering out the visual odometry data corresponding to some moments that meet preset conditions from the motion-related information corresponding to multiple consecutive moments is to prevent the visual odometry data used in the objective function from being the data of the image acquisition device in a critical motion state when optimizing the internal parameters of the image acquisition device. As described above, the internal parameters of the image acquisition device are optimized based on the visual odometry data of the image acquisition device in a critical motion state, and the final result is greatly affected by noise and has low accuracy.

[0081] To this end, the embodiments of the present disclosure can calculate the angular velocity estimation value of the image acquisition device at each moment based on the initial pose estimation data of the target images taken at adjacent moments in the screened visual odometer data in the non-critical motion state.

[0082] In some embodiments, the angular velocity estimate may be based on the initial pose estimate data T of the image acquisition device at the current time t output by the visual odometer. t And the initial pose estimation data T of the image acquisition device at the previous time t-1 t-1 , the estimated angular velocity ω at time t is calculated by the following formula: t :

[0083]

[0084] Among them, τ t represents the real time corresponding to time t, τ t-1 Indicates the real time corresponding to time t-1, log(·) logarithmic mapping. R t and R t-1 are the rotation matrices at time t and t-1 respectively.

[0085] Angular velocity can be used to identify the rotation direction and rotation amplitude of the image acquisition device during movement.

[0086] After determining the angular velocity estimation value of the image acquisition device at each moment, it is possible to determine whether the image acquisition device is in a critical motion state at the current moment based on the rotation of each angular velocity estimation value in the rotation direction. If it is not in a critical motion state, then the image acquisition device is not in a critical motion state at the current moment and multiple consecutive moments thereafter, so that the visual odometer data at these moments can be screened out. If the image acquisition device is in a critical motion state at the current moment, the visual odometer data corresponding to the current moment does not meet the preset conditions and can be discarded, so as to continue to judge the next moment.

[0087] In some embodiments, if it is determined through the angular velocity estimation value that the image acquisition device rotates a small amount in the rotation direction at the current moment, it can be considered that the movement of the image acquisition device at the current moment may be critical movement, and the corresponding visual odometer data can be discarded. If it is determined through the angular velocity estimation value that the image acquisition device rotates a large amount in the rotation direction at the current moment, it can be considered that the movement of the image acquisition device at the current moment may be non-critical movement, and the corresponding visual odometer data can be retained; it can also be considered that the rotation amplitude of the image acquisition device in the rotation direction at the subsequent multiple consecutive moments is also large. Therefore, at the current moment and its subsequent multiple consecutive moments, the image acquisition device should be in a non-critical motion state, and the corresponding visual odometer data can be retained.

[0088] In an optional implementation of this embodiment, the step of filtering out visual odometer data in a non-critical motion state based on the angular velocity estimation value further includes the following steps:

[0089] Obtaining a modulus value of the angular velocity estimate;

[0090] Obtaining an angular velocity estimation value whose modulus value is greater than a set modulus value threshold as a target estimation value;

[0091] Performing principal component analysis on the target estimation value to determine three rotation directions of the image acquisition device during the movement; the three rotation directions are in an orthogonal relationship;

[0092] Projecting the estimated angular velocity value in the three rotation directions respectively to obtain a projection result of the estimated angular velocity value in each rotation direction;

[0093] Based on the projection result of the angular velocity estimation value in each rotation direction, determining whether the image acquisition device is in a non-critical motion state at the time corresponding to the angular velocity estimation value;

[0094] Based on the moment when the image acquisition device is in a non-critical motion state, the visual odometer data in the non-critical motion state is screened out.

[0095] In this optional implementation, as described above, the angular velocity estimate is used to identify the rotation amplitude of the image acquisition device during movement.

[0096] In some embodiments, the rotation of the image acquisition device during the motion process can be represented by three rotation directions that are orthogonal to each other. The three rotation directions can include a main rotation direction, a secondary rotation direction, and a minimum rotation direction. In some embodiments, the rotation angles of the image acquisition device in the three rotation directions can be understood as yaw angle, pitch angle, and tilt angle.

[0097] In the disclosed embodiment, the three rotation directions are determined by performing principal component analysis on all calculated angular velocity estimation values, and the three rotation directions can be represented by corresponding vector features. In order to more accurately determine the three rotation directions, principal component analysis can be performed based on angular velocity estimation values ​​with larger modulus values. Therefore, a set modulus threshold can be set in advance, and the angular velocity estimation values ​​with modulus values ​​greater than the set modulus threshold are determined as target estimation values, and principal component analysis is performed on the screened target estimation values. It can be understood that the larger the modulus value of the angular velocity estimation value, the larger the rotation amplitude of the image acquisition device at the corresponding moment.

[0098] In some embodiments, the principal component analysis process can be expressed as follows:

[0099] ω α ,ω β ,ω γ = eigvec(Ω T Ω)

[0100] Among them, ω α ,ω β ,ω γ Respectively represent the eigenvectors corresponding to the three rotation directions, ω α Represents the eigenvector corresponding to the main rotation direction, ω β Represents the eigenvector corresponding to the secondary rotation direction, ω γ Indicates the eigenvector corresponding to the minimum rotation direction. Eigvec represents eigenvalue decomposition, and Ω is expressed as follows:

[0101]

[0102] Among them, ω0……ω N The N angular velocity estimates corresponding to the N moments whose modulus values ​​are greater than the set modulus threshold are the N target estimates, ω μRepresents the mean of the angular velocity estimates corresponding to the N moments.

[0103] Determine the above ω α ,ω β ,ω γ After that, the three rotation directions are determined. Then, the angular velocity estimation value corresponding to each moment can be projected to the three rotation directions respectively, and according to the projection results to the three rotation directions at each moment, it can be determined whether the image acquisition device is in a non-critical motion state at that moment.

[0104] It can be understood that if the projection results of the angular velocity estimation value at a certain moment in the three rotation directions are all small or 0, it can be considered that the movement of the image acquisition device at that moment may be critical movement, and the visual odometer data corresponding to that moment will not be selected. If the projection result of the angular velocity estimation value at a certain moment in any one of the three rotation directions is large, it can be considered that the movement of the image acquisition device at that moment may not be critical movement, and the visual odometer data corresponding to that moment can be screened out.

[0105] In an optional implementation of this embodiment, the step of determining whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value based on the projection result of the angular velocity estimation value in each rotation direction further includes the following steps:

[0106] Compare the projection result of each of the angular velocity estimation values ​​in each of the rotation directions with a set projection threshold;

[0107] According to the comparison result, it is determined whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value.

[0108] In this optional implementation, a projection threshold may be preset, and the projection result of the angular velocity estimation value at any moment in any rotation direction may be compared with the set projection threshold, and whether the image acquisition device is in a non-critical motion state at that moment may be determined based on the comparison result.

[0109] In some embodiments, if the projection result of the angular velocity estimation value at any time in any one of the three rotation directions is greater than or equal to the set projection threshold, it can be considered that the movement of the image acquisition device at that time is non-critical movement, and the visual odometer data corresponding to that time can be selected. If the projection results of the angular velocity estimation value at any time in the three rotation directions are all less than the set projection threshold, it can be considered that the movement of the image acquisition device at that time may be critical movement, and the visual odometer data corresponding to that time is not selected.

[0110] In an optional implementation of this embodiment, different rotation directions correspond to different set projection thresholds; and the step of determining, according to the comparison result, whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value further includes the following steps:

[0111] Screening out the projection results on any one of the three rotation directions, the angular velocity estimation value being greater than or equal to the set projection threshold value corresponding to the rotation direction, and determining the time corresponding to the screened angular velocity estimation value as the target time;

[0112] Determine that at the target moment and a fixed number of consecutive moments after the target moment, the image acquisition device is in a non-critical motion state. In this optional implementation, the set projection thresholds corresponding to the three different rotation directions may be different, the set projection threshold corresponding to the main rotation direction of the three different rotation directions is the largest, the set projection threshold in the secondary rotation direction is the second largest, and the set projection threshold in the minimum rotation direction is the smallest. The three different rotation directions correspond to different set projection thresholds, the purpose of which is to be able to evenly screen out the motion-related information corresponding to the moments where there are projections and the projection results are large in the above three different rotation directions.

[0113] When the projection result of the angular velocity estimation value at a certain moment in the main rotation direction is greater than the set projection threshold corresponding to the main rotation direction, the moment can be determined as the target moment, and the image acquisition device is in a non-critical motion state at the target moment and a fixed number of consecutive moments after the target moment, and then the visual odometer data corresponding to the target moment and a fixed number of consecutive moments after the target moment are added to the visual odometer data set corresponding to the main rotation direction.

[0114] When the projection result of the angular velocity estimation value at a certain moment in the secondary rotation direction is greater than the set projection threshold corresponding to the secondary rotation direction, the moment can be determined as the target moment, and the target moment and a fixed number of consecutive moments after the target moment are determined, and the image acquisition device is in a non-critical motion state, and then the visual odometer data corresponding to the target moment and a fixed number of consecutive moments after the target moment are added to the visual odometer data set corresponding to the secondary rotation direction.

[0115] When the projection result of the angular velocity estimation value at a certain moment in the minimum rotation direction is greater than the set projection threshold corresponding to the minimum rotation direction, the moment can be determined as the target moment, and the target moment and a fixed number of consecutive moments after the target moment are determined, and the image acquisition device is in a non-critical motion state, and then the visual odometer data corresponding to the target moment and a fixed number of consecutive moments after the target moment are added to the visual odometer data set corresponding to the minimum rotation direction.

[0116] Finally, the visual odometry data sets corresponding to the three rotation directions can be merged to obtain the final selected visual odometry data. That is, after the visual odometry data sets respectively selected for the three rotation directions are merged, the final selected visual odometry data can be obtained.

[0117] The following example illustrates a process of determining the target time based on the projection results:

[0118] First, we can use ω α ,ω β ,ω γ Construct the rotation matrix R ω :

[0119] R ω =[ω α ,ω β ,ω γ ] T

[0120] Secondly, for the estimated angular velocity at a certain time t0 calculate make:

[0121]

[0122] Finally, judge (θ is one of α, β, γ, indicating the current rotation direction) is greater than s θ ·ω thred ,s θ ·ω thred Set the projection threshold corresponding to the current rotation direction; if Greater than or equal to s θ ·ω thred , then time t0 is the target time, and the number of motion-related information groups M corresponding to the current rotation direction can be recorded θ =M θ +1, and the time t0 and its subsequent neighborhood The visual odometer data corresponding to the frame image is added to the information set corresponding to the current rotation direction

[0123] Where: θ∈{α,β,γ}, s θ In order to balance the difficulty of data collection in each rotation direction, the main rotation direction ω is taken in this embodiment. α The corresponding proportionality coefficient s α =1, secondary rotation direction ω β The corresponding proportionality coefficient s β =0.6, minimum rotation direction ω γ The corresponding proportionality coefficient sγ =0.4. represents a set {t} that satisfies In this embodiment, That is to say Indicates that the target time t is followed by moment. θ Indicates the rotation direction ω corresponding to θ α ,ω β ,ω γ The number of visual odometer data (or the number of target moments) selected is as follows: θ The more visual odometer data are screened, the higher the calibration accuracy of the image participating device internal parameters, but the slower the data collection speed. Therefore, the number of visual odometer data to be screened can be determined according to actual needs.

[0124] In some embodiments, the filtered visual odometry data set may be represented as follows:

[0125]

[0126] Among them, t represents the selected target time, represents the set of target time t, express The initial pose estimation data corresponding to each moment in the set, Represents a set of estimated 3D coordinates of image feature points, Indicates that the image feature point x can be observed i The image set or the time set corresponding to the image set, Represents the image feature point x on the image corresponding to time t i The two-dimensional coordinate observations of .

[0127] Figure 2 FIG. 2 is a schematic diagram showing a process flow of implementing camera intrinsic calibration according to an embodiment of the present disclosure. Figure 2 As shown in the figure, when a vehicle is driving on the road, the on-board camera continuously collects road images during driving, and the road images can be output to the visual odometer, which calculates the visual odometer data of the camera during the vehicle driving based on the road images. Other non-image sensors can also be installed on the vehicle in conjunction with the camera, and other non-image sensors can output non-image observation data during the vehicle driving. k}, such as GPS and IMU data. The initial value of the camera intrinsic parameter can be the factory parameter of the camera, or the calibration value of the intrinsic parameter of other cameras of the same model.

[0128] Based on the visual odometry data output by the visual odometry, non-critical motion trajectories are screened from the vehicle's driving trajectory. After enough visual odometry data corresponding to the non-critical motion trajectories are collected, a factor graph can be constructed based on the collected visual odometry data, the initial value of the camera's intrinsic parameters and the corresponding non-image observation data under the non-critical motion state, and the maximum a posteriori estimate of the camera's intrinsic parameters can be optimized and solved. The maximum a posteriori estimate can be determined as the calibration value of the camera's intrinsic parameters.

[0129] The following are embodiments of the apparatus of the present disclosure, which can be used to execute embodiments of the method of the present disclosure.

[0130] Figure 3 FIG. 1 shows a structural block diagram of a calibration device according to an embodiment of the present disclosure. The device can be implemented as part or all of an electronic device through software, hardware, or a combination of both. Figure 3 As shown, the calibration device comprises:

[0131] The first acquisition module 301 is configured to acquire the initial value of the internal parameter of the image acquisition device and the visual odometer data, wherein the visual odometer data is generated based on the target images captured at multiple consecutive moments during the movement of the image acquisition device; the visual odometer data includes image feature point data of the target image and initial pose estimation data of the image acquisition device when capturing the target image, wherein the image feature point data includes: the three-dimensional coordinate estimation value of the image feature point in the world coordinate system and the two-dimensional coordinate observation value on the target image;

[0132] The first screening module 302 is configured to screen out the visual odometer data of non-critical motion state from the visual odometer data based on the initial pose estimation data of the target images taken at adjacent moments;

[0133] The solving module 303 is configured to solve the estimated value of the intrinsic parameter of the image acquisition device based on at least the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data.

[0134] In this embodiment, the calibration device can be executed on a server or a terminal. The image acquisition device can be a monocular RGB camera mounted on a vehicle. The vehicle can be a data acquisition vehicle or a crowdsourcing vehicle suitable for collecting road data.

[0135] The initial value of the internal parameter of the image acquisition device can be recorded as in It is composed of the horizontal focal length, vertical focal length, optical center position and distortion coefficient of the image acquisition device. is the distortion parameter of the image acquisition device, which is related to the specific distortion model selected. In some embodiments, the distortion model can be Radtan, Equidistant, etc.

[0136] In some embodiments, the initial value of the intrinsic parameter of the image acquisition device can be determined by factory parameters, and / or calibration results of the same model of image acquisition device. Generally, the error of the initial value of the intrinsic parameter of the image acquisition device is large, and it is difficult to support high-precision visual tasks. Therefore, the embodiment of the present disclosure proposes a calibration device, which optimizes and estimates the initial value of the intrinsic parameter of the image acquisition device and the motion-related information generated by the image acquisition device during the motion process, and then obtains a more accurate estimate of the intrinsic parameter of the image acquisition device, which is used to calibrate the intrinsic parameter of the image acquisition device.

[0137] In some embodiments, the motion-related information generated by the image acquisition device during the motion process may include but is not limited to the visual odometer data of the image acquisition device. The visual odometer data may be generated based on the target images captured at multiple consecutive moments during the motion of the image acquisition device. The visual odometer data may include but is not limited to: image feature point data of the target image and the initial pose estimation data of the image acquisition device when capturing the target image; the image feature point data may include but is not limited to the three-dimensional coordinate estimation values ​​of multiple image feature points in the world coordinate system and the two-dimensional coordinate observation values ​​of the multiple image feature points on the target image, respectively. The target image is an image in which multiple image feature points can be observed in the images captured by the image acquisition device at multiple consecutive moments.

[0138] Visual odometry is a process of determining the position and direction of an image acquisition device by analyzing images acquired by the image acquisition device at multiple consecutive moments. It can be understood that in the embodiments of the present disclosure, the image acquisition device is mounted on a vehicle, and the movement process of the image acquisition device can be understood as the movement process of the vehicle.

[0139] Through the visual odometry system, the initial pose estimation data of the image acquisition device at each moment can be determined based on the images acquired by the image acquisition device at multiple consecutive moments. The three-dimensional coordinate estimation values ​​of the image feature points on the images acquired at multiple consecutive moments in the world coordinate system and the two-dimensional coordinate observation values ​​of the multiple image feature points on the images acquired at different moments can also be determined. It can be understood that the two-dimensional coordinate observation values ​​are the image coordinates on the images corresponding to each moment, and the two-dimensional coordinate observation values ​​of the same image feature point in the image acquisition environment are different on different images.

[0140] In the embodiment of the present disclosure, the initial pose estimation data of the image acquisition device can be obtained by using the visual odometer. Initial value of 3D coordinate estimation of image feature points And the matching results of image feature points in the world coordinate system and image coordinate system That is, the matching result between the estimated 3D coordinates of the same image feature point and the 2D coordinates observed in each image. Represents the mapping from the image feature point i to the j-th set of two-dimensional image points in the t-th frame image of all the two-dimensional coordinates of the image feature point that can be observed, that is, is the two-dimensional coordinate observation value in the t-th frame image.

[0141] In some embodiments, the visual odometer may adopt existing technologies, such as ORB-SLAM3, SVO, etc. In this embodiment, it may be implemented based on SVO.

[0142] In the disclosed embodiment, in order to prevent the motion trajectory of the image acquisition device at multiple consecutive moments from causing critical motion sequences (CMS), the visual odometer data corresponding to some moments that meet the preset conditions can be screened out from the visual odometer data corresponding to the target images acquired at multiple consecutive moments. The visual odometer data corresponding to some moments that meet the preset conditions can be understood as the visual odometer data corresponding to the moment when the image acquisition device is in a non-critical motion state. Under critical motion, the initial pose estimation data of the image acquisition device output by the visual odometer cannot be uniquely optimized to solve the intrinsic parameters of the image acquisition device, or the solved intrinsic parameter estimation value of the image acquisition device is greatly affected by noise and has poor accuracy.

[0143] In some embodiments, the motion trajectory of the image acquisition device at the selected partial moment does not constitute critical motion, that is, it is not in a critical motion state. Therefore, based on the initial pose estimation data of the image acquisition device at multiple consecutive moments output by the visual odometer, the visual odometer data corresponding to the partial moments that do not constitute a critical motion state can be screened out from the visual odometer data corresponding to the multiple consecutive moments. In some embodiments, the partial moments that meet the preset conditions can be understood as the camera intrinsic parameters of the image acquisition device with higher accuracy can be solved based on the initial pose estimation data of the image acquisition device at the partial moment, and the motion trajectory of the image acquisition device at the partial moment does not constitute a critical motion. It should be noted that the partial moment can be a part of the continuous moments of the above-mentioned multiple consecutive moments, that is, the partial moment can include multiple moments. In some embodiments, when determining whether the image acquisition device is in a critical motion state at the current moment, it can be determined based on the initial position estimation data of the previous moment of the adjacent moment and the current moment.

[0144] After filtering out the line-of-sight odometer data corresponding to some moments that meet the preset conditions, the estimated values ​​of the intrinsic parameters of the image acquisition device can be optimized based on the line-of-sight odometer data corresponding to the some moments and the initial values ​​of the intrinsic parameters of the image acquisition device. The solved estimated values ​​of the intrinsic parameters of the image acquisition device can be used to calibrate the internal parameters of the image acquisition device.

[0145] In this embodiment, after the visual odometer data corresponding to the motion trajectory of the image acquisition device in a non-critical motion state is screened, an optimization equation can be established based on the screened visual odometer data and the initial value of the intrinsic parameter of the image acquisition device, so that the estimated value of the intrinsic parameter of the image acquisition device with the smallest error is obtained through continuous iterative optimization. In the above process, the motion characteristics of the image acquisition device are analyzed, and the visual odometer data of the image acquisition device in a non-critical motion state are screened. Therefore, based on the screened visual odometer data, after the optimization equation is established, the internal parameters of the image acquisition device are optimized on the basis of the initial value of the internal parameters of the image acquisition device, and finally the estimated value of the internal parameter of the image acquisition device with higher accuracy can be obtained, and the estimated value can be used as the calibration value of the internal parameter of the image acquisition device.

[0146] In some embodiments, the objective function of the optimization equation can be established by referring to the existing technology, and no specific limitation is made here. For example, the estimated value of the intrinsic parameter of the image acquisition device can be obtained by establishing a factor graph and optimizing the maximum a posteriori estimation method of the intrinsic parameter of the image acquisition device.

[0147] In some embodiments, when optimizing and solving the intrinsic parameter estimation value of the image acquisition device, the two-dimensional coordinate observation values ​​of the image feature points in the visual odometer data corresponding to the screened partial moments are used as the observation data in the optimization objective function, and based on the projection equation of the three-dimensional coordinates of the image feature points in the world coordinate system to the two-dimensional image space corresponding to the partial moments, the three-dimensional coordinate estimation values ​​of the image feature points in the visual odometer data can be projected to the two-dimensional image space to obtain the two-dimensional coordinate estimation values ​​of the image feature points at these partial moments. These two-dimensional coordinate estimation values ​​are used as the estimation data in the objective function, and then the minimization of the error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of the same image feature point is used as the optimization goal, and finally the intrinsic parameter estimation value of the image acquisition device can be optimized and solved.

[0148] The disclosed embodiment proposes a device capable of automatically calibrating the internal parameters of an image acquisition device mounted on a vehicle, wherein the device obtains visual odometer data generated by the image acquisition device as the vehicle moves, and screens out the visual odometer data of the image acquisition device in a non-critical motion state, thereby optimizing and solving the estimated internal parameters of the image acquisition device based on the initial value of the internal parameters of the image acquisition device and the screened visual odometer data. In this way, since no human intervention is required, the automatic calibration of the internal parameters of the vehicle-mounted image acquisition device can be achieved, and the cost of the internal parameter calibration of the image acquisition device can be reduced. In addition, since the disclosed embodiment analyzes the motion characteristics of the image acquisition device during the calibration process and screens out the visual odometer data of the image acquisition device in a non-critical motion state, the calibration accuracy of the internal parameters of the image acquisition device is improved, and the calibration efficiency is improved.

[0149] In an optional implementation of this embodiment, if the image acquisition device is further fixedly connected to a non-image sensor, the apparatus further includes:

[0150] A second acquisition module is configured to acquire non-image observation data output by the non-image sensor during the movement of the image acquisition device;

[0151] A second screening module is configured to screen the non-image observation data of non-critical motion state from the non-image observation data;

[0152] The solution module is specifically implemented as follows:

[0153] Based on the initial value of the intrinsic parameter of the image acquisition device, the screened visual odometer data and the non-image observation data, the estimated value of the intrinsic parameter of the image acquisition device is solved.

[0154] In this optional implementation, the image acquisition device can be further fixedly connected to the non-image sensor, and obtain the non-image observation data output by the non-image sensor fixedly connected to the image acquisition device during the movement of the image acquisition device. In some embodiments, the non-image sensor, such as GPS, inertial measurement unit, etc., can be fixedly mounted on the vehicle together with the image acquisition device. In some embodiments, the image acquisition device and the non-image sensor can be fixedly connected to the image acquisition device and installed on the same vehicle. During the movement of the vehicle, the GPS can generate the geographic coordinates of the motion trajectory points of the image acquisition device, and the inertial measurement unit can measure the three-axis attitude angle and acceleration information of the image acquisition device during the movement. When optimizing and solving the internal parameter estimation value of the image acquisition device, the visual odometer data can be used as the state data to establish the observation equation, so that the non-image observation data of the image acquisition device estimated based on the visual odometer data is used as a factor in the optimization objective function, so that the optimization target further includes the minimum error between the non-image observation data measured by the non-image sensor, such as GPS, inertial measurement unit, etc. and the estimated value of the observation data estimated based on the visual odometer data.

[0155] That is to say, in other embodiments, non-image observation data of non-image sensors connected to the image acquisition device can also be added to the process of optimizing the intrinsic parameter estimation value of the image acquisition device, thereby improving the calibration accuracy of the intrinsic parameters of the image acquisition device. It is understandable that in the process of optimizing the intrinsic parameter estimation value of the image acquisition device, the addition of non-image observation data of non-image sensors is not necessary, but it can improve the accuracy of the final optimization result.

[0156] In an optional implementation of this embodiment, the solution module may be implemented as follows:

[0157] Based on the screened visual odometer data and the initial value of the internal parameter of the image acquisition device, the three-dimensional coordinate estimation values ​​of the multiple image feature points are mapped to the target image corresponding to the visual odometer data to obtain the two-dimensional coordinate estimation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data;

[0158] By optimizing the errors between the two-dimensional coordinate estimation values ​​and the two-dimensional coordinate observation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data, the maximum a posteriori estimation value of the intrinsic parameter of the image acquisition device is solved as the intrinsic parameter estimation value of the image acquisition device.

[0159] In this optional implementation, after the visual odometer data corresponding to the partial moments that meet the preset conditions are screened out, an optimization equation can be established to solve the internal parameter estimation value of the image acquisition device. In some embodiments, the optimization solution process can be represented by an objective function, and the optimization goal of the objective function is to minimize the error between the observed data and the estimated data at each moment in the partial moments corresponding to the screened visual odometer data.

[0160] In the disclosed embodiment, the observation data at each moment includes the two-dimensional coordinate observation value of each image feature point in the image of the visual odometer data corresponding to the moment, and the estimated data at each moment is the two-dimensional coordinate estimation value obtained after mapping the three-dimensional coordinate estimation value of each image feature point to the image acquired at the moment based on the visual odometer data and the initial value of the internal parameter of the image acquisition device. Optimizing the internal parameter estimation value of the image acquisition device can be understood as continuously optimizing and adjusting the initial pose estimation data, the three-dimensional coordinate estimation value of the image feature point in the world coordinate system, and the initial value of the internal parameter of the image acquisition device, so as to minimize the error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point. That is, the internal parameter of the image acquisition device corresponding to the minimum error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point is the internal parameter estimation value finally obtained.

[0161] When the internal parameters of the image acquisition device are known, the three-dimensional space point can be mapped to the two-dimensional image space through the following projection equation, that is, the three-dimensional coordinate estimation value of the same image feature point and the two-dimensional coordinate estimation value on a certain image can be expressed by the following projection equation:

[0162]

[0163] Among them, T t Represents the initial pose estimation data of the image acquisition device at time t, which is output by the visual odometer, including the rotation matrix R between the world coordinate system and the image coordinate system t and translation vectors represents the i-th 3D coordinate point in the world coordinate system, which corresponds to the 3D coordinate estimate of the i-th image feature point output by the visual odometer. It represents the projection position of the three-dimensional coordinate point i in the image coordinate system at time t, that is, the estimated two-dimensional coordinate value of the i-th image feature point on the image collected at time t.

[0164] In an optional implementation of this embodiment, based on the initial value of the intrinsic parameter of the image acquisition device, the screened visual odometer data and the non-image observation data, solving the estimated value of the intrinsic parameter of the image acquisition device can be implemented as follows:

[0165] Based on the screened visual odometer data and the initial value of the internal parameter of the image acquisition device, the three-dimensional coordinate estimation values ​​of the multiple image feature points are mapped to the target image corresponding to the visual odometer data to obtain the two-dimensional coordinate estimation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data;

[0166] Based on the selected visual odometer data and the initial value of the internal parameter of the image acquisition device, the acquisition time of the target image corresponding to the visual odometer data is expressed by an observation equation, and the non-image sensor estimation value of the non-image sensor of the image acquisition device is fixedly connected;

[0167] By optimizing the first error between the two-dimensional coordinate estimation value and the two-dimensional coordinate observation value of the multiple image feature points in the target image corresponding to the visual odometer data, and the second error between the non-image observation data of the non-image sensor fixedly connected to the image acquisition device at the acquisition time and the non-image sensor estimation value, the maximum a posteriori estimation value of the intrinsic parameter of the image acquisition device is solved as the intrinsic parameter estimation value.

[0168] In this optional implementation, after the visual odometer data corresponding to the partial moments that meet the preset conditions are screened out, an optimization equation can be established to solve the internal parameter estimation value of the image acquisition device. In some embodiments, the optimization solution process can be represented by an objective function, and the optimization goal of the objective function is to minimize the error between the observed data and the estimated data at each moment in the partial moments.

[0169] In the disclosed embodiment, the observation data at each moment includes the two-dimensional coordinate observation value of each image feature point in the image of the visual odometer data corresponding to the moment, and the non-image observation data at each moment output by the non-image sensor fixedly connected to the image acquisition device and installed on the vehicle; and the estimated data at each moment includes the two-dimensional coordinate estimation value obtained by mapping the three-dimensional coordinate estimation value of each image feature point to the target image acquired at the moment based on the visual odometer data and the initial value of the intrinsic parameter of the image acquisition device, and the non-image sensor estimation value expressed by the observation equation based on the visual odometer data and the initial value of the intrinsic parameter of the image acquisition device.

[0170] Optimizing the internal parameter estimation value of the image acquisition device can be understood as continuously optimizing and adjusting the initial internal parameter value and initial pose estimation data of the image acquisition device, and the three-dimensional coordinate estimation value of the image feature points in the world coordinate system, so as to minimize the sum of the first error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point, and the second error between the non-image observation data and the non-image sensor estimation value at each moment in the partial moment. That is, the sum of the first error between the two-dimensional coordinate observation value and the two-dimensional coordinate estimation value of each image feature point at each moment in the partial moment, and the second error between the observation data and the non-image sensor estimation value at each moment in the partial moment is minimized, and the corresponding internal parameter of the image acquisition device is the final internal parameter estimation value.

[0171] The following takes the construction of factor graph as an example to explain the objective function of solving the maximum a posteriori estimation of the intrinsic parameters of the image acquisition device:

[0172]

[0173] Among them, t is the partial time set At a certain moment in k is the non-image observation data output by the kth non-image sensor, Z k corresponds to z k The observation equation is used to solve the non-image sensor estimation value of the non-image sensor connected to the image acquisition device. K and d are the internal parameters of the image acquisition device. K is composed of the horizontal focal length, vertical focal length, optical center position and distortion coefficient of the image acquisition device. d is the distortion parameter. {T t} is the initial pose estimation data of the image acquisition device at time t, {x i} is the estimated three-dimensional coordinate value of the i-th image feature point, j is x i The j-th two-dimensional point in the corresponding image, For x i The 2D coordinate observations in the target image acquired at time t.

[0174] In some embodiments, the above objective function can be solved by an unconstrained nonlinear optimization problem, such as using the Levenberg-Marquardt algorithm. During the iterative optimization process, the initial value of the internal parameter of the image acquisition device is And the initial values ​​of the pose are the initial pose estimation data output by the visual odometer And the estimated three-dimensional coordinates of the image feature points Finally, the final solution K can be obtained by optimization. * ,d * , complete the online self-calibration of the internal parameters of the image acquisition device.

[0175] In an optional implementation of this embodiment, the adjacent time includes time t and time t-1, and the first screening module may be implemented as follows:

[0176] Calculate the angular velocity estimate of the image acquisition device at time t based on the initial pose estimation data of the target images captured at time t and time t-1;

[0177] Based on the angular velocity estimation value, visual odometer data of a non-critical motion state is filtered out.

[0178] In this optional implementation, the purpose of filtering out the visual odometry data corresponding to some moments that meet preset conditions from the motion-related information corresponding to multiple consecutive moments is to prevent the visual odometry data used in the objective function from being the data of the image acquisition device in a critical motion state when optimizing the internal parameters of the image acquisition device. As described above, the internal parameters of the image acquisition device are optimized based on the visual odometry data of the image acquisition device in a critical motion state, and the final result is greatly affected by noise and has low accuracy.

[0179] To this end, the embodiments of the present disclosure can calculate the angular velocity estimation value of the image acquisition device at each moment based on the initial pose estimation data of the target images taken at adjacent moments in the screened visual odometer data in the non-critical motion state.

[0180] In some embodiments, the angular velocity estimate may be based on the initial pose estimate data T of the image acquisition device at the current time t output by the visual odometer. t And the initial pose estimation data T of the image acquisition device at the previous time t-1 t-1 , the estimated angular velocity ω at time t is calculated by the following formula: t :

[0181]

[0182] Among them, τ t represents the real time corresponding to time t, τ t-1 Indicates the real time corresponding to time t-1, log(·) logarithmic mapping. R t and R t-1 are the rotation matrices at time t and t-1 respectively.

[0183] Angular velocity can be used to identify the rotation direction and rotation amplitude of the image acquisition device during movement.

[0184] After determining the angular velocity estimation value of the image acquisition device at each moment, it is possible to determine whether the image acquisition device is in a critical motion state at the current moment based on the rotation of each angular velocity estimation value in the rotation direction. If it is not in a critical motion state, then the image acquisition device is not in a critical motion state at the current moment and multiple consecutive moments thereafter, so that the visual odometer data at these moments can be screened out. If the image acquisition device is in a critical motion state at the current moment, the visual odometer data corresponding to the current moment does not meet the preset conditions and can be discarded, so as to continue to judge the next moment.

[0185] In some embodiments, if it is determined through the angular velocity estimation value that the image acquisition device rotates a small amount in the rotation direction at the current moment, it can be considered that the movement of the image acquisition device at the current moment may be critical movement, and the corresponding visual odometer data can be discarded. If it is determined through the angular velocity estimation value that the image acquisition device rotates a large amount in the rotation direction at the current moment, it can be considered that the movement of the image acquisition device at the current moment may be non-critical movement, and the corresponding visual odometer data can be retained; it can also be considered that the rotation amplitude of the image acquisition device in the rotation direction at the subsequent multiple consecutive moments is also large. Therefore, at the current moment and its subsequent multiple consecutive moments, the image acquisition device should be in a non-critical motion state, and the corresponding visual odometer data can be retained.

[0186] In an optional implementation of this embodiment, based on the angular velocity estimation value, filtering out the visual odometer data in a non-critical motion state may be implemented as follows:

[0187] Obtaining a modulus value of the angular velocity estimate;

[0188] Obtaining an angular velocity estimation value whose modulus value is greater than a set modulus value threshold as a target estimation value;

[0189] Performing principal component analysis on the target estimation value to determine three rotation directions of the image acquisition device during the movement; the three rotation directions are in an orthogonal relationship;

[0190] Projecting the estimated angular velocity value in the three rotation directions respectively to obtain a projection result of the estimated angular velocity value in each rotation direction;

[0191] Based on the projection result of the angular velocity estimation value in each rotation direction, determining whether the image acquisition device is in a non-critical motion state at the time corresponding to the angular velocity estimation value;

[0192] Based on the moment when the image acquisition device is in a non-critical motion state, the visual odometer data in the non-critical motion state is screened out.

[0193] In some embodiments, the rotation of the image acquisition device during the motion process can be represented by three rotation directions that are orthogonal to each other. The three rotation directions can include a main rotation direction, a secondary rotation direction, and a minimum rotation direction. In some embodiments, the rotation angles of the image acquisition device in the three rotation directions can be understood as yaw angle, pitch angle, and tilt angle.

[0194] In the disclosed embodiment, the three rotation directions are determined by performing principal component analysis on all calculated angular velocity estimation values, and the three rotation directions can be represented by corresponding vector features. In order to more accurately determine the three rotation directions, principal component analysis can be performed based on angular velocity estimation values ​​with larger modulus values. Therefore, a set modulus threshold can be set in advance, and the angular velocity estimation values ​​with modulus values ​​greater than the set modulus threshold are determined as target estimation values, and principal component analysis is performed on the screened target estimation values. It can be understood that the larger the modulus value of the angular velocity estimation value, the larger the rotation amplitude of the image acquisition device at the corresponding moment.

[0195] In some embodiments, the principal component analysis process can be expressed as follows:

[0196] ω α ,ω β ,ω γ = eigvec(Ω T Ω)

[0197] Among them, ω α ,ω β ,ω γ Respectively represent the eigenvectors corresponding to the three rotation directions, ω α Represents the eigenvector corresponding to the main rotation direction, ω β Represents the eigenvector corresponding to the secondary rotation direction, ω γ Indicates the eigenvector corresponding to the minimum rotation direction. Eigvec represents eigenvalue decomposition, and Ω is expressed as follows:

[0198]

[0199] Among them, ω0……ω N The N angular velocity estimates corresponding to the N moments whose modulus values ​​are greater than the set modulus threshold are the N target estimates, ω μ Represents the mean of the angular velocity estimates corresponding to the N moments.

[0200] Determine the above ω α ,ω β ,ω γAfter that, the three rotation directions are determined. Then, the angular velocity estimation value corresponding to each moment can be projected to the three rotation directions respectively, and according to the projection results to the three rotation directions at each moment, it can be determined whether the image acquisition device is in a non-critical motion state at that moment.

[0201] It can be understood that if the projection results of the angular velocity estimation value at a certain moment in the three rotation directions are all small or 0, it can be considered that the movement of the image acquisition device at that moment may be critical movement, and the visual odometer data corresponding to that moment will not be selected. If the projection result of the angular velocity estimation value at a certain moment in any one of the three rotation directions is large, it can be considered that the movement of the image acquisition device at that moment may not be critical movement, and the visual odometer data corresponding to that moment can be screened out.

[0202] In an optional implementation of this embodiment, determining whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value based on the projection result of the angular velocity estimation value in each rotation direction may be implemented as follows:

[0203] Compare the projection result of each of the angular velocity estimation values ​​in each of the rotation directions with a set projection threshold;

[0204] According to the comparison result, it is determined whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value.

[0205] In this optional implementation, a projection threshold may be preset, and the projection result of the angular velocity estimation value at any moment in any rotation direction may be compared with the set projection threshold, and whether the image acquisition device is in a non-critical motion state at that moment may be determined based on the comparison result.

[0206] In some embodiments, if the projection result of the angular velocity estimation value at any time in any one of the three rotation directions is greater than or equal to the set projection threshold, it can be considered that the movement of the image acquisition device at that time is non-critical movement, and the visual odometer data corresponding to that time can be selected. If the projection results of the angular velocity estimation value at any time in the three rotation directions are all less than the set projection threshold, it can be considered that the movement of the image acquisition device at that time may be critical movement, and the visual odometer data corresponding to that time is not selected.

[0207] In an optional implementation of this embodiment, different rotation directions correspond to different set projection thresholds; determining, according to the comparison result, whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value may be implemented as follows:

[0208] Screening out the projection results on any one of the three rotation directions, the angular velocity estimation value being greater than or equal to the set projection threshold value corresponding to the rotation direction, and determining the time corresponding to the screened angular velocity estimation value as the target time;

[0209] It is determined that at the target moment and a fixed number of consecutive moments after the target moment, the image acquisition device is in a non-critical motion state.

[0210] In this optional implementation, the projection thresholds corresponding to the three different rotation directions may be different, the projection threshold corresponding to the main rotation direction among the three different rotation directions is the largest, the projection threshold corresponding to the secondary rotation direction is the second largest, and the projection threshold corresponding to the smallest rotation direction is the smallest. The three different rotation directions correspond to different projection thresholds, in order to evenly screen out the motion-related information corresponding to the moments when there are projections in the three different rotation directions and the projection results are large.

[0211] When the projection result of the angular velocity estimation value at a certain moment in the main rotation direction is greater than the set projection threshold corresponding to the main rotation direction, the moment can be determined as the target moment, and the image acquisition device is in a non-critical motion state at the target moment and a fixed number of consecutive moments after the target moment, and then the visual odometer data corresponding to the target moment and a fixed number of consecutive moments after the target moment are added to the visual odometer data set corresponding to the main rotation direction.

[0212] When the projection result of the angular velocity estimation value at a certain moment in the secondary rotation direction is greater than the set projection threshold corresponding to the secondary rotation direction, the moment can be determined as the target moment, and the target moment and a fixed number of consecutive moments after the target moment are determined, and the image acquisition device is in a non-critical motion state, and then the visual odometer data corresponding to the target moment and a fixed number of consecutive moments after the target moment are added to the visual odometer data set corresponding to the secondary rotation direction.

[0213] When the projection result of the angular velocity estimation value at a certain moment in the minimum rotation direction is greater than the set projection threshold corresponding to the minimum rotation direction, the moment can be determined as the target moment, and the target moment and a fixed number of consecutive moments after the target moment are determined, and the image acquisition device is in a non-critical motion state, and then the visual odometer data corresponding to the target moment and a fixed number of consecutive moments after the target moment are added to the visual odometer data set corresponding to the minimum rotation direction.

[0214] Finally, the visual odometry data sets corresponding to the three rotation directions can be merged to obtain the final selected visual odometry data. That is, after the visual odometry data sets respectively selected for the three rotation directions are merged, the final selected visual odometry data can be obtained.

[0215] The following example illustrates a process of determining the target time based on the projection results:

[0216] First, we can use ω α ,ω β ,ω γ Construct the rotation matrix R ω :

[0217] R ω =[ω α ,ω β ,ω γ ] T

[0218] Secondly, for the estimated angular velocity at a certain time t0 calculate make:

[0219]

[0220] Finally, judge (θ is one of α, β, γ, indicating the current rotation direction) is greater than s θ ·ω thred ,s θ ·ω thred Set the projection threshold corresponding to the current rotation direction; if Greater than or equal to s θ ·ω thred , then time t0 is the target time, and the number of motion-related information groups M corresponding to the current rotation direction can be recorded θ =M θ +1, and the time t0 and its subsequent neighborhood The visual odometer data corresponding to the frame image is added to the information set corresponding to the current rotation direction

[0221] Where: θ∈{α,β,γ}, s θ In order to balance the difficulty of data collection in each rotation direction, the main rotation direction ω is taken in this embodiment. α The corresponding proportionality coefficient s α =1, secondary rotation direction ω β The corresponding proportionality coefficient s β =0.6, minimum rotation direction ω γ The corresponding proportionality coefficient s γ =0.4. represents a set {t} that satisfies In this embodiment, That is to say Indicates that the target time t is followed by moment. θ Indicates the rotation direction ω corresponding to θ α ,ω β ,ω γ The number of visual odometer data (or the number of target moments) selected is as follows: θ The more visual odometer data are screened, the higher the calibration accuracy of the image participating device internal parameters, but the slower the data collection speed. Therefore, the number of visual odometer data to be screened can be determined according to actual needs.

[0222] In some embodiments, the filtered visual odometry data set may be represented as follows:

[0223]

[0224] Among them, t represents the selected target time, represents the set of target time t, express The initial pose estimation data corresponding to each moment in the set, Represents a set of estimated 3D coordinates of image feature points, Indicates that the image feature point x can be observed i The image set or the time set corresponding to the image set, Represents the image feature point x on the image corresponding to time t i The two-dimensional coordinate observations of .

[0225] Figure 4 It is a structural schematic diagram of an electronic device suitable for implementing a calibration method according to an embodiment of the present disclosure.

[0226] like Figure 4As shown, the electronic device 400 includes a processing unit 401, which can be implemented as a processing unit such as a CPU, a GPU, an FPGA, an NPU, etc. The processing unit 401 can perform various processes in the embodiments of any of the above methods of the present disclosure according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage part 408 to a random access memory (RAM) 403. In RAM403, various programs and data required for the operation of the electronic device 400 are also stored. The processing unit 401, ROM402, and RAM403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0227] The following components are connected to the I / O interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 410 as needed, so that a computer program read therefrom is installed into the storage section 408 as needed.

[0228] In particular, according to an embodiment of the present disclosure, any method in the above referenced embodiment of the present disclosure may be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing any method in the embodiment of the present disclosure. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 409, and / or installed from a removable medium 411.

[0229] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0230] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0231] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the device described in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into the device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0232] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A calibration method, wherein: include: Acquire the initial value of the internal parameter and the visual odometer data of the image acquisition device, wherein the visual odometer data is generated based on the target images captured at multiple consecutive moments during the movement of the image acquisition device; the visual odometer data includes the image feature point data of the target image and the initial pose estimation data of the image acquisition device when capturing the target image, wherein the image feature point data includes: the three-dimensional coordinate estimation value of the image feature point in the world coordinate system and the two-dimensional coordinate observation value on the target image; Based on the initial pose estimation data of the target images taken at adjacent moments, the visual odometer data of the non-critical motion state is screened out from the visual odometer data; An estimated value of the intrinsic parameter of the image acquisition device is solved based on at least an initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data.

2. The method according to claim 1, wherein: If the image acquisition device is further fixedly connected to a non-image sensor, the method further comprises: Acquire non-image observation data output by the non-image sensor during the movement of the image acquisition device; Screening non-image observation data of non-critical motion state from the non-image observation data; At least based on the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data, solving the estimated value of the intrinsic parameter of the image acquisition device is specifically: Based on the initial value of the intrinsic parameter of the image acquisition device, the screened visual odometer data and the non-image observation data, the estimated value of the intrinsic parameter of the image acquisition device is solved.

3. The method according to claim 1, wherein: Solving the estimated value of the intrinsic parameter of the image acquisition device based at least on the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data includes: Based on the screened visual odometer data and the initial value of the internal parameter of the image acquisition device, the three-dimensional coordinate estimation values ​​of the multiple image feature points are mapped to the target image corresponding to the visual odometer data to obtain the two-dimensional coordinate estimation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data; By optimizing the errors between the two-dimensional coordinate estimation values ​​and the two-dimensional coordinate observation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data, the maximum a posteriori estimation value of the intrinsic parameter of the image acquisition device is solved as the intrinsic parameter estimation value of the image acquisition device.

4. The method according to claim 2, wherein: Based on the initial value of the intrinsic parameter of the image acquisition device, the screened visual odometer data and the non-image observation data, solving the estimated value of the intrinsic parameter of the image acquisition device, including: Based on the screened visual odometer data and the initial value of the internal parameter of the image acquisition device, the three-dimensional coordinate estimation values ​​of the multiple image feature points are mapped to the target image corresponding to the visual odometer data to obtain the two-dimensional coordinate estimation values ​​of the multiple image feature points in the target image corresponding to the visual odometer data; Based on the selected visual odometer data and the initial value of the internal parameter of the image acquisition device, the acquisition time of the target image corresponding to the visual odometer data is expressed by an observation equation, and the non-image sensor estimation value of the non-image sensor of the image acquisition device is fixedly connected; By optimizing the first error between the two-dimensional coordinate estimation value and the two-dimensional coordinate observation value of the multiple image feature points in the target image corresponding to the visual odometer data, and the second error between the non-image observation data of the non-image sensor fixedly connected to the image acquisition device at the acquisition time and the non-image sensor estimation value, the maximum a posteriori estimation value of the intrinsic parameter of the image acquisition device is solved as the intrinsic parameter estimation value.

5. The method according to any one of claims 1 to 4, wherein: The adjacent moments include moment t and moment t-1, and the initial pose estimation data based on the target image captured at the adjacent moments, and the visual odometer data of a non-critical motion state are screened out from the visual odometer data, including: Calculate the angular velocity estimate of the image acquisition device at time t based on the initial pose estimation data of the target images captured at time t and time t-1; Based on the angular velocity estimation value, visual odometer data of a non-critical motion state is filtered out.

6. The method according to claim 5, wherein: Based on the angular velocity estimation value, the visual odometer data of the non-critical motion state is screened out, including: Obtaining a modulus value of the angular velocity estimate; Obtaining an angular velocity estimation value whose modulus value is greater than a set modulus value threshold as a target estimation value; Performing principal component analysis on the target estimation value to determine three rotation directions of the image acquisition device during the movement; the three rotation directions are in an orthogonal relationship; Projecting the estimated angular velocity value in the three rotation directions respectively to obtain a projection result of the estimated angular velocity value in each rotation direction; Based on the projection result of the angular velocity estimation value in each rotation direction, determining whether the image acquisition device is in a non-critical motion state at the time corresponding to the angular velocity estimation value; Based on the moment when the image acquisition device is in a non-critical motion state, the visual odometer data in the non-critical motion state is screened out.

7. The method according to claim 6, wherein: Determining, based on the projection result of the angular velocity estimation value in each rotation direction, whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value, comprises: Compare the projection result of each of the angular velocity estimation values ​​in each of the rotation directions with a set projection threshold; According to the comparison result, it is determined whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value.

8. The method according to claim 7, wherein: Different rotation directions correspond to different set projection thresholds; Determining, according to the comparison result, whether the image acquisition device is in a non-critical motion state at the moment corresponding to the angular velocity estimation value comprises: Screening out the projection results on any one of the three rotation directions, the angular velocity estimation value being greater than or equal to the set projection threshold value corresponding to the rotation direction, and determining the time corresponding to the screened angular velocity estimation value as the target time; It is determined that at the target moment and a fixed number of consecutive moments after the target moment, the image acquisition device is in a non-critical motion state.

9. A calibration device, wherein: include: A first acquisition module is configured to acquire an initial value of an internal parameter of an image acquisition device and visual odometer data, wherein the visual odometer data is generated based on target images captured at multiple consecutive moments during the motion of the image acquisition device; the visual odometer data includes image feature point data of the target image and initial pose estimation data of the image acquisition device when capturing the target image, wherein the image feature point data includes: a three-dimensional coordinate estimation value of the image feature point in a world coordinate system and a two-dimensional coordinate observation value on the target image; A first screening module is configured to screen out visual odometer data of a non-critical motion state from the visual odometer data based on initial pose estimation data of target images captured at adjacent moments; The solution module is configured to solve the estimated value of the intrinsic parameter of the image acquisition device based on at least the initial value of the intrinsic parameter of the image acquisition device and the screened visual odometer data.

10. A computer-readable storage medium having computer instructions stored thereon, wherein: When the computer instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images