Camera parameter calibration method and device, electronic equipment and readable storage medium

Abstract

The application provides a camera parameter calibration method and device, electronic equipment and readable storage medium, and relates to the technical field of data processing. The method comprises the following steps: acquiring inertial sensor data and image data of a target object at N first time points; acquiring first position data of the target object at each of the N first time points based on the inertial sensor data at the N first time points; performing position conversion on the image data at the N first time points based on a preset camera parameter set to obtain M sets of second position data; summing distances between second position data corresponding to the N first time points included in each set of second position data and first position data corresponding to the same first time points to obtain M sets of first error distances; and calibrating a preset camera parameter corresponding to the smallest first error distance in the M sets of first error distances as a target parameter of the camera. The application improves the flexibility of parameter calibration.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and specifically to a camera parameter calibration method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] To realize smart city construction, surveillance cameras are installed during the city's construction process to monitor various objects within the city and convert the monitored locations to a common world coordinate system to generate a 3D city model. In related technologies, to achieve this conversion, camera parameters need to be calibrated, using these parameters to transform the captured objects into world coordinates. However, in these technologies, parameter calibration for each camera requires either a specific landmark or specific camera movement. For scenarios where specific landmarks cannot be set up or where cameras are fixed, there is a lack of flexibility in camera parameter calibration due to the inability to use landmarks or move cameras for calibration.

[0003] It is evident that the relevant technologies suffer from poor flexibility in camera parameter calibration. Summary of the Invention

[0004] This invention provides a camera parameter calibration method, apparatus, electronic device, and readable storage medium to solve the problem of poor flexibility in camera parameter calibration in related technologies.

[0005] To solve the above problems, the present invention is implemented as follows:

[0006] In a first aspect, embodiments of the present invention provide a camera parameter calibration method, including:

[0007] Acquire inertial sensor data and camera image data of the target object at N first moments, wherein the target object is an object moving within the camera's field of view, and N is a positive integer greater than 1;

[0008] The first position data of the target object at each of the N first moments is obtained based on the inertial sensor data at the N first moments;

[0009] Based on a preset camera parameter set, the image data of the N first moments are transformed to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1.

[0010] The distances between the N second position data corresponding to the first time moment included in each group of second position data and the first position data corresponding to the same first time moment are summed to obtain M groups of first error distances;

[0011] The preset camera parameters corresponding to the smallest first error distance among the M groups of first error distances are calibrated as the target parameters of the camera.

[0012] Secondly, embodiments of the present invention also provide a camera parameter calibration device, comprising:

[0013] The first acquisition module is used to acquire inertial sensor data and camera image data of the target object at N first moments, wherein the target object is an object moving within the camera's field of view, and N is a positive integer greater than 1.

[0014] The second acquisition module is used to acquire the first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments;

[0015] The position conversion module is used to perform position conversion on the image data of the N first moments based on a preset camera parameter set to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1.

[0016] The processing module is used to sum the distances between the N second position data corresponding to the first time moment included in each group of second position data and the first position data corresponding to the same first time moment to obtain M groups of first error distances;

[0017] The calibration module is used to calibrate the preset camera parameters corresponding to the smallest first error distance among the M groups of first error distances as the target parameters of the camera.

[0018] Thirdly, embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the camera parameter calibration method described in the first aspect above.

[0019] Fourthly, embodiments of the present invention also provide a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the camera parameter calibration method described in the first aspect above.

[0020] In this embodiment of the invention, inertial sensor data and image data captured by a camera are obtained at N first moments. The target object is an object moving within the camera's field of view, and N is a positive integer greater than 1. Based on the inertial sensor data at each of the N first moments, first position data of the target object at each of the N first moments is obtained. Based on a preset camera parameter set, the image data at the N first moments are transformed to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the image data at each of the N first moments. The corresponding second position data, the first position data, and M sets of second position data are position data in the same coordinate system, where M is a positive integer greater than 1; the distances between the N second position data corresponding to the first time moment included in each set of second position data and the first position data corresponding to the same first time moment are summed to obtain M sets of first error distances; the preset camera parameter corresponding to the smallest first error distance in the M sets of first error distances is calibrated as the target parameter of the camera, thereby realizing the calibration of camera parameters. Camera parameter calibration does not require a specific calibration object or camera movement, and can be flexibly applied to different scenarios, thus improving the flexibility of camera parameter calibration. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of a camera parameter calibration method provided in an embodiment of the present invention;

[0023] Figure 2 This is a schematic diagram of the first location data provided in an embodiment of the present invention;

[0024] Figure 3 This is a schematic diagram of image data provided in an embodiment of the present invention;

[0025] Figure 4 This is a schematic diagram of displacement calculation provided in an embodiment of the present invention;

[0026] Figure 5 This is one of the schematic diagrams illustrating the relationship between the fourth position data and the fifth position data provided in the embodiments of the present invention;

[0027] Figure 6 This is the second schematic diagram illustrating the relationship between the fourth and fifth position data provided in this embodiment of the invention;

[0028] Figure 7 This is a schematic diagram of target object detection provided in an embodiment of the present invention;

[0029] Figure 8 This is a structural diagram of a camera parameter calibration device provided in an embodiment of the present invention;

[0030] Figure 9 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] Please see Figure 1 , Figure 1 This is a flowchart of a camera parameter calibration method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes the following steps:

[0033] Step 101: Obtain inertial sensor data and camera image data of the target object at N first moments, wherein the target object is an object moving within the camera's field of view, and N is a positive integer greater than 1.

[0034] The target object mentioned above is an object that moves within the camera's field of view, and the target object is equipped with an inertial sensor to acquire inertial sensor data. The target object is usually a pedestrian carrying an inertial sensor or a mobile device carrying an inertial sensor.

[0035] The aforementioned image data includes the target object, and the image data includes the position of the target object. It should be noted that the N first-moment image data are image data captured by the camera under the same parameter conditions, including focal length and pose parameters.

[0036] The aforementioned N first moments are moments within a continuous time period. During this continuous time period, the target object moves within the camera's field of view. The camera captures images of the target object to obtain N image data points; and the inertial sensor acquires N inertial sensor data points.

[0037] In this system, the time of image data capture corresponds one-to-one with the time of inertial sensor acquisition, in order to reduce errors caused by displacement of the target object due to time inconsistencies and improve the accuracy of parameter calibration.

[0038] Step 102: Based on the inertial sensor data of the N first moments, obtain the first position data of the target object at each of the N first moments.

[0039] The aforementioned first position data is position data acquired based on inertial sensor data. Specifically, it involves acquiring the movement within N first moments using inertial sensors, including the direction and distance of movement. Then, using the direction and distance of movement, the first position data corresponding to each first moment is obtained. Here, the direction and distance of movement refer to the target object's direction and distance of movement in the world coordinate system.

[0040] It should be noted that the method of obtaining the first position data corresponding to each first moment by means of movement direction and movement distance varies depending on the shooting scenario. In outdoor scenarios, the first position data is obtained by means of the satellite positioning data of the target object, as well as the movement direction and movement distance; while in indoor scenarios, since the satellite positioning data of the target object cannot be obtained, the first position data needs to be obtained by means of preset points, as detailed in the following embodiments.

[0041] Step 103: Perform position transformation on the image data of the N first moments based on the preset camera parameter set to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1.

[0042] The aforementioned preset camera parameter set includes M sets of preset camera parameters, representing the range of camera parameters that can exist. These M sets of preset camera parameters include the target parameters for the current camera. The preset camera parameter set is used to perform position transformation on N image data points at first moments, converting the positions of the N target objects in the images at those first moments into second position data in the world coordinate system. This ensures that the first and second position data are in the same coordinate system, i.e., both are in the world coordinate system. Then, using the first and second position data, a set of parameters is selected from the M sets of preset camera parameters as the target parameters for the current camera for calibration.

[0043] The preset camera parameter set includes M sets of preset camera parameters. Based on each set of preset parameters, the image data of N first moments are transformed to obtain M sets of second position data. Each set of second position data includes N second position data.

[0044] Step 104: Sum the distances between the N second position data corresponding to the first time moment included in each group of second position data and the first position data corresponding to the same first time moment to obtain M groups of first error distances.

[0045] Step 105: Calibrate the preset camera parameters corresponding to the smallest first error distance in the M groups of first error distances as the target parameters of the camera.

[0046] The aforementioned first error distance is the sum of the distances between a set of second position data and the first position data at each of the N first time moments, i.e., the sum of the N distances. The first error distance obtained by summing these distances characterizes the error between the set of second position data and the first position data, and thus the error of the preset camera parameters corresponding to that set of second position data. Specifically, the larger the first error distance, the greater the difference between the corresponding preset camera parameters and the actual parameters of the current camera; the smaller the first error distance, the smaller the difference between the corresponding preset camera parameters and the actual parameters of the current camera. After obtaining M sets of first error distances, the smallest first error distance among the M sets is determined. This smallest difference between the preset camera parameters corresponding to this first error distance and the actual parameters of the current camera is calibrated as the target parameters of the camera.

[0047] Among them, the first position data of the target object at N first moments are as follows: Figure 2 As shown, the image data is as follows Figure 3 As shown, it includes the position data of N target objects in the image at the first moment. By... Figure 3 The position data of the N target objects in the image at the first moment are transformed to obtain the second position data, which is then compared with... Figure 2 By comparing the first position data shown, the target parameters of the camera can be obtained from the M sets of preset camera parameters.

[0048] Furthermore, the target object moves in a zigzag pattern to improve the accuracy of obtaining the first and second position data.

[0049] In this embodiment of the invention, inertial sensor data and image data captured by a camera are obtained at N first moments. The target object is an object moving within the camera's field of view, and N is a positive integer greater than 1. Based on the inertial sensor data at each of the N first moments, first position data of the target object at each of the N first moments is obtained. Based on a preset camera parameter set, the image data at the N first moments are transformed to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the image data at each of the N first moments. The corresponding second position data, the first position data, and M sets of second position data are position data in the same coordinate system, where M is a positive integer greater than 1. The distances between the N second position data corresponding to the first time moment in each set of second position data and the first position data corresponding to the same first time moment are summed to obtain M sets of first error distances. The preset camera parameter corresponding to the smallest first error distance in the M sets of first error distances is calibrated as the target parameter of the camera, thus achieving camera parameter calibration. This calibration does not require a specific calibration object or camera movement, allowing for flexible application in different scenarios and improving the flexibility of camera parameter calibration. Simultaneously, for scenarios where cameras have already been deployed and installed, the method of this embodiment can achieve rapid calibration; the calibration of a single camera can be completed within minutes, greatly improving calibration efficiency.

[0050] In one embodiment, the inertial sensor data includes quaternions and acceleration data, and the step of obtaining the first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments includes:

[0051] The quaternion at each first moment is calculated based on the first preset formula to obtain the first heading angle at each first moment;

[0052] Based on the number of change cycles of acceleration data between each first moment and the previous first moment adjacent to each first moment, the number of walking steps at each first moment is determined, and one number of walking steps corresponds to one change cycle of the acceleration data;

[0053] The first heading angle, the number of steps taken, and the set step length at each first moment are calculated based on trigonometric functions to obtain the displacement of each first moment in the world coordinate system.

[0054] The first position data at each first moment is obtained based on the displacement in the world coordinate system at each first moment;

[0055] The first preset formula is expressed as follows:

[0056]

[0057] Ψ is the first heading angle, and q0, q1, q2 and q3 are the quaternions.

[0058] The inertial sensor data output by the aforementioned inertial sensor is in complex extended form, specifically denoted as q = q0 + iq1 + jq2 + kq3, including quaternions q0, q1, q2, and q3. The pitch, yaw, and roll angles can be calculated using the quaternion-to-Euler angle formula. It should be noted that, in this embodiment of the invention, to reduce computational load, the inertial sensor is placed horizontally during the target object's movement. At this time, both the pitch and roll angles are 0 degrees, and only the yaw angle needs to be calculated. That is, the first yaw angle at each first moment is obtained by calculating the quaternion at each first moment using the first preset formula.

[0059] After calculating the first yaw angle, the distance traveled between the two first moments needs to be determined. For the moving target object, the displacement at each first moment can be determined directly by multiplying time and velocity, or it can be determined by the target object's movement pattern. During the target object's movement, its center of gravity changes periodically, meaning its acceleration in the Z-axis direction changes periodically. By calculating the period of acceleration change between the two first moments, the number of steps the target object takes can be determined. Finally, multiplying the number of steps by the set step length yields the displacement at each first moment.

[0060] Among them, such as Figure 4 As shown, in the process of calculating displacement, the displacement in the X-axis direction and the displacement in the Y-axis direction are determined by trigonometric functions.

[0061] The above method obtains the first position data of each first moment based on the displacement in the world coordinate system at each first moment. That is, the first position data of each first moment is obtained by the displacement in the world coordinate system at each first moment under different shooting scenarios. See the following embodiments for details.

[0062] In this embodiment of the invention, the first heading angle of each first moment is obtained by calculating the quaternion of each first moment based on a first preset formula; the number of walking steps at each first moment is determined based on the number of change cycles of acceleration data between each first moment and the previous first moment adjacent to each first moment, with one walking step corresponding to one change cycle of acceleration data; the displacement of each first moment in the world coordinate system is obtained by calculating the first heading angle, the number of walking steps, and the set step length at each first moment based on trigonometric functions, thereby determining the displacement of the target object at each first moment, and then obtaining the first position data of each first moment based on the displacement of each first moment in the world coordinate system.

[0063] In one embodiment, obtaining the first position data at each first moment based on the displacement in the world coordinate system at each first moment includes:

[0064] Obtain the satellite positioning data of the target object at each first moment;

[0065] Based on the earliest satellite positioning data among the N first moments and the displacement of each first moment in the world coordinate system, dead reckoning data for each first moment is generated;

[0066] Based on preset processing parameters, the dead reckoning data at each first moment is transformed to obtain the first intermediate position data at each first moment;

[0067] The distance between the first intermediate position data and the satellite positioning data at each first moment is averaged to obtain the second error distance at each first moment;

[0068] Delete the satellite positioning data in each first moment where the second error distance is greater than a preset error distance threshold to obtain the first location data.

[0069] The earliest position in the dead reckoning data of the first moment mentioned above is the position corresponding to the earliest satellite positioning data among N first moments. By using the displacement of each first moment in the world coordinate system and the position corresponding to the earliest satellite positioning data among N first moments, position data including each of the N first moments is generated, that is, dead reckoning data for each first moment.

[0070] It should be noted that satellite positioning data contains a certain degree of error. Therefore, it is necessary to filter the satellite positioning data, removing data with significant errors to improve the accuracy of the position data. Since dead reckoning data is generated based on displacement and has a certain rotational and translational relationship with satellite positioning data, dead reckoning data can be used to filter the satellite positioning data.

[0071] Specifically, the rotation and translation relationship between dead reckoning data and satellite positioning data is identified by preset processing parameters, including rotation angle and translation amount. Based on these parameters, the dead reckoning data for each first time point is transformed to obtain the first intermediate position data for each first time point. This first intermediate position data serves as reference data based on the dead reckoning data and can be used to optimize the satellite positioning data. If the second error distance between the first intermediate position data and the satellite positioning data is greater than a preset error distance threshold, the satellite positioning data is considered to have a large error and needs to be deleted. Conversely, if the second error distance is less than or equal to the preset error distance threshold, the satellite positioning data is considered acceptable as the first position data for calibrating camera parameters.

[0072] In this embodiment of the invention, satellite positioning data of the target object at each first moment is acquired; dead reckoning data for each first moment is generated based on the earliest satellite positioning data among N first moments and the displacement in the world coordinate system at each first moment; position transformation is performed on the dead reckoning data for each first moment based on preset processing parameters to obtain the first intermediate position data for each first moment; the distance between the first intermediate position data and the satellite positioning data at each first moment is averaged to obtain the second error distance for each first moment; satellite positioning data at each first moment whose second error distance is greater than a preset error distance threshold is deleted to obtain the first position data. This achieves the acquisition of the first position data through satellite positioning data and the displacement in the world coordinate system at each first moment, and deletes satellite positioning data with large errors, thereby improving the accuracy of the first position data.

[0073] In one embodiment, the preset error distance threshold and the preset processing parameters are obtained in the following manner:

[0074] Obtain a preset processing parameter set, which includes multiple sets of processing parameters, each set of processing parameters including a rotation angle and a translation vector;

[0075] Based on the rotation angle and translation vector of each set of processing parameters, the dead reckoning data at each first moment is transformed to obtain the second intermediate position data corresponding to each set of processing parameters;

[0076] The distance between the satellite positioning data at each first moment and the second intermediate position data corresponding to each set of processing parameters is averaged to obtain the third error distance corresponding to each set of processing parameters;

[0077] The smallest third error distance among the third error distances corresponding to each set of processing parameters is set as the preset error distance threshold, and the processing parameter corresponding to the smallest third error distance is set as the preset processing parameter.

[0078] It should be noted that for different sets of satellite positioning data, the rotation and translation relationship between the dead reckoning data generated based on it and the satellite positioning data also differs. New preset processing parameters need to be obtained for each set of satellite positioning data.

[0079] The above-mentioned set of preset processing parameters includes multiple sets of processing parameters, among which there are preset processing parameters. In this embodiment of the invention, the satellite positioning data is transformed by multiple sets of processing parameters to obtain the second intermediate position data, and the third error distance corresponding to each set of processing parameters is calculated to determine the preset processing parameters and the preset error distance threshold.

[0080] Specifically, suppose a set of processing parameters has a rotation angle of theta and a translation vector of theta. <move x move y The coordinates of the dead reckoning point sequence are transformed to the world coordinate system based on the rotation angle and translation vector. The specific formula is as follows:

[0081] x jp’ =(x jp -move x )*cos(theta)-(y jp -move y sin(theta)

[0082] y jp’ =(x jp -move x )*sin(theta)+(y jp -move y )*cos(theta)

[0083] Where, x jp y jp For satellite positioning data, x jp’ y jp’ This is the second intermediate position data. Then, the average distance between the satellite positioning data at each first moment and the second intermediate position data corresponding to each set of processing parameters is calculated to obtain the third error distance dist_error corresponding to that set of processing parameters. After calculating the third error distance corresponding to each set of processing parameters, the smallest third error distance among the third error distances corresponding to each set of processing parameters is set as a preset error distance threshold, and the processing parameter corresponding to the smallest third error distance is set as a preset processing parameter, so as to determine the first position data based on the preset processing parameters and the preset error distance threshold.

[0084] In one embodiment, obtaining the first position data at each first moment based on the displacement in the world coordinate system at each first moment includes:

[0085] Acquire third position data of multiple preset points, and the second time corresponding to each preset point of the target object passing through the multiple preset points, wherein the second time corresponding to each preset point is the same as one of the N first times, and the multiple preset points are points in the indoor space;

[0086] The first position data is generated based on the third position data of the multiple preset points and the displacement in the world coordinate system at each first moment, wherein the first position data at the second moment is the third position data of the preset point corresponding to the second moment.

[0087] It should be noted that the scenarios in which cameras are installed vary, and there are situations where it is impossible to locate the target object via satellite, such as indoor spaces. In such cases, it is impossible to obtain the first location data using satellite positioning data. Therefore, in this embodiment of the invention, the first location data is obtained by using preset points.

[0088] The preset points are locations within the indoor space. Their positions in the world coordinate system have been pre-confirmed, and the target object will pass through these preset points during its movement. Based on the third position data of multiple preset points and the displacement in the world coordinate system at each first moment, first position data is generated. That is, starting from each preset point, multiple continuous trajectories are generated, each containing the first position data. All trajectories are then connected in chronological order to form N sets of first position data at each first moment.

[0089] Furthermore, in this embodiment of the invention, preset points are used to calibrate the position determined based on displacement. The more preset points there are, the higher the accuracy of the first position data generated. In order to improve the accuracy of the final first position data, the number of preset points should be greater than or equal to three.

[0090] In this embodiment of the invention, by acquiring third position data of multiple preset points and the second time corresponding to each preset point where the target object passes through, wherein the second time corresponding to each preset point is the same as one of the N first times, and the multiple preset points are points in an indoor space; based on the third position data of the multiple preset points and the displacement in the world coordinate system at each first time, the first position data is generated, wherein the first position data at the second time is the third position data of the preset point corresponding to the second time. Thus, determining the first position data through preset points eliminates the need to acquire satellite positioning data of the target object, enabling application to more camera installation scenarios and further improving the flexibility of camera parameter calibration.

[0091] In one embodiment, the image data includes fourth position data of the target object in the image, and each of the M sets of preset camera parameters includes a pitch angle and a second heading angle. The position transformation based on the preset camera parameter set and the N first-time image data to obtain M sets of second position data includes:

[0092] Obtain the fifth position data of the camera in the world coordinate system, and the height of the camera;

[0093] Based on trigonometric functions, the preset camera parameters for each group, including the pitch angle, the height, and the image data at each first moment, are calculated to obtain M sets of third intermediate position data of the target object relative to the camera at each first moment;

[0094] Based on the fifth position data of the camera and the second heading angle included in each set of preset camera parameters, the position of the M sets of third intermediate position data at each first moment is transformed to obtain the M sets of second position data.

[0095] The pitch angle and the second heading angle mentioned above are the camera parameters to be calibrated. The fourth position data of the target object in the image is transformed by using M sets of preset camera parameters to obtain the third intermediate position data in the world coordinate system. Then, it is compared with the first position data to determine one set of preset camera parameters in the M sets of preset camera parameters as the target parameters of the camera.

[0096] For example, the relationship between the fourth position data of the target object in the image and the third intermediate position data in the world coordinate system is as follows: Figure 5 and Figure 6As shown, the coordinate system in the image is UO1V, the coordinate of the fourth position data on the V-axis is P1, and the coordinate of the fourth position data on the U-axis is Q1; the camera's location is O2; the world coordinate system is XO3Y, the first position data of the target object in the world coordinate system has the Y-axis coordinate P, and the first position data of the target object in the world coordinate system has the X-axis coordinate Q. Specifically, the fourth position data is converted into the Y-axis coordinate of the third intermediate position data in the world coordinate system based on trigonometric functions using the following formula:

[0097] Wy = H × tan(β + γ)

[0098] Where Wy is the Y-axis coordinate of the third intermediate data, γ is the top-down view, H is the camera height, and β is calculated using the following formula.

[0099] β=atan((cy-Q1y)÷fy

[0100] cy is the V-axis coordinate of O1 in the image, Q1y is the V-axis coordinate of Q1, and fy is the focal length coordinate on the V-axis.

[0101] The following formula converts the fourth position data into the X-axis coordinate of the third intermediate position data in the world coordinate system based on trigonometric functions:

[0102]

[0103] Where Wx is the X-axis coordinate of the third intermediate position data, d is the distance between the target object and the camera (i.e., the distance between point P and point O3), and α is obtained by the following formula:

[0104]

[0105] Q1x is the U-axis coordinate of Q1, cx is the U-axis coordinate of O1 in the image, and fx is the focal length coordinate on the U-axis.

[0106] After obtaining the third intermediate position data (Wx, Wy), the second position data (Wx', Wy') is calculated based on the second heading angle and the fifth position data (Mx, My) from the camera, using the following formula:

[0107] Wx'=Wx×cosθ+Wy×sinθ+Mx

[0108] Wy′=Wy×cosθ-Wx×sinθ+My

[0109] Where θ is the second heading angle.

[0110] In one embodiment, the fourth location data is the position of the bottom center of the detection box in the image after the detection box including the target object is identified in the image based on the convolutional neural network.

[0111] The target object mentioned above is identified through a neural network model, such as training on a general pedestrian dataset. Pedestrian detection is performed on images containing the target object, and identity re-identification technology is used to extract the pixel features of pedestrians within the detection bounding box. Similarity calculations are then performed between these pixel features and the pixel features of the target object to obtain the desired result. Figure 7 The detection box with the highest similarity exceeding the threshold. It should be noted that the target object's position is determined by the point of contact below it during localization; therefore, the pixel coordinates of the midpoint of the bottom edge of the detection box are calculated, and these coordinates are the fourth position data.

[0112] Please see Figure 8 , Figure 8 This is a structural diagram of a camera parameter calibration device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the camera parameter calibration device 800 includes:

[0113] The first acquisition module 801 is used to acquire inertial sensor data and camera image data of the target object at N first moments, wherein the target object is an object moving within the camera's field of view, and N is a positive integer greater than 1.

[0114] The second acquisition module 802 is used to acquire the first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments;

[0115] The position conversion module 803 is used to perform position conversion on the image data of the N first moments based on a preset camera parameter set to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1.

[0116] Processing module 804 is used to sum the distances between the N second position data corresponding to the first time moment included in each group of second position data and the first position data corresponding to the same first time moment to obtain M groups of first error distances;

[0117] The calibration module 805 is used to calibrate the preset camera parameters corresponding to the smallest first error distance in the M groups of first error distances as the target parameters of the camera.

[0118] In one embodiment, the inertial sensor data includes quaternions and acceleration data, and the second acquisition module 802 includes:

[0119] The first calculation unit is used to calculate the quaternion for each first moment based on the first preset formula to obtain the first heading angle for each first moment;

[0120] The processing unit is configured to determine the number of walking steps at each first moment based on the number of change cycles of the acceleration data between each first moment and the previous first moment adjacent to each first moment, wherein one walking step corresponds to one change cycle of the acceleration data;

[0121] The second calculation unit is used to calculate the first heading angle, the number of walking steps and the set step length at each first moment based on trigonometric functions, so as to obtain the displacement of each first moment in the world coordinate system.

[0122] The first acquisition unit is used to acquire the first position data of each first moment based on the displacement in the world coordinate system at each first moment;

[0123] The first preset formula is expressed as follows:

[0124]

[0125] Ψ is the first heading angle, and q0, q1, q2 and q3 are the quaternions.

[0126] In one embodiment, the first acquisition unit includes:

[0127] The first acquisition subunit is used to acquire the satellite positioning data of the target object at each first moment;

[0128] The first generation subunit is used to generate dead reckoning data for each of the N first moments based on the earliest satellite positioning data among the N first moments and the displacement of each first moment in the world coordinate system.

[0129] The position conversion subunit is used to perform position conversion on the dead reckoning data at each first moment based on preset processing parameters to obtain the first intermediate position data at each first moment.

[0130] The first processing subunit is used to average the distance between the first intermediate position data and the satellite positioning data at each first moment to obtain the second error distance at each first moment;

[0131] The second processing subunit is used to delete satellite positioning data in each first moment where the second error distance is greater than a preset error distance threshold, and obtain the first location data.

[0132] In one embodiment, the preset error distance threshold and the preset processing parameters are obtained in the following manner:

[0133] Obtain a preset processing parameter set, which includes multiple sets of processing parameters, each set of processing parameters including a rotation angle and a translation vector;

[0134] Based on the rotation angle and translation vector of each set of processing parameters, the dead reckoning data at each first moment is transformed to obtain the second intermediate position data corresponding to each set of processing parameters;

[0135] The distance between the satellite positioning data at each first moment and the second intermediate position data corresponding to each set of processing parameters is averaged to obtain the third error distance corresponding to each set of processing parameters;

[0136] The smallest third error distance among the third error distances corresponding to each set of processing parameters is set as the preset error distance threshold, and the processing parameter corresponding to the smallest third error distance is set as the preset processing parameter.

[0137] In one embodiment, the first acquisition unit includes:

[0138] The second acquisition subunit is used to acquire third position data of multiple preset points, and the second time corresponding to each preset point of the target object passing through the multiple preset points. The second time corresponding to each preset point is the same as one of the N first times. The multiple preset points are points in the indoor space.

[0139] The second generation subunit is used to generate the first position data based on the third position data of the plurality of preset points and the displacement in the world coordinate system at each first moment, wherein the first position data at the second moment is the third position data of the preset point corresponding to the second moment.

[0140] In one embodiment, the image data includes fourth position data of the target object in the image, each of the M sets of preset camera parameters includes a pitch angle and a second heading angle, and the position conversion module 803 includes:

[0141] The second acquisition unit is used to acquire the fifth position data of the camera in the world coordinate system, and the height of the camera;

[0142] The third calculation unit is used to calculate the preset camera parameters, including the pitch angle, the height, and the image data at each first moment, based on trigonometric functions, to obtain M sets of third intermediate position data of the target object relative to the camera at each first moment;

[0143] The position conversion unit is used to perform position conversion on the M sets of third intermediate position data at each first moment based on the fifth position data of the camera and the second heading angle included in each set of preset camera parameters, to obtain the M sets of second position data.

[0144] In one embodiment, the fourth location data is the position of the bottom center of the detection box in the image after the detection box including the target object is identified in the image based on the convolutional neural network.

[0145] The camera parameter calibration device provided in this embodiment of the invention can realize each process of each embodiment of the above-described camera parameter calibration method. The technical features are one-to-one and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0146] It should be noted that the camera parameter calibration device in the embodiments of the present invention can be a device, or it can be a component, integrated circuit, or chip in an electronic device.

[0147] This invention also provides an electronic device, see [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device includes a memory 901, a processor 902, and a program or instructions stored in the memory 901 that run on the processor. When the program or instructions are executed by the processor 902, they can achieve the following: Figure 1 Any steps in the corresponding method embodiments and the achievement of the same beneficial effects will not be repeated here.

[0148] The processor 902 can be a CPU, ASIC, FPGA, or GPU.

[0149] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by hardware related to program instructions, and the program can be stored in a readable medium.

[0150] This invention also provides a readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described functions. Figure 1Any step in the corresponding method embodiment, which achieves the same technical effect, will not be described again here to avoid repetition. The storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0151] In the embodiments of this invention, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: A alone, B alone, C alone, both A and B present, both B and C present, both A and C present, and A, B, and C present.

[0152] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0153] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or second terminal device, etc.) to execute the methods of the various embodiments of this application.

[0154] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for calibrating camera parameters, characterized in that, include: Acquire inertial sensor data and camera image data of the target object at N first moments, wherein the target object is an object moving within the camera's field of view, and N is a positive integer greater than 1; The first position data of the target object at each of the N first moments is obtained based on the inertial sensor data at the N first moments; Based on a preset camera parameter set, the image data of the N first moments are transformed to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1. The distances between the N second position data corresponding to the first time moment included in each group of second position data and the first position data corresponding to the same first time moment are summed to obtain M groups of first error distances; The preset camera parameters corresponding to the smallest first error distance among the M groups of first error distances are calibrated as the target parameters of the camera.

2. The method as described in claim 1, characterized in that, The inertial sensor data includes quaternions and acceleration data. The step of obtaining the first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments includes: The quaternion at each first moment is calculated based on the first preset formula to obtain the first heading angle at each first moment; Based on the number of change cycles of acceleration data between each first moment and the previous first moment adjacent to each first moment, the number of walking steps at each first moment is determined, and one number of walking steps corresponds to one change cycle of the acceleration data; The first heading angle, the number of steps taken, and the set step length at each first moment are calculated based on trigonometric functions to obtain the displacement of each first moment in the world coordinate system. The first position data at each first moment is obtained based on the displacement in the world coordinate system at each first moment; The first preset formula is expressed as follows: Ψ is the first heading angle, and q0, q1, q2 and q3 are the quaternions.

3. The method as described in claim 2, characterized in that, The step of obtaining the first position data for each first moment based on the displacement in the world coordinate system at each first moment includes: Obtain the satellite positioning data of the target object at each first moment; Based on the earliest satellite positioning data among the N first moments and the displacement of each first moment in the world coordinate system, dead reckoning data for each first moment is generated; Based on preset processing parameters, the dead reckoning data at each first moment is transformed to obtain the first intermediate position data at each first moment; The distance between the first intermediate position data and the satellite positioning data at each first moment is averaged to obtain the second error distance at each first moment; Delete the satellite positioning data in each first moment where the second error distance is greater than a preset error distance threshold to obtain the first location data.

4. The method as described in claim 3, characterized in that, The preset error distance threshold and the preset processing parameters are obtained in the following way: Obtain a preset processing parameter set, which includes multiple sets of processing parameters, each set of processing parameters including a rotation angle and a translation vector; Based on the rotation angle and translation vector of each set of processing parameters, the dead reckoning data at each first moment is transformed to obtain the second intermediate position data corresponding to each set of processing parameters; The distance between the satellite positioning data at each first moment and the second intermediate position data corresponding to each set of processing parameters is averaged to obtain the third error distance corresponding to each set of processing parameters; The smallest third error distance among the third error distances corresponding to each set of processing parameters is set as the preset error distance threshold, and the processing parameter corresponding to the smallest third error distance is set as the preset processing parameter.

5. The method as described in claim 2, characterized in that, The step of obtaining the first position data for each first moment based on the displacement in the world coordinate system at each first moment includes: Acquire third position data of multiple preset points, and the second time corresponding to each preset point of the target object passing through the multiple preset points, wherein the second time corresponding to each preset point is the same as one of the N first times, and the multiple preset points are points in the indoor space; The first position data is generated based on the third position data of the multiple preset points and the displacement in the world coordinate system at each first moment, wherein the first position data at the second moment is the third position data of the preset point corresponding to the second moment.

6. The method as described in claim 1, characterized in that, The image data includes the fourth position data of the target object in the image. Each of the M sets of preset camera parameters includes a pitch angle and a second heading angle. The position transformation of the N first-time image data based on the preset camera parameter set yields M sets of second position data, including: Obtain the fifth position data of the camera in the world coordinate system, and the height of the camera; Based on trigonometric functions, the preset camera parameters for each group, including the pitch angle, the height, and the image data at each first moment, are calculated to obtain M sets of third intermediate position data of the target object relative to the camera at each first moment; Based on the fifth position data of the camera and the second heading angle included in each set of preset camera parameters, the position of the M sets of third intermediate position data at each first moment is transformed to obtain the M sets of second position data.

7. The method as described in claim 6, characterized in that, The fourth location data is the position of the bottom center of the detection box in the image after the detection box including the target object is identified in the image based on the convolutional neural network.

8. A camera parameter calibration device, characterized in that, include: The first acquisition module is used to acquire inertial sensor data and camera image data of the target object at N first moments, wherein the target object is an object moving within the camera's field of view, and N is a positive integer greater than 1. The second acquisition module is used to acquire the first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments; The position conversion module is used to perform position conversion on the image data of the N first moments based on a preset camera parameter set to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets includes the second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1. The processing module is used to sum the distances between the N second position data corresponding to the first time moment included in each group of second position data and the first position data corresponding to the same first time moment to obtain M groups of first error distances; The calibration module is used to calibrate the preset camera parameters corresponding to the smallest first error distance among the M groups of first error distances as the target parameters of the camera.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps in the camera parameter calibration method as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps in the camera parameter calibration method as described in any one of claims 1 to 7.

Images