Camera parameter calibration method and device, electronic equipment and readable storage medium

Abstract

The invention provides a camera parameter calibration method and device, electronic equipment and a readable storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining inertial sensor data of a target object at N first moments and image data shot by a camera; acquiring first position data of the target object at each of the N first moments based on the inertial sensor data of the N first moments; performing position conversion on the image data of the N first moments based on a preset camera parameter set to obtain M groups of second position data; summing the distance between the second position data corresponding to N first moments included in each group of second position data and the first position data corresponding to the same first moment to obtain M groups of first error distances; and calibrating a preset camera parameter corresponding to the minimum first error distance in the M groups of first error distances as a target parameter of the camera. The flexibility of parameter calibration is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, device, electronic device, and readable storage medium for calibrating camera parameters. Background Art

[0002] To realize the construction of an intelligent city, monitoring cameras are installed during the city construction process to monitor various objects in the city and convert the monitored positions into the same world coordinate to generate a 3D city model. In the related art, to convert the monitored positions into the same world coordinate, it is necessary to calibrate the parameters of the camera, and the objects captured by the camera are converted to the world coordinate through the parameters of the camera. However, in the related art, the parameter calibration of each camera needs to be calibrated through a specific calibration object or requires the camera to perform specific movements for calibration. For scenarios where a specific calibration object cannot be set or for fixed cameras, there is a problem that the camera parameter calibration cannot be performed through a specific calibration object or camera movement, resulting in poor flexibility in camera parameter calibration.

[0003] It can be seen that there is a problem of poor flexibility in camera parameter calibration in the related art. Summary of the Invention

[0004] Embodiments of the present invention provide a method, device, electronic device, and readable storage medium for calibrating camera parameters to solve the problem of poor flexibility in camera parameter calibration in the related art.

[0005] To solve the above problems, the present invention is implemented as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for calibrating camera parameters, including:

[0007] Obtaining inertial sensor data and image data captured by a camera of a target object at N first moments, where the target object is an object moving within the shooting range of the camera, and N is a positive integer greater than 1;

[0008] Obtaining first position data of the target object at each of the N first moments based on the inertial sensor data of the N first moments;

[0009] Performing position conversion on the image data of the N first moments based on a preset camera parameter set to obtain M sets of second position data, where the preset camera parameter set includes M sets of preset camera parameters, each set of second position data in the M sets of second position data includes second position data corresponding to each of the N first moments, the first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1;

[0010] Sum the distances between the second position data corresponding to the N first moments included in each group of the second position data and the first position data corresponding to the same first moment to obtain M groups of first error distances;

[0011] Calibrate the preset camera parameters corresponding to the smallest first error distance among the M groups of first error distances as the target parameters of the camera.

[0012] In a second aspect, an embodiment of the present invention further provides a camera parameter calibration device, including:

[0013] A first acquisition module, configured to acquire inertial sensor data and image data captured by a camera of a target object at N first moments, where the target object is an object moving within the shooting range of the camera, and N is a positive integer greater than 1;

[0014] A second acquisition module, configured to obtain first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments;

[0015] A position conversion module, configured to perform position conversion on the image data at the N first moments based on a preset set of camera parameters to obtain M groups of second position data, where the preset set of camera parameters includes M groups of preset camera parameters, each group of the M groups of second position data includes second position data corresponding to each of the N first moments, and the first position data and the M groups of second position data are position data in the same coordinate system, and M is a positive integer greater than 1;

[0016] A processing module, configured to sum the distances between the second position data corresponding to the N first moments included in each group of the second position data and the first position data corresponding to the same first moment to obtain M groups of first error distances;

[0017] A calibration module, configured to calibrate the preset camera parameters corresponding to the smallest first error distance among the M groups of first error distances as the target parameters of the camera.

[0018] In a third aspect, an embodiment of the present invention further provides an electronic device, including a processor, a memory, and a computer program stored on the memory and executable on the processor, where when the computer program is executed by the processor, the steps in the camera parameter calibration method described in the first aspect above are implemented.

[0019] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium for storing a program, where when the program is executed by a processor, the steps in the camera parameter calibration method described in the first aspect above are implemented.

[0020] In an embodiment of the present invention, by acquiring inertial sensor data of a target object at N first moments and image data captured by a camera, the target object is an object moving within the range captured by the camera, N is a positive integer greater than 1; based on the inertial sensor data at N first moments, first position data of the target object at each of the N first moments is obtained; based on a preset camera parameter set, position conversion is performed on the image data at N first moments to obtain M sets of second position data, the preset camera parameter set includes M sets of preset camera parameters, each set of second position data in the M sets of second position data includes second position data corresponding to each of the N first moments, the first position data and the M sets of second position data are position data in the same coordinate system, M is a positive integer greater than 1; the distances between the second position data corresponding to the N first moments included in each set of second position data and the first position data corresponding to the same first moment are summed to obtain M sets of first error distances; the preset camera parameter corresponding to the smallest first error distance among the M sets of first error distances is calibrated as the target parameter of the camera, realizing the calibration of the camera parameters. There is no need for a specific calibration object or camera movement for camera parameter calibration, which can be flexibly applied to different scenarios, thereby improving the flexibility of camera parameter calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments of the present invention. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0022] Figure 1 is a flowchart of a method for calibrating camera parameters provided by an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of the first position data provided by an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of the image data provided by an embodiment of the present invention;

[0025] Figure 4 is a schematic diagram of displacement calculation provided by an embodiment of the present invention;

[0026] Figure 5 is one of the schematic diagrams of the relationship between the fourth position data and the fifth position data provided by an embodiment of the present invention;

[0027] Figure 6 is the second of the schematic diagrams of the relationship between the fourth position data and the fifth position data provided by an embodiment of the present invention;

[0028] Figure 7 It is a schematic diagram of target object detection provided by an embodiment of the present invention;

[0029] Figure 8 It is a structural diagram of a camera parameter calibration device provided by an embodiment of the present invention;

[0030] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. Specific embodiments

[0031] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0032] Please refer to Figure 1 , Figure 1 It is a flowchart of a camera parameter calibration method provided by an embodiment of the present invention. As Figure 1 shown, it includes the following steps:

[0033] Step 101, obtain inertial sensor data of the target object at N first moments and image data captured by the camera. The target object is an object moving within the camera shooting range, and N is a positive integer greater than 1.

[0034] The above-mentioned target object is an object moving within the camera shooting range, and the target object is equipped with an inertial sensor to obtain inertial sensor data through the inertial sensor configured for the target object. The target object is usually set as a pedestrian carrying an inertial sensor or a mobile device carrying an inertial sensor.

[0035] The above-mentioned image data is image data including the target object, and the position of the target object is included in the image data. It should be noted that the image data at N first moments is image data captured by the camera under the same parameter conditions. The parameters of the camera include focal length, pose parameters, etc.

[0036] The above-mentioned N first moments are moments in a continuous time period. In this continuous time period, the target object moves within the camera shooting range, and the camera shoots the target object to obtain N image data; N inertial sensor data is obtained through the inertial sensor.

[0037] Among them, the moments of shooting the image data correspond one-to-one with the moments of obtaining the inertial sensor time, so as to reduce the error caused by the displacement of the target object due to inconsistent moments and improve the accuracy of parameter calibration.

[0038] Step 102: Obtain the first position data of the target object at each of the N first moments based on the inertial sensor data of the N first moments.

[0039] The above first position data is the position data obtained based on the inertial sensor data. Specifically, the movement situation within the N first moments is obtained through the inertial sensor, including the movement direction and the movement distance. Then, based on the movement direction and the movement distance, the first position data corresponding to each first moment is obtained. Among them, the movement direction and the movement distance are the movement direction and the movement distance of the target object in the world coordinate system.

[0040] It should be noted that according to different shooting scenarios, the method of obtaining the first position data corresponding to each first moment through the movement direction and the movement distance also has differences. In the outdoor scenario, the first position data is obtained through the satellite positioning data of the target object, as well as the movement direction and the movement distance. In the indoor scenario, since the satellite positioning data of the target object cannot be obtained, the first position data needs to be obtained through preset points. See the subsequent embodiments for details.

[0041] Step 103: Perform position conversion on the image data of the N first moments based on a preset camera parameter set to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets of second position data includes the second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1.

[0042] The M sets of preset camera parameters included in the above preset camera parameter set are the existing ranges of camera parameters. The M sets of preset camera parameters include the target parameters of the current camera. By performing position conversion on the image data of the N first moments through the preset camera parameter set, the positions of the target object in the images at the N first moments are converted into second position data in the world coordinate system, so that the first position data and the second position data are in the same coordinate system, that is, both are in the world coordinate system. Then, a set of parameters is selected from the M sets of preset camera parameters as the target parameters of the current camera for calibration based on the first position data and the second position data.

[0043] Among them, the preset camera parameter set includes M sets of preset camera parameters. Position conversion is performed on the image data of the N first moments based on each set of preset parameters to obtain M sets of second position data. Each set of second position data includes N second position data.

[0044] Step 104: Sum the distances between the second position data corresponding to the N first moments included in each group of second position data and the first position data corresponding to the same first moment, to obtain M groups of first error distances.

[0045] Step 105: Calibrate the preset camera parameters corresponding to the smallest first error distance among the M groups of first error distances as the target parameters of the camera.

[0046] After the distances between the second position data and the first position data at each of the N first moments for a set of first error distances, that is, the sum of N distances. The first error distance obtained by summing can characterize the error between this set of second position data and the first position data, and thus can characterize the error of the preset camera parameters corresponding to this set of second position data. Among them, the larger the first error distance, the greater the gap between the corresponding preset camera parameters and the actual parameters of the current camera; the smaller the first error distance, the smaller the gap between the corresponding preset camera parameters and the actual parameters of the current camera. After obtaining the M groups of first error distances, determine the smallest first error distance among the M groups of first error distances. The preset camera parameters corresponding to this first error distance have the smallest gap with the actual parameters of the current camera, and are calibrated as the target parameters of the camera.

[0047] Among them, the first position data of the target object at N first moments is as Figure 2 shown, and the image data is as Figure 3 shown, including the position data of the target object in the image at N first moments. By performing position conversion on the Figure 3 shown position data of the target object in the image at N first moments to obtain the second position data, and then comparing it with the Figure 2 shown first position data, the target parameters of the camera can be obtained from the M groups of preset camera parameters.

[0048] Furthermore, the trajectory of the target object during movement is zigzag to improve the accuracy of obtaining the first position data and the second position data.

[0049] In an embodiment of the present invention, by acquiring inertial sensor data of a target object at N first moments and image data captured by a camera, the target object is an object moving within the range captured by the camera, N is a positive integer greater than 1; based on the inertial sensor data at the N first moments, the first position data of the target object at each of the N first moments is obtained; based on a preset camera parameter set, position conversion is performed on the image data at the N first moments to obtain M sets of second position data, the preset camera parameter set includes M sets of preset camera parameters, each set of second position data in the M sets of second position data includes the second position data corresponding to each of the N first moments, the first position data and the M sets of second position data are position data in the same coordinate system, M is a positive integer greater than 1; the distances between the second position data corresponding to the N first moments included in each set of second position data and the first position data corresponding to the same first moment are summed to obtain M sets of first error distances; the preset camera parameter corresponding to the smallest first error distance among the M sets of first error distances is calibrated as the target parameter of the camera, realizing the calibration of the camera parameters. There is no need for a specific calibration object or camera movement to calibrate the camera parameters, which can be flexibly applied to different scenarios, thereby improving the flexibility of camera parameter calibration. At the same time, for the scenario where the camera has been deployed and installed, using the method of this embodiment of the present invention can achieve rapid calibration, and the calibration of a single camera can be completed within a few minutes, greatly improving the calibration efficiency.

[0050] In one embodiment, the inertial sensor data includes quaternions and acceleration data, and the obtaining the first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments includes:

[0051] Calculating the quaternions at each of the first moments based on a first preset formula to obtain the first heading angle at each of the first moments;

[0052] Determining the number of walking steps at each of the first moments based on the number of change cycles of the acceleration data between each of the first moments and the previous first moment adjacent to each of the first moments, one walking step corresponding to one change cycle of the acceleration data;

[0053] Calculating the displacement of each of the first moments in the world coordinate system based on the first heading angle, the number of walking steps, and a set step length at each of the first moments using trigonometric functions;

[0054] Obtaining the first position data of each of the first moments based on the displacement of each of the first moments in the world coordinate system;

[0055] The first preset formula is represented as follows:

[0056]

[0057] Ψ is the first heading angle, and q0, q1, q2, and q3 are the quaternions.

[0058] The inertial sensor data output by the above inertial sensor is in the form of complex number expansion, which can be specifically recorded as q = q0 + iq1 + jq2 + kq3, including the quaternions q0, q1, q2, and q3. Through the quaternion-to-Euler angle formula, the pitch angle, heading angle, and roll angle can be calculated. It should be noted that in the embodiments of the present invention, to reduce the calculation amount, the inertial sensor is placed horizontally during the movement of the target object. At this time, both the pitch angle and the roll angle are 0 degrees, and only the heading angle needs to be calculated, that is, the quaternions at each first moment are calculated through the first preset formula to obtain the first heading angle at each first moment.

[0059] After calculating the first yaw angle, it is necessary to determine the moving distance between two first moments. For a moving target object, the displacement at each first moment can be directly determined by the product of time and speed, or the displacement at each first moment can be determined according to the movement law of the target object. During the process of the target object moving by walking, the center of gravity of the target object has periodic changes, that is, the acceleration in the Z-axis direction shows periodic changes. By calculating the period of the acceleration change between two first moments, the number of walking steps of the target object can be determined. Finally, the number of walking steps is multiplied by the set step length to obtain the displacement at each first moment.

[0060] Among them, as Figure 4 shown, during the process of calculating the displacement, the displacement in the X-axis direction and the displacement in the Y-axis direction are determined through trigonometric functions.

[0061] The above-mentioned first position data at each first moment is obtained based on the displacement of each first moment in the world coordinate system, that is, the first position data at each first moment is obtained through the displacement of each first moment in the world coordinate system under different shooting scenarios. For details, see the subsequent embodiments.

[0062] In the embodiments of the present invention, the first heading angle at each first moment is obtained by calculating the quaternions at each first moment based on the first preset formula; the number of walking steps at each first moment is determined based on the number of change periods of the acceleration data between each first moment and the previous first moment adjacent to each first moment. One number of walking steps corresponds to one change period of the acceleration data; the displacement of each first moment in the world coordinate system is calculated based on the first heading angle, the number of walking steps, and the set step length at each first moment through trigonometric functions, so as to realize the determination of the displacement of the target object at each first moment, and then the first position data at each first moment is obtained based on the displacement of each first moment in the world coordinate system.

[0063] In one embodiment, obtaining the first position data at each of the first moments based on the displacements of the target object in the world coordinate system at each of the first moments includes:

[0064] Obtaining satellite positioning data of the target object at each of the first moments;

[0065] Generating dead reckoning data at each of the first moments based on the earliest satellite positioning data among the N first moments and the displacements of the target object in the world coordinate system at each of the first moments;

[0066] Performing position transformation on the dead reckoning data at each of the first moments based on preset processing parameters to obtain first intermediate position data at each of the first moments;

[0067] Averaging the distances between the first intermediate position data at each of the first moments and the satellite positioning data to obtain a second error distance at each of the first moments;

[0068] Deleting the satellite positioning data in which the second error distance is greater than a preset error distance threshold in the satellite positioning data at each of the first moments to obtain the first position data.

[0069] The position at the earliest moment in the dead reckoning data at the above-mentioned first moments is the position corresponding to the earliest satellite positioning data among the N first moments. Through the displacements of the target object in the world coordinate system at each of the first moments and the position corresponding to the earliest satellite positioning data among the N first moments, position data including each of the N first moments is generated, that is, the dead reckoning data at each of the first moments.

[0070] It should be noted that there are certain errors in satellite positioning data, and it is necessary to filter the satellite positioning data and delete the satellite positioning data with large errors to improve the accuracy of the position data. Since the dead reckoning data is data generated based on displacements and has a certain rotation and translation relationship with the satellite positioning data, the satellite positioning data can be filtered through the dead reckoning data.

[0071] Specifically, the rotation and translation relationship between dead reckoning data and satellite positioning data is identified by preset processing parameters. The preset processing parameters include a rotation angle and a translation amount. Based on the preset processing parameters, the position of the dead reckoning data at each first time is transformed to obtain the first intermediate position data at each first time. The first intermediate position data is reference data obtained based on the dead reckoning data and can be used to optimize the satellite positioning data. Among them, when the second error distance between the first intermediate position data and the satellite positioning data is greater than the preset error distance threshold, it is considered that the error of the satellite positioning data is large and the satellite positioning data needs to be deleted; while when the second error distance between the first intermediate position data and the satellite positioning data is less than or equal to the preset error distance threshold, it is considered that the satellite positioning data can be used as the first position data to calibrate the camera parameters.

[0072] In an embodiment of the present invention, satellite positioning data of a target object at each first time is obtained; dead reckoning data at each first time is generated based on the earliest satellite positioning data among the N first times and the displacement of each first time in the world coordinate system; the position of the dead reckoning data at each first time is transformed based on the preset processing parameters to obtain the first intermediate position data at each first time; the distances between the first intermediate position data at each first time and the satellite positioning data are averaged to obtain the second error distance at each first time; the satellite positioning data with a second error distance greater than the preset error distance threshold among the satellite positioning data at each first time is deleted to obtain the first position data, thereby realizing the acquisition of the first position data through the satellite positioning data and the displacement of each first time in the world coordinate system, and deleting the satellite positioning data with large errors, thereby improving the accuracy of the first position data.

[0073] In one embodiment, the preset error distance threshold and the preset processing parameters are obtained in the following manner:

[0074] Obtain a preset processing parameter set, where the preset processing parameter set includes multiple groups of processing parameters, and each group of processing parameters includes a rotation angle and a translation vector;

[0075] Based on the rotation angle and translation vector included in each group of processing parameters, the position of the dead reckoning data at each first time is transformed to obtain the second intermediate position data corresponding to each group of processing parameters;

[0076] The distances between the satellite positioning data at each first time and the second intermediate position data corresponding to each group of processing parameters are averaged to obtain the third error distance corresponding to each group of processing parameters;

[0077] Set the minimum third error distance among the third error distances corresponding to each group of processing parameters as the preset error distance threshold, and set the processing parameters corresponding to the minimum third error distance as the preset processing parameters.

[0078] It should be noted that for different sets of satellite positioning data, there are also differences in the rotation and translation relationships between the dead reckoning data generated based on them and the satellite positioning data. For each set of satellite positioning data, new preset processing parameters need to be obtained.

[0079] Among the multiple groups of processing parameters included in the above preset processing parameter set, there are preset processing parameters. In the embodiments of the present invention, the satellite positioning data is subjected to position conversion through multiple groups of processing parameters to obtain second intermediate position data, and the third error distance corresponding to each group of processing parameters is calculated to determine the preset processing parameters and the preset error distance threshold.

[0080] Specifically, assume that the rotation angle of a set of processing parameters is theta, and the translation vector is <move x , move y >. According to the rotation angle and the translation vector, the coordinates of the dead reckoning point sequence are converted to the world coordinate system, and the specific formula is as follows:

[0081] x jp’ = (x jp - move x ) * cos(theta) - (y jp - move y ) * sin(theta)

[0082] y jp’ = (x jp - move x ) * sin(theta) + (y jp - move y ) * cos(theta)

[0083] Among them, x jp , y jp are satellite positioning data, and x jp’ , y jp’ are second intermediate position data. Then, calculate the average value of the distances between the satellite positioning data at each first moment and the second intermediate position data corresponding to each group of processing parameters to obtain the third error distance dist_error corresponding to this group of processing parameters. After calculating the third error distance corresponding to each group of processing parameters, set the minimum third error distance among the third error distances corresponding to each group of processing parameters as the preset error distance threshold, and set the processing parameters corresponding to the minimum third error distance as the preset processing parameters, so as to determine the first position data based on the preset processing parameters and the preset error distance threshold.

[0084] In one embodiment, obtaining the first position data at each first moment based on the displacement of each first moment in the world coordinate system includes:

[0085] Obtaining the third position data of a plurality of preset points, and the second moment corresponding to each preset point among the plurality of preset points that the target object passes through, where the second moment corresponding to each preset point is the same as one of the N first moments, and the plurality of preset points are points in the indoor space;

[0086] Generating the first position data based on the third position data of the plurality of preset points and the displacement of each first moment in the world coordinate system, where the first position data at the second moment is the third position data of the preset point corresponding to the second moment.

[0087] It should be noted that the scenarios where the camera is installed are different, and there are situations where the target object cannot be located by satellite, such as in the indoor space. In this case, the first position data cannot be obtained with satellite positioning data. Therefore, in the embodiments of the present invention, the first position data is obtained through preset points.

[0088] Among them, the preset points are points in the indoor space, and the position data of the preset points in the world coordinate system has been pre-determined. The target object passes through the preset points during the movement. The above-mentioned generating the first position data based on the third position data of the plurality of preset points and the displacement of each first moment in the world coordinate system means generating multiple continuous trajectories starting from each preset point. Each trajectory includes the first position data, and then all the trajectories are connected in chronological order to form the first position data at N first moments.

[0089] Further, in the embodiments of the present invention, the preset points are used to calibrate the position determined based on the displacement. The more preset points there are, the higher the accuracy of the generated first position data. To improve the accuracy of the final first position data, the number of preset points should be greater than or equal to three.

[0090] In an embodiment of the present invention, by obtaining third position data of multiple preset points, and a second moment corresponding to each preset point among the multiple preset points that the target object passes through, where the second moment corresponding to each preset point is the same as one of the N first moments, and the multiple preset points are points in the indoor space; generating the first position data based on the third position data of the multiple preset points and the displacement of each first moment in the world coordinate system, where the first position data at the second moment is the third position data of the preset point corresponding to the second moment. In this way, determining the first position data through the preset points does not require obtaining satellite positioning data of the target object, can be applied to more camera installation scenarios, and further improves the flexibility of camera parameter calibration.

[0091] In one embodiment, the image data includes fourth position data of the target object in the image, and each set of preset camera parameters in the M sets of preset camera parameters includes a pitch angle and a second heading angle. The position conversion based on the preset camera parameter set and the image data of the N first moments to obtain M sets of second position data includes:

[0092] Obtaining fifth position data of the camera in the world coordinate system and the height of the camera;

[0093] Calculating, based on trigonometric functions, for each set of preset camera parameters including the pitch angle, the height, and the image data of each first moment, to obtain M sets of third intermediate position data of the target object relative to the camera at each first moment;

[0094] Performing position conversion on the M sets of third intermediate position data at each first moment based on the fifth position data of the camera and the second heading angle included in each set of preset camera parameters to obtain the M sets of second position data.

[0095] The above pitch angle and second heading angle are camera parameters to be calibrated. By performing position conversion on the fourth position data of the target object in the image through the M sets of preset camera parameters to obtain third intermediate position data in the world coordinate system, and then comparing it with the first position data, one set of preset camera parameters in the M sets of preset camera parameters is determined as the target parameters of the camera.

[0096] For example, the relationship between the fourth position data of the target object in the image and the third intermediate position data in the world coordinate system is as Figure 5 and Figure 6As shown, the coordinate system in the image is UO1V. The coordinate of the fourth position data on the V-axis is P1, and the coordinate of the fourth position data on the U-axis is Q1. The position where the camera is located is O2. The world coordinate system is XO3Y. The coordinate of the first position data of the target object on the Y-axis in the world coordinate system is P, and the coordinate of the first position data of the target object on the X-axis in the world coordinate system is Q. Specifically, the following formula is used to convert the fourth position data into the Y-axis coordinate of the third intermediate position data in the world coordinate system based on trigonometric functions:

[0097] Wy = H × tan(β + γ)

[0098] Among them, Wy is the Y-axis coordinate of the third intermediate data, γ is the depression angle, H is the height of the camera, and β is calculated through the following formula

[0099] β = atan((cy - Q1y) ÷ fy

[0100] cy is the V-axis coordinate of O1 in the image, Q1y is the V-axis coordinate of Q1, and fy is the coordinate of the focal length on the V-axis.

[0101] The following formula is used to convert the fourth position data into the X-axis coordinate of the third intermediate position data in the world coordinate system based on trigonometric functions:

[0102]

[0103] Among them, Wx is the X-axis coordinate of the third intermediate position data, d is the distance between the target object and the camera (i.e., the distance between point P and point O3), and α is obtained through the following formula:

[0104]

[0105] Q1x is the U-axis coordinate of Q1, cx is the U-axis coordinate of O1 in the image, and fx is the coordinate of the focal length on the U-axis.

[0106] After obtaining the third intermediate position data (Wx, Wy), the second position data (Wx', Wy') is calculated based on the second heading angle and the fifth position data (Mx, My) of the camera. The formula is as follows:

[0107] Wx' = Wx × cosθ + Wy × sinθ + Mx

[0108] Wy′ = Wy × cosθ - Wx × sinθ + My

[0109] Among them, θ is the second heading angle.

[0110] In one embodiment, the fourth position data is the position of the bottom center of the detection frame in the image after a detection frame including the target object is recognized in the image based on a convolutional neural network.

[0111] The above-mentioned target object is recognized through a neural network model. For example, it is trained using a general pedestrian dataset, pedestrian detection is performed on an image captured including the target object, and an identity re-identification technology is used to extract the pixel features of the pedestrian in the detection frame and calculate the similarity with the pixel features of the target object, resulting in Figure 7 a detection frame with the highest similarity and exceeding the threshold. It should be noted that the position of the target object during positioning is the position where it contacts below, so the pixel coordinates of the midpoint of the bottom edge of the detection frame are calculated, and this coordinate is the fourth position data.

[0112] Please refer to Figure 8 , Figure 8 which is a structural diagram of a camera parameter calibration device provided by an embodiment of the present invention. As Figure 8 shown, the camera parameter calibration device 800 includes:

[0113] A first acquisition module 801, configured to acquire inertial sensor data of the target object at N first moments and image data captured by a camera, where the target object is an object moving within the camera's shooting range, and N is a positive integer greater than 1;

[0114] A second acquisition module 802, configured to obtain first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments;

[0115] A position conversion module 803, configured to perform position conversion on the image data at the N first moments based on a preset camera parameter set to obtain M sets of second position data. The preset camera parameter set includes M sets of preset camera parameters. Each set of second position data in the M sets of second position data includes second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1;

[0116] A processing module 804, configured to sum the distances between the second position data corresponding to the N first moments included in each set of second position data and the first position data corresponding to the same first moment to obtain M sets of first error distances;

[0117] A calibration module 805, configured to calibrate the preset camera parameter corresponding to the smallest first error distance among the M sets of first error distances as the target parameter of the camera.

[0118] In one embodiment, the inertial sensor data includes quaternions and acceleration data, and the second acquisition module 802 includes:

[0119] A first calculation unit, configured to calculate the quaternion at each first moment based on a first preset formula to obtain the first heading angle at each first moment;

[0120] A processing unit, configured to determine the number of walking steps at each first moment based on the number of change cycles of the acceleration data between each first moment and the previous first moment adjacent to each first moment, where one walking step corresponds to one change cycle of the acceleration data;

[0121] A second calculation unit, configured to calculate the displacement of each first moment in the world coordinate system based on the first heading angle, the number of walking steps, and a set step length at each first moment using trigonometric functions;

[0122] A first acquisition unit, configured to obtain the first position data at each first moment based on the displacement of each first moment in the world coordinate system;

[0123] The first preset formula is expressed as follows:

[0124]

[0125] Ψ is the first heading angle, and q0, q1, q2, and q3 are the quaternions.

[0126] In one embodiment, the first acquisition unit includes:

[0127] A first acquisition subunit, configured to acquire the satellite positioning data of the target object at each first moment;

[0128] A first generation subunit, configured to generate dead reckoning data at each first moment based on the earliest satellite positioning data among the N first moments and the displacement of each first moment in the world coordinate system;

[0129] A position conversion subunit, configured to perform position conversion on the dead reckoning data at each first moment based on preset processing parameters to obtain first intermediate position data at each first moment;

[0130] A first processing subunit, configured to average the distance between the first intermediate position data and the satellite positioning data at each first moment to obtain a second error distance at each first moment;

[0131] A second processing subunit, configured to delete the satellite positioning data in which the second error distance at each of the first moments is greater than a preset error distance threshold, so as to obtain the first position data.

[0132] In one embodiment, the preset error distance threshold and the preset processing parameters are obtained in the following manner:

[0133] Obtain a set of preset processing parameters, where the set of preset processing parameters includes multiple groups of processing parameters, and each group of processing parameters includes a rotation angle and a translation vector;

[0134] Based on the rotation angle and the translation vector included in each group of processing parameters, perform position conversion on the dead reckoning data at each of the first moments to obtain second intermediate position data corresponding to each group of processing parameters;

[0135] Average the distances between the satellite positioning data at each of the first moments and the second intermediate position data corresponding to each group of processing parameters to obtain a third error distance corresponding to each group of processing parameters;

[0136] Set the smallest third error distance among the third error distances corresponding to each group of processing parameters as the preset error distance threshold, and set the processing parameters corresponding to the smallest third error distance as the preset processing parameters.

[0137] In one embodiment, the first obtaining unit includes:

[0138] A second obtaining subunit, configured to obtain third position data of multiple preset points, and a second moment corresponding to each preset point among the multiple preset points that the target object passes through. The second moment corresponding to each preset point is the same as one of the N first moments, and the multiple preset points are points in the indoor space;

[0139] A second generating subunit, configured to generate the first position data based on the third position data of the multiple preset points and the displacement of each of the first moments in the world coordinate system, where the first position data at the second moment is the third position data of the preset point corresponding to the second moment.

[0140] In one embodiment, the image data includes fourth position data of the target object in the image, and each group of preset camera parameters in the M groups of preset camera parameters includes a pitch angle and a second heading angle. The position conversion module 803 includes:

[0141] A second obtaining unit, configured to obtain fifth position data of the camera in the world coordinate system and the height of the camera;

[0142] A third calculation unit, configured to calculate, based on trigonometric functions, each set of preset camera parameters including the pitch angle, the height, and the image data at each first moment, to obtain M sets of third intermediate position data of the target object relative to the camera at each first moment;

[0143] A position conversion unit, configured to perform position conversion on the M sets of third intermediate position data at each first moment based on the fifth position data of the camera and the second heading angle included in each set of preset camera parameters, to obtain the M sets of second position data.

[0144] In one embodiment, the fourth position data is the position of the bottom center of the detection frame in the image after a detection frame including the target object is recognized in the image based on a convolutional neural network.

[0145] The camera parameter calibration device provided by the embodiments of the present invention can implement each process of the above camera parameter calibration method. The technical features correspond one by one and can achieve the same technical effects. To avoid repetition, it will not be elaborated here.

[0146] It should be noted that the camera parameter calibration device in the embodiments of the present invention can be a device, or a component, an integrated circuit, or a chip in an electronic device.

[0147] The embodiments of the present invention further provide an electronic device. Refer to Figure 9 , Figure 9 is a schematic structural diagram of an electronic device provided by the embodiments of the present invention. The electronic device includes a memory 901, a processor 902, and a program or instruction running on the memory 901. When the program or instruction is executed by the processor 902, it can implement Figure 1 any step in the corresponding method embodiment and achieve the same beneficial effects, which will not be elaborated here.

[0148] Among them, the processor 902 can be a CPU, an ASIC, an FPGA, or a GPU.

[0149] Those of ordinary skill in the art can understand that all or part of the steps for implementing the method in the above embodiments can be completed by hardware related to program instructions, and the program can be stored in a readable medium.

[0150] The embodiments of the present invention further provide a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it can implement the above Figure 1Any step in the corresponding method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be elaborated here. The storage medium includes, for example, Read-Only Memory (ROM), Random Access Memory (RAM), magnetic disk, or optical disc, etc.

[0151] The terms "first", "second", etc. in the embodiments of the present invention are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. In addition, the terms "include" and "have" and any of their variations are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily limit to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products, or devices. In addition, in this application, the use of "and / or" means at least one of the connected objects. For example, A and / or B and / or C means including A alone, B alone, C alone, as well as the situations where A and B exist, B and C exist, A and C exist, and A, B, and C all exist, a total of 7 situations.

[0152] It should be noted that in this article, the term "include", "comprise" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not clearly listed, or also includes elements inherent to this process, method, article or device. Without more limitations, the element defined by the statement "including one..." does not exclude the existence of other identical elements in the process, method, article or device including this element.

[0153] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc) and includes several instructions to enable a terminal (which can be a mobile phone, computer, server, air conditioner, or a second terminal device, etc.) to execute the methods of various embodiments of this application.

[0154] The embodiments of the present application have been described above in conjunction with the accompanying drawings. However, the present application is not limited to the above specific embodiments. The above specific embodiments are merely illustrative rather than restrictive. Under the inspiration of the present application, those of ordinary skill in the art can also make many forms without departing from the purpose of the present application and the scope protected by the claims, and all of them fall within the protection scope of the present application.

Claims

1. A method for calibrating camera parameters, characterized in that, Including: Obtaining inertial sensor data of a target object at N first moments and image data captured by a camera, where the target object is an object moving within the range captured by the camera, and N is a positive integer greater than 1; Based on the inertial sensor data at the N first moments, obtaining first position data of the target object at each of the N first moments; Performing position conversion on the image data at the N first moments based on a preset camera parameter set, obtaining M sets of second position data, where the preset camera parameter set includes M sets of preset camera parameters, and each set of second position data in the M sets of second position data includes second position data corresponding to each of the N first moments. The first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1; Summing the distances between the second position data corresponding to the N first moments included in each set of second position data and the first position data corresponding to the same first moment, obtaining M sets of first error distances; Calibrating the preset camera parameter corresponding to the smallest first error distance among the M sets of first error distances as the target parameter of the camera.

2. The method according to claim 1, characterized in that, The inertial sensor data includes quaternion and acceleration data. The obtaining of the first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments includes: Calculating the quaternion at each first moment based on a first preset formula to obtain the first heading angle at each first moment; Determining the number of walking steps at each first moment based on the number of change cycles of the acceleration data between each first moment and the previous first moment adjacent to each first moment. One walking step corresponds to one change cycle of the acceleration data; Calculating the displacement of each first moment in the world coordinate system based on the first heading angle, the number of walking steps, and a set step length at each first moment using trigonometric functions; Obtaining the first position data at each first moment based on the displacement of each first moment in the world coordinate system; The first preset formula is represented as follows: Ψ is the first heading angle, and q0, q1, q2, and q3 are the quaternions.

3. The method according to claim 2, wherein The obtaining of the first position data at each first moment based on the displacement of each first moment in the world coordinate system includes: Obtaining satellite positioning data of the target object at each first moment; Generating dead reckoning data at each first moment based on the earliest satellite positioning data among the N first moments and the displacement of each first moment in the world coordinate system; Performing position conversion on the dead reckoning data at each first moment based on preset processing parameters to obtain first intermediate position data at each first moment; Averaging the distances between the first intermediate position data at each first moment and the satellite positioning data to obtain a second error distance at each first moment; Delete the satellite positioning data with the second error distance greater than the preset error distance threshold in the satellite positioning data at each of the first moments to obtain the first position data.

4. The method according to claim 3, wherein The preset error distance threshold and the preset processing parameters are obtained in the following manner: Obtain a preset processing parameter set, where the preset processing parameter set includes multiple sets of processing parameters, and each set of processing parameters includes a rotation angle and a translation vector; Based on the rotation angle and translation vector included in each set of processing parameters, perform position transformation on the dead reckoning data at each of the first moments to obtain second intermediate position data corresponding to each set of processing parameters; Average the distances between the satellite positioning data at each of the first moments and the second intermediate position data corresponding to each set of processing parameters to obtain a third error distance corresponding to each set of processing parameters; Set the smallest third error distance among the third error distances corresponding to each set of processing parameters as the preset error distance threshold, and set the processing parameters corresponding to the smallest third error distance as the preset processing parameters.

5. The method according to claim 2, wherein The obtaining of the first position data at each of the first moments based on the displacement of the target object in the world coordinate system includes: Obtain the third position data of multiple preset points, and the second moment corresponding to each preset point that the target object passes through. The second moment corresponding to each preset point is the same as one of the N first moments. The multiple preset points are points in the indoor space; Generate the first position data based on the third position data of the multiple preset points and the displacement of the target object in the world coordinate system at each of the first moments, where the first position data at the second moment is the third position data of the preset point corresponding to the second moment.

6. The method according to claim 1, characterized in that, The image data includes the fourth position data of the target object in the image. Each set of preset camera parameters in the M sets of preset camera parameters includes a pitch angle and a second heading angle. The performing of position transformation on the image data at the N first moments based on the preset camera parameter set to obtain M sets of second position data includes: Obtain the fifth position data of the camera in the world coordinate system and the height of the camera; Based on trigonometric functions, calculate the image data at each of the first moments using each set of preset camera parameters including the pitch angle, the height, and the target object to obtain M sets of third intermediate position data of the target object relative to the camera at each of the first moments; Based on the fifth position data of the camera and the second heading angle included in each set of preset camera parameters, perform position transformation on the M sets of third intermediate position data at each of the first moments to obtain the M sets of second position data.

7. The method according to claim 6, wherein The fourth position data is the position of the bottom center of the detection frame in the image after a detection frame including the target object is recognized in the image based on a convolutional neural network.

8. A camera parameter calibration device, characterized in that, Includes: A first acquisition module, configured to acquire inertial sensor data of a target object at N first moments and image data captured by a camera, where the target object is an object moving within the shooting range of the camera, and N is a positive integer greater than 1; A second acquisition module, configured to obtain first position data of the target object at each of the N first moments based on the inertial sensor data at the N first moments; A position conversion module, configured to perform position conversion on the image data at the N first moments based on a preset camera parameter set to obtain M sets of second position data, where the preset camera parameter set includes M sets of preset camera parameters, each set of second position data in the M sets of second position data includes second position data corresponding to each of the N first moments, the first position data and the M sets of second position data are position data in the same coordinate system, and M is a positive integer greater than 1; A processing module, configured to sum the distances between the second position data corresponding to the N first moments included in each set of second position data and the first position data corresponding to the same first moment to obtain M sets of first error distances; A calibration module, configured to calibrate the preset camera parameter corresponding to the smallest first error distance among the M sets of first error distances as the target parameter of the camera.

9. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored on the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the camera parameter calibration method according to any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps in the camera parameter calibration method according to any one of claims 1 to 7.

Images