Calibration method and apparatus for sensor

Abstract

A calibration method and apparatus for a sensor, which relate to the technical field of sensors. The sensor is one of at least one sensor arranged on a vehicle. The method may comprise: identifying at least one traffic participant in an environment where a vehicle is located; acquiring a first data set associated with the at least one traffic participant, wherein the first data set comprises standardized data of the at least one traffic participant; determining a traveling state of the vehicle; if the vehicle does not shake, performing intrinsic parameter calibration on the sensor on the basis of the first data set; and if the vehicle shakes, performing joint calibration on extrinsic parameters of at least one sensor of the vehicle on the basis of the first data set. The method is used for realizing adaptive calibration for universal multi-sensors, so as to improve the fusion precision of sensing information of the sensors.

Description

Sensor calibration method and device

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of the People's Republic of China on August 24, 2023, with application number 202311076082.3 and application name “A method and device for calibrating a sensor”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of sensor technology, and in particular to a sensor calibration method and device. Background Art

[0004] With the rapid development of intelligent automotive technology, the intelligent driving functions of vehicles are becoming increasingly perfect, and more and more sensors are installed on vehicles. The measurement accuracy of on-board sensors is crucial to the realization of intelligent driving functions and driving safety.

[0005] Most sensors require calibration after installation to ensure measurement accuracy. Sensor calibration involves both intrinsic and extrinsic calibration. Internal parameters are a set of parameters that determine the sensor's internal mapping relationships, while extrinsic parameters determine the transformation relationship between the sensor's coordinate system and a specified coordinate system. Internal calibration is used to obtain the sensor's internal mapping relationships, while extrinsic calibration is used to transform two or more sensors into a unified spatial coordinate system. Currently, commonly used sensor calibration methods either rely on 3D radar sensors as a medium, lacking universal online sensor calibration methods, or require large datasets to train deep learning models for calibration, which presents significant training challenges. These methods are unsuitable for real-time calibration of multiple types of onboard sensors in vehicle scenarios.

[0006] Therefore, how to design a calibration method for general vehicle-mounted sensors remains an important issue that needs to be solved urgently.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a sensor calibration method and apparatus for implementing adaptive calibration of general multi-sensors to improve the fusion accuracy of sensor perception information.

[0009] On the first aspect, an embodiment of the present application provides a sensor calibration method, which can be implemented by a calibration device. The calibration device can be an independent device, a chip or component in the device, or software. It can be deployed on the vehicle side, or in the cloud, or on a roadside device, or on a remote server, or on a local server, etc. The present application does not limit the product form and deployment method of the calibration device.

[0010] The sensor may be one of at least one sensor arranged on a vehicle, and the method may include: identifying at least one traffic participant in the environment where the vehicle is located; obtaining a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; determining the driving state of the vehicle; if the vehicle does not shake, calibrating the internal parameters of the sensor according to the first data set; if the vehicle shakes, jointly calibrating the external parameters of at least one sensor of the vehicle according to the first data set.

[0011] Through the above method, the calibration device can use the standardized traffic participants of the scene elements as the benchmark instead of the calibration plate to perform internal parameter calibration and / or external parameter calibration on the vehicle-mounted sensors. The calibration method of the vehicle-mounted sensors can be decoupled from media such as three-dimensional radar sensors, and there is no need to train a deep learning model for calibration with a large data set. The calibration time and labor costs can be greatly saved, making the calibration method universal. At the same time, the error of joint calibration of multiple sensors is reduced, and the fusion accuracy of sensor information is improved.

[0012] In combination with the first aspect, in a possible implementation, the sensor includes a camera, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: if the vehicle does not shake within a first time period, the first data set is used as a calibration board, and the sensor is calibrated with internal parameters according to multiple frames of data relative to the same traffic participant collected within the first time period.

[0013] In combination with the first aspect, in a possible implementation, the sensor includes an inertial measurement unit IMU, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: if the vehicle does not shake within a second time period, the first data set is used as a calibration board, and the sensor is calibrated with internal parameters according to the posture perception data collected within the second time period.

[0014] In combination with the first aspect, in a possible implementation method, if the vehicle shakes, the external parameters of at least one sensor of the vehicle are jointly calibrated based on the first data set, including: during the shaking of the vehicle, using the first data set as a calibration board, and based on the perception data of traffic participants collected during the shaking process, the external parameters of at least one sensor of the vehicle are jointly calibrated.

[0015] In combination with the first aspect, in a possible implementation, determining the driving state of the vehicle includes: obtaining at least one driving parameter through the at least one sensor, wherein the at least one sensor includes at least one of the following types of sensors: a gyroscope, an accelerometer, or a magnetometer; and determining the driving state of the vehicle based on the at least one driving parameter.

[0016] In combination with the first aspect, in a possible implementation manner, obtaining the first data set associated with the at least one traffic participant includes: receiving the first data set from a cloud server.

[0017] In combination with the first aspect, in a possible implementation manner, the first data set is obtained according to a target tracking model, and the target tracking model includes a deep network model.

[0018] In combination with the first aspect, in a possible implementation, the method further includes: weighting the confidence of the at least one sensor based on prior information to obtain a weight factor associated with the at least one sensor; obtaining a posture estimation result of the vehicle based on the weight factor associated with the at least one sensor and the perception data of the at least one sensor; and performing error compensation on the on-board devices of the vehicle based on the posture estimation result and the three-dimensional rigid body model.

[0019] In combination with the first aspect, in a possible implementation, the vehicle-mounted device includes a light-sensing device of the vehicle, and the light-sensing device includes at least one of the following: an augmented reality head-up display AR-HUD device, an optical projection device, and an intelligent ranging device.

[0020] In a second aspect, an embodiment of the present application provides a sensor calibration device, wherein the sensor is one of at least one sensor arranged on a vehicle, and the device includes: an identification unit for identifying at least one traffic participant in the environment where the vehicle is located; an acquisition unit for acquiring a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; a determination unit for determining the driving state of the vehicle; an internal parameter calibration unit for calibrating the internal parameters of the sensor according to the first data set if the vehicle does not shake; and an external parameter calibration unit for jointly calibrating the external parameters of at least one sensor of the vehicle according to the first data set if the vehicle shakes.

[0021] In combination with the second aspect, in a possible implementation, the sensor includes a camera, and the internal parameter calibration unit is specifically used to: if the vehicle does not shake within a first time period, use the first data set as a calibration plate, and perform internal parameter calibration on the sensor based on multiple frames of data relative to the same traffic participant collected within the first time period.

[0022] In combination with the second aspect, in one possible implementation, the sensor includes an inertial measurement unit IMU, and the intrinsic parameter calibration unit is specifically used to: if the vehicle does not shake within the second time period, use the first data set as a calibration board, and perform intrinsic parameter calibration on the sensor based on the posture perception data collected within the second time period.

[0023] In combination with the second aspect, in a possible implementation method, the external parameter calibration unit is specifically used to: during the shaking of the vehicle, use the first data set as a calibration board, and jointly calibrate the external parameters of at least one sensor of the vehicle based on the perception data of traffic participants collected during the shaking process.

[0024] In combination with the second aspect, in one possible implementation, the determination unit is specifically used to: obtain at least one driving parameter through the at least one sensor, wherein the at least one sensor includes at least one of the following types of sensors: a gyroscope, an accelerometer, or a magnetometer; and determine the driving state of the vehicle based on the at least one driving parameter.

[0025] In combination with the second aspect, in a possible implementation manner, the acquisition unit is specifically configured to: receive the first data set from a cloud server.

[0026] In combination with the second aspect, in a possible implementation manner, the first data set is obtained according to a target tracking model, and the target tracking model includes a deep network model.

[0027] In combination with the second aspect, in a possible implementation, the device also includes an error compensation unit, which is used to weight the confidence of the at least one sensor based on prior information and obtain a weight factor associated with the at least one sensor; obtain the vehicle's posture estimation result based on the weight factor associated with the at least one sensor and the perception data of the at least one sensor; and perform error compensation on the vehicle's on-board devices based on the posture estimation result and the three-dimensional rigid body model.

[0028] In combination with the second aspect, in a possible implementation, the vehicle-mounted device includes a light-sensing device of the vehicle, and the light-sensing device includes at least one of the following: an augmented reality head-up display AR-HUD device, an optical projection device, and an intelligent ranging device.

[0029] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor coupled to a memory: the processor is configured to execute a computer program or instruction stored in the memory so that the electronic device executes the method described in the first aspect and any possible implementation of the first aspect.

[0030] In a fourth aspect, an embodiment of the present application provides a vehicle, comprising a unit for implementing the method described in the first aspect and any possible implementation of the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides a readable storage medium, including a program or instruction. When the program or instruction is executed, the method described in the first aspect and any possible implementation of the first aspect is executed.

[0032] In a sixth aspect, an embodiment of the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first aspect and any possible implementation of the first aspect.

[0033] In a seventh aspect, an embodiment of the present application provides a terminal device, comprising a unit for implementing the method described in the first aspect and any possible design of the first aspect. By way of example, the terminal device includes, but is not limited to: intelligent transportation equipment (such as cars, ships, drones, trains, trucks, etc.), intelligent manufacturing equipment (such as robots, industrial equipment, intelligent logistics, smart factories, etc.), and intelligent terminals (mobile phones, computers, tablets, PDAs, desktops, headphones, speakers, wearable devices, in-vehicle devices, etc.).

[0034] Based on the implementations provided in the above aspects, the embodiments of the present application can be further combined to provide more implementations.

[0035] The technical effects that can be achieved by any possible implementation method in any of the second to seventh aspects mentioned above can be referred to the description of the technical effects that can be achieved by any possible implementation method in any of the first aspect mentioned above, and the repetitions will not be discussed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram showing an application scenario to which an embodiment of the present application is applicable;

[0037] FIG2 is a schematic diagram showing the principle of the calibration method according to an embodiment of the present application;

[0038] FIG3 is a schematic diagram showing a flow chart of a calibration method according to an embodiment of the present application;

[0039] FIG4 is a schematic diagram showing a flow chart of a calibration method according to an embodiment of the present application;

[0040] FIG5 shows a schematic structural diagram of a calibration device according to an embodiment of the present application;

[0041] FIG6 shows a schematic structural diagram of a communication device according to an embodiment of the present application. DETAILED DESCRIPTION

[0042] The embodiments of the present application provide a sensor calibration method and device for realizing adaptive calibration of a general multi-sensor to improve the fusion accuracy of the sensor's perception information. The method and the device are based on the same technical concept. Since the principles of the method and the device for solving problems are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated. Moreover, in the various embodiments of the present application, unless otherwise specified and there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationships.

[0043] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0044] Furthermore, unless otherwise specified, ordinal numbers such as "first" and "second" in the embodiments of this application are used to distinguish between multiple objects and are not used to define the priority or importance of multiple objects. For example, "first sensor" and "second sensor" are only used to distinguish different sensors, and do not indicate a difference in priority or importance between the two sensors.

[0045] The sensor calibration scheme proposed in the embodiment of the present application can be applied to various types of sensors. For example, it can be, but is not limited to, a gyroscope, an accelerometer, a magnetometer, or an on-board optical sensor, such as an augmented reality head-up display (AR-HUD) device, an optical projection device (such as a smart car light), and an intelligent ranging device (such as an adaptive ranging device). More specifically, the sensors to which the calibration scheme is applicable can be applied in vehicle scenarios, as well as in drone scenarios, aircraft scenarios, and the like. In these scenarios, due to road conditions (such as road surface flatness) or environmental factors (such as weather influences such as wind and airflow), the vehicle body where the sensor is located may vibrate violently, thereby causing the problem of perception error. It is necessary to calibrate the sensors used or compensate for the perception information in a timely manner to achieve more accurate detection.

[0046] For ease of understanding, the following is an introduction to the specific details of the method by taking the calibration method of sensors applied to the vehicle field as an example. It should be understood that the vehicle here is an example of intelligent transportation equipment, and the vehicle can specifically be a car, a ship, a drone, a train, a truck, etc. In other optional embodiments, the vehicle can be replaced with intelligent manufacturing equipment, including but not limited to robots, industrial equipment, intelligent logistics, intelligent factory equipment, etc. In other optional embodiments, the vehicle can be replaced with an intelligent terminal, including but not limited to mobile phones, computers, tablets, PDAs, desktops, headphones, speakers, wearable devices, vehicle-mounted equipment, etc., which will not be repeated here.

[0047] FIG1 is a schematic diagram of an application scenario applicable to an embodiment of the present application. This application scenario may include a vehicle 100. In one possible implementation, this application scenario may also include a cloud server 200, and the vehicle 100 and cloud server 200 may communicate via a network. In one embodiment, cloud server 200 may also be implemented as a virtual machine.

[0048] Some or all functions of the vehicle 100 are controlled by a computing platform 150 (or computer system). The computing platform 150 may include at least one processor 151, which may execute instructions 153 stored in a non-transitory computer-readable medium such as a memory 152. In some embodiments, the computing platform 150 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner. The processor 151 may be any conventional processor, such as a central processing unit (CPU). Alternatively, the processor 151 may also include a graphics processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SoC), an application specific integrated circuit (ASIC), or a combination thereof.

[0049] Alternatively, the vehicle 100 may be a car, truck, motorcycle, bus, boat, airplane, helicopter, lawn mower, recreational vehicle, amusement park vehicle, construction equipment, tram, golf cart, train, etc., which is not particularly limited in this embodiment of the present application. In one possible implementation, the vehicle 100 may be an electric vehicle (EV), such as a two-wheel drive electric vehicle or a four-wheel drive electric vehicle, which is not particularly limited in this embodiment of the present application.

[0050] The structure of the vehicle in FIG1 should not be understood as limiting the embodiments of the present application.

[0051] The calibration method of the embodiment of the present application can be implemented by a calibration device, which can be an independent device, or a chip or component in the vehicle 100 shown in Figure 1, or a software module, which can be deployed on the relevant on-board equipment of the vehicle 100. The embodiment of the present application does not limit the product form and deployment method of the calibration device. In specific implementation, the calibration of the internal parameters and / or external parameters of different sensors can be achieved by the same calibration device, or the calibration of the internal parameters and / or external parameters of related sensors can be achieved by different calibration devices. The embodiment of the present application does not limit this. In the following, for the convenience of understanding and description, the calibration scheme of the embodiment of the present application is introduced by taking the method of implementing the calibration of the internal parameters and / or external parameters of different sensors by the same calibration device as an example.

[0052] As shown in FIG2 , the calibration device can interact with other modules in the vehicle to implement the calibration scheme. For example, the vehicle can have a sensing system. On the one hand, the at least one sensor can include an inertial measurement unit (IMU), such as a gyroscope, an accelerometer, or a magnetometer, which can be used to collect different driving parameters of the vehicle, such as the angular velocity, angular acceleration, and heading angle of the vehicle, so as to evaluate the driving state of the vehicle, for example, whether the vehicle is currently in a parked state, a stable running state, or a shaking state (such as shaking caused by bumpy road conditions, shaking caused by speed bumps, etc., which is not limited in the embodiments of the present application).

[0053] On the other hand, the at least one sensor may include but is not limited to a camera, light detection and ranging (LIDAR), millimeter-wave radar (RADAR) or other sensor devices, which can monitor and collect data about the vehicle's surroundings at all times and identify at least one traffic participant in the vehicle's current environment. The at least one traffic participant may be any traffic participant in the vehicle's environment other than the vehicle itself, including but not limited to the vehicle's current lane and the lane line of the current lane; the vehicle in front and behind the vehicle in the current lane; the adjacent lane of the current lane and the lane line of the adjacent lane; several other vehicles in the adjacent lane; pedestrians in the current lane, adjacent lane, or road intersection; various traffic facilities: such as crosswalks, roadside units (RSUs), traffic lights (such as motor vehicle lights, non-motor vehicle lights, crosswalk lights, direction indicators, lane lights, flashing lights, road and railway intersection lights, etc.), fences, lighting facilities, sight guidance signs, highway reflectors, highway information boards, etc. In an optional embodiment, the at least one road user may be a road user having standardized dimensions.

[0054] The calibration device may make a calibration decision based on the driving state of the vehicle and the first data set associated with the first traffic participant to determine how to perform internal parameter calibration and / or external parameter calibration on at least one sensor of the vehicle.

[0055] For example, the first data set may include standardized data of at least one traffic participant. If the vehicle is detected to be shaking, such as when the vehicle is stationary or in a stable operating state, the calibration device may use the first data set as a calibration template and perform internal parameter calibration on the vehicle's sensors based on the first data set, such as internal parameter calibration of a gyroscope, an accelerometer, or a magnetometer. If the vehicle is detected to be shaking, the calibration device may use the first data set as a calibration template and perform a joint calibration on the external parameters of at least one of the vehicle's sensors based on the first data set, such as joint calibration of the external parameters of a gyroscope, an accelerometer, and a magnetometer.

[0056] It should be understood that in specific implementation, the standardized data associated with traffic participants can be stored on a media medium accessible to the calibration device, such as a vehicle storage medium, a cloud server, or a third-party storage device, etc., and the embodiments of this application do not limit this.

[0057] After completing the calibration, in an optional embodiment, the calibration device can also weight the confidence of different sensors according to the relative position of at least one identified traffic participant and the output value of the IMU to obtain a weight factor, solve the final pose estimation result by complementary filtering, and construct a transformation matrix between the IMU and other on-board devices based on external parameters to convert the IMU's estimation result to the on-board device through a three-dimensional rigid body model and perform error compensation. Exemplarily, the on-board device can be a photosensitive device, including but not limited to an AR-HUD device, an optical projection device (such as a smart car light), an intelligent ranging device (such as an adaptive ranging device), etc.

[0058] The above method uses standardized traffic participants as scene elements instead of calibration plates to perform intrinsic and / or extrinsic calibration of vehicle-mounted sensors. This decouples the calibration of vehicle-mounted sensors from intermediaries such as three-dimensional radar sensors and eliminates the need for large datasets to train deep learning models for calibration. This significantly reduces calibration time and labor costs, making the calibration method universal. It also reduces errors in joint calibration of multiple sensors and improves the accuracy of sensor information fusion. This method can reduce sensor estimation errors during vehicle vibration and improve vehicle driving safety, providing auxiliary support for safe driving.

[0059] For ease of understanding, the calibration method of the embodiment of the present application is introduced below by taking the vehicle-mounted device as an example of a vehicle-mounted light-sensing device.

[0060] As shown in FIG3 , the calibration method may include the following steps:

[0061] S310: The calibration device identifies at least one traffic participant in the environment where the vehicle is located.

[0062] The at least one traffic participant may include any traffic participant in the vehicle's environment other than the vehicle itself, including but not limited to the vehicle's current lane and the lane line of the current lane; the vehicle in front and behind the vehicle in the current lane; the adjacent lane of the current lane and the lane lines of the adjacent lane; several other vehicles on the adjacent lane; pedestrians in the current lane, or adjacent lane, or road intersection; various traffic facilities: such as crosswalks, roadside units (RSU), traffic lights (such as motor vehicle lights, non-motor vehicle lights, crosswalk lights, direction indicators, lane lights, flashing lights, road and railway intersection lights, etc.), fences, lighting facilities, sight-guiding signs, highway reflectors, highway information boards, etc.

[0063] In an optional embodiment, the at least one traffic participant may be a traffic participant with standardized dimensions. The standardization may refer to an international standard, a national standard, a standard for a province, city, county, township, or the like, or a standard for a designated region (e.g., a smart factory). The present embodiment does not limit the implementation of this standardization.

[0064] When S310 is specifically implemented, the calibration device may identify at least one traffic participant in the environment where the vehicle is located through at least one sensor provided on the vehicle and a target detection algorithm.

[0065] S320: The calibration device obtains a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant.

[0066] In one example, when implementing S320, the calibration device may receive the first dataset in real time from the cloud server. Alternatively, in another example, a local storage device in the vehicle may pre-store standardized data for various common traffic participants. When implementing S320, the calibration device may directly obtain the first dataset from the local storage device in the vehicle. This embodiment of the present application does not limit the method for obtaining the first dataset.

[0067] It is worth noting that within a certain time period, the standardized data of the same traffic participant is usually stable and unchanged. The calibration device only needs to execute S320 once within the time period, without repeated execution, which can reduce energy consumption.

[0068] In an optional embodiment, the standardized data in the first data set can be obtained according to a target tracking model, which may include a deep network model, such as a DarkNet network. The target tracking model can construct a standard traffic sign data set (including pictures and annotation information) based on the obstacle information in the vehicle's current environment, and complete the training of the first data set based on transfer learning. In an optional embodiment, at least 50 pieces of each type of data can be obtained, and the training iteration is more than 5,000 times to ensure the accuracy of the obtained standardized data.

[0069] S330: The calibration device determines the driving state of the vehicle.

[0070] In a specific implementation, the calibration device obtains at least one driving parameter through at least one sensor of the vehicle, and determines the driving state of the vehicle according to the at least one driving parameter.

[0071] Exemplarily, the at least one sensor may include at least one of the following types of sensors: a gyroscope, an accelerometer, or a magnetometer. The gyroscope, accelerometer, or magnetometer may be arranged at the center of the roof of the vehicle body, or may be arranged on the same vertical line as the center of gravity of the vehicle body and closer to a horizontal setting. The gyroscope may be used to collect the angular velocity of the three rotational degrees of freedom of the vehicle body (such as yaw angle, roll angle, or pitch angle). In an optional embodiment, the angular acceleration value may also be obtained by integrating the angular velocity value over time. The accelerometer may be used to collect the acceleration of the three rotational degrees of freedom of the vehicle body. The magnetometer may be used to measure the vehicle body in one rotational degree of freedom (such as yaw angle).

[0072] Take the evaluation of the driving state of a vehicle based on acceleration as an example. An acceleration threshold value can be preset, such as 1m / s^2. If the acceleration value obtained by the calibration device is lower than the acceleration threshold value, it can be considered that the vehicle is in a stable running state. If the acceleration value obtained by the calibration device is greater than or equal to the acceleration threshold value, it can be considered that the vehicle is shaking. If the vehicle linear velocity and acceleration values ​​obtained by the calibration device are both lower than the corresponding threshold values ​​(for example, the linear velocity threshold value is 1m / s, and the acceleration threshold value is 1m / s^2), it can be considered that the vehicle is in a parked state.

[0073] It should be understood that this is merely an example of the method for determining the driving state of a vehicle in an embodiment of the present application, and not a limitation. In other embodiments, the calibration device may also determine the driving state of the vehicle by other means, which will not be repeated here.

[0074] S340: If the vehicle does not shake, the calibration device performs internal parameter calibration on the sensor according to the first data set.

[0075] For example, taking the sensor as a camera, during the specific implementation of S340, if the vehicle does not experience any vibration within a first time period, e.g., is in a stable operating state, the calibration device uses the first dataset as a calibration template and performs intrinsic parameter calibration on the sensor based on multiple frames of data collected during the first time period relative to the same traffic participant. During this process, the calibration device can, for example, obtain the camera's intrinsic parameter information, such as the affine transformation matrix from the image coordinate system to the left-right of the pixel, using the Zhang Zhengyou intrinsic parameter calibration method.

[0076] For example, if the sensor is an IMU (e.g., including a gyroscope, accelerometer, and magnetometer), during the specific implementation of S340, the vehicle does not shake during the second time period, for example, it is in a stable operating state. The calibration device uses the first data set as a calibration plate and performs intrinsic parameter calibration on the sensor based on the posture perception data collected during the second time period. During this process, the calibration device can, for example, use the kalibr calibration tool to complete the intrinsic parameter calibration.

[0077] S350: If the vehicle shakes, the calibration device performs a joint calibration on the external parameters of at least one sensor of the vehicle according to the first data set.

[0078] For example, if the sensor is an IMU (e.g., including a gyroscope, accelerometer, and magnetometer), during the specific implementation of S350, during the vehicle's jitter, the calibration device may use the first dataset as a calibration template and, based on the traffic participant's perception data or vehicle status information collected during the jitter, jointly calibrate the extrinsic parameters of at least one sensor of the vehicle. During this process, the calibration device may, for example, utilize the Kalibr calibration tool to complete the extrinsic parameter calibration.

[0079] Therefore, through the above method, the calibration device can use the standardized traffic participants of the scene elements as the benchmark instead of the calibration plate to perform internal parameter calibration and / or external parameter calibration of the vehicle-mounted sensors. The calibration method of the vehicle-mounted sensors can be decoupled from media such as three-dimensional radar sensors, and there is no need to train a deep learning model for calibration with a large data set. The calibration time and labor costs can be greatly saved, making the calibration method universal. At the same time, the error of joint calibration of multiple sensors is reduced and the fusion accuracy of sensor information is improved.

[0080] After completing the calibration of the sensor, in an optional embodiment, the calibration device can also perform error compensation on the on-board device that provides assisted driving functions based on at least one sensor of the vehicle, so as to achieve anti-shake correction for the on-board device and ensure the accurate implementation of the assisted driving function.

[0081] Taking the example that the at least one sensor of the vehicle includes a gyroscope, an accelerometer, and a magnetometer, and the on-board device includes an on-board optical sensor, as shown in FIG4 , the method may further include the following steps:

[0082] S410: The calibration device weights the confidence of at least one sensor according to prior information to obtain a weight factor associated with the at least one sensor.

[0083] In specific implementation, the calibration device can capture the postures of different traffic participants relative to the vehicle body based on the camera, and the size of the position error estimated by the IMU, and obtain the corresponding weight factor by normalizing the posture errors.

[0084] For example, the weight factors corresponding to the gyroscope, accelerometer and magnetometer can be represented by parameters w1, w2 and w3 respectively, and these weight factors can satisfy the following expression (1): w1+w2+w3=1 (1);

[0085] Among them, the magnetometer input accuracy is usually the highest, the gyroscope has a higher accuracy in a short time, and the accelerometer has a higher accuracy in a long time. The values ​​of w1 and w2 can change over time.

[0086] S420: The calibration device obtains a pose estimation result of the vehicle based on the weight factor associated with the at least one sensor and the perception data of the at least one sensor.

[0087] For example, the calibration device may perform weighted summation based on a weight factor associated with at least one sensor and the sensing data of at least one sensor to obtain a relatively accurate estimated value of each rotation angle.

[0088] S430: The calibration device performs error compensation on the vehicle's onboard components based on the pose estimation result and the three-dimensional rigid body model.

[0089] For example, the calibration device can construct a transformation matrix from the IMU to the photosensitive device based on external parameters, and convert the estimation result of the IMU to the photosensitive device through a three-dimensional rigid body model to perform projection compensation.

[0090] Therefore, through the above method, after calibrating at least one sensor of the vehicle online, an error compensation method can be provided to the vehicle-mounted device that uses the perception data of the at least one sensor, so as to reduce the estimation error of the sensor during vehicle shaking, and also provide vehicle driving safety and provide auxiliary support for the safe driving of the vehicle.

[0091] The embodiment of the present application also provides a calibration device for executing the calibration method performed in the above method embodiment. The relevant features can be found in the above method embodiment and will not be repeated here.

[0092] As shown in Figure 5, the device 500 may include: an identification unit 501 for identifying at least one traffic participant in the environment where the vehicle is located; an acquisition unit 502 for acquiring a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; a determination unit 503 for determining the driving state of the vehicle; an internal parameter calibration unit 504 for calibrating the internal parameters of the sensor according to the first data set if the vehicle does not shake; and an external parameter calibration unit 505 for jointly calibrating the external parameters of at least one sensor of the vehicle according to the first data set if the vehicle shakes. For specific implementation methods, please refer to the method steps implemented by the calibration device in the above method embodiment, which will not be repeated here.

[0093] It should be understood that the division of the various units in the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or they can be physically separated. In addition, the units in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, and the memory stores instructions. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the various units of the device, where the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory within the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all of the units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), which realizes the functions of some or all of the above units by designing the logical relationship of the components in the circuit. For another example, in another implementation, the hardware circuit can be implemented by a programmable logic device (PLD). Taking a field programmable gate array (FPGA) as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, thereby realizing the functions of some or all of the above units. All units of the above devices can be implemented in the form of software called by the processor, or in the form of hardware circuits, or in part by the form of software called by the processor, and the rest by hardware circuits.

[0094] In an embodiment of the present application, a processor is a circuit with a signal processing capability. In one implementation, the processor may be a circuit with instruction reading and execution capability, such as a CPU, a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP); in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and implementing the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0095] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0096] In addition, the various units in the above devices can be fully or partially integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the various units of the device. The type of the at least one processor can be different, for example, including a CPU and FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0097] In a simple embodiment, those skilled in the art can imagine that the communication devices in the above embodiments may all adopt the form shown in FIG. 6 .

[0098] The apparatus 600 shown in FIG6 includes at least one processor 610 and a communication interface 630. In an optional design, a memory 620 may also be included.

[0099] The specific connection medium between the processor 610 and the memory 620 is not limited in the embodiment of the present application.

[0100] In the apparatus shown in FIG. 6 , the processor 610 may transmit data through the communication interface 630 when communicating with other devices.

[0101] When the communication device adopts the form shown in FIG6 , the processor 610 in FIG6 can call the computer-executable instructions stored in the memory 620 so that the device 600 can execute any of the above method embodiments.

[0102] An embodiment of the present application also relates to a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes the method of any of the above embodiments.

[0103] In a possible implementation, the processor may be coupled to the memory through an interface.

[0104] In a possible implementation, the chip system may also directly include a memory, in which a computer program or computer instructions are stored.

[0105] For example, the memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache memory. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0106] An embodiment of the present application further relates to a processor, which is used to call a computer program or computer instruction stored in a memory so that the processor executes the method described in any of the above embodiments.

[0107] For example, in the embodiments of the present application, the processor is an integrated circuit chip with signal processing capabilities. For example, the processor can be an FPGA, a general-purpose processor, a DSP, an ASIC or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, a system on chip (SoC), a CPU, a network processor (NP), a microcontroller unit (MCU), a PLD or other integrated chip, and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0108] It should be understood that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0109] In one possible implementation, an embodiment of the present application provides a computer-readable storage medium, which stores program code. When the program code runs on the computer, the computer executes the above method embodiment.

[0110] In a possible implementation, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the computer is caused to execute the above method embodiment.

[0111] Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0112] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0113] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0114] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these changes and variations. In the various embodiments of the present application, unless otherwise specified or logically conflicting, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

Claims

1. A sensor calibration method, characterized in that: The sensor is one of at least one sensor disposed on a vehicle, and the method comprises: identifying at least one traffic participant in the environment of the vehicle; Acquire a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; determining a driving state of the vehicle; If the vehicle does not shake, calibrate the sensor internally according to the first data set; If the vehicle shakes, external parameters of at least one sensor of the vehicle are jointly calibrated according to the first data set.

2. The method according to claim 1, characterized in that The sensor includes a camera, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: If the vehicle does not shake within the first time period, the first data set is used as a calibration template, and the sensor is calibrated with internal parameters according to multiple data frames collected within the first time period relative to the same traffic participant.

3. The method according to claim 1 or 2, characterized in that: The sensor includes an inertial measurement unit IMU, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: If the vehicle does not shake within the second time period, the first data set is used as a calibration board, and the sensor is calibrated with internal parameters according to the posture perception data collected within the second time period.

4. The method according to any one of claims 1 to 3, characterized in that If the vehicle shakes, jointly calibrating external parameters of at least one sensor of the vehicle according to the first data set includes: During the jittering of the vehicle, the first data set is used as a calibration template, and the external parameters of at least one sensor of the vehicle are jointly calibrated according to the perception data of traffic participants collected during the jittering process.

5. The method according to any one of claims 1 to 4, characterized in that The determining the driving state of the vehicle includes: Acquiring at least one driving parameter through the at least one sensor, wherein the at least one sensor comprises at least one of the following types of sensors: a gyroscope, an accelerometer, or a magnetometer; A driving state of the vehicle is determined according to the at least one driving parameter.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining of a first data set associated with the at least one traffic participant includes: The first data set is received from a cloud server.

7. The method according to claim 6, characterized in that The first data set is obtained according to a target tracking model, and the target tracking model includes a deep network model.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Weighting the confidence of the at least one sensor according to the prior information to obtain a weight factor associated with the at least one sensor; Obtaining a pose estimation result of the vehicle according to a weight factor associated with the at least one sensor and perception data of the at least one sensor; According to the pose estimation result and the three-dimensional rigid body model, error compensation is performed on the on-board components of the vehicle.

9. The method according to claim 8, characterized in that The vehicle-mounted device includes a light-sensing device of the vehicle, and the light-sensing device includes at least one of the following: an augmented reality head-up display AR-HUD device, an optical projection device, and an intelligent ranging device.

10. A sensor calibration device, characterized in that: The sensor is one of at least one sensor disposed on a vehicle, and the device comprises: an identification unit, configured to identify at least one traffic participant in the environment of the vehicle; An acquisition unit, configured to acquire a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; a determination unit, configured to determine a driving state of the vehicle; an internal parameter calibration unit, configured to perform internal parameter calibration on the sensor according to the first data set if the vehicle does not vibrate; An external parameter calibration unit is used to jointly calibrate the external parameters of at least one sensor of the vehicle according to the first data set if the vehicle shakes.

11. An electronic device, characterized in that: comprising a processor coupled to a memory: The processor is configured to execute a computer program or instruction stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 9.

12. A vehicle, characterized in that: The method comprises means for implementing the method as claimed in any one of claims 1 to 9.

13. A readable storage medium, characterized in that: The method comprises a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 9 is executed.

14. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images