Sensor calibration method and device

Abstract

The invention discloses a sensor calibration method and device, and relates to the technical field of sensors. Wherein the sensor is one of at least one sensor arranged on a vehicle, and the method can comprise the steps that at least one traffic participant in the environment where the vehicle is located is recognized; obtaining a first data set associated with the at least one traffic participant, the first data set including standardized data of the at least one traffic participant; determining a driving state of the vehicle; if the vehicle does not shake, internal reference calibration is carried out on the sensor according to the first data set; and if the vehicle shakes, performing joint calibration on external parameters of at least one sensor of the vehicle according to the first data set. The method is used for realizing adaptive calibration of multiple universal sensors so as to improve the fusion precision of sensing information of the sensors.

Description

Technical Field

[0001] The present application relates to the field of sensor technology, and in particular to a sensor calibration method and device. Background Art

[0002] With the rapid development of automobile intelligent technology, the intelligent driving functions of vehicles are becoming more and more perfect, and more and more sensors are installed on vehicles. The measurement accuracy of on-board sensors is crucial to the realization of intelligent driving functions and driving safety.

[0003] Most sensors need to be calibrated after installation to ensure the measurement accuracy of the sensor. Sensor calibration includes internal parameter calibration and external parameter calibration. Internal parameters are a set of parameters that determine the internal mapping relationship of the sensor, and external parameters are parameters that determine the conversion relationship between the sensor's coordinate system and the specified coordinate system. Internal parameter calibration is to obtain the mapping relationship inside the sensor, and external calibration is to convert two or more sensors into a unified spatial coordinate system. Currently, the commonly used sensor calibration methods either need to use three-dimensional radar sensors as a medium to achieve calibration, lacking a general online sensor calibration method, or require a large amount of data sets to train the deep learning model used for calibration, which is difficult to train and is not suitable for real-time calibration of various types of vehicle-mounted sensors in vehicle scenarios.

[0004] Therefore, how to design a calibration method for general vehicle-mounted sensors remains an important issue that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a sensor calibration method and device for implementing adaptive calibration of common multi-sensors to improve the fusion accuracy of sensor perception information.

[0006] On the first aspect, an embodiment of the present application provides a sensor calibration method, which can be implemented by a calibration device. The calibration device can be an independent device, a chip or component in a device, or software. It can be deployed on the vehicle side, or in the cloud, or on a roadside device, or on a remote server, or on a local server, etc. The present application does not limit the product form and deployment method of the calibration device.

[0007] The sensor may be one of at least one sensor arranged on a vehicle, and the method may include: identifying at least one traffic participant in the environment where the vehicle is located; obtaining a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; determining the driving state of the vehicle; if the vehicle does not shake, calibrating the internal parameters of the sensor according to the first data set; if the vehicle shakes, jointly calibrating the external parameters of at least one sensor of the vehicle according to the first data set.

[0008] Through the above method, the calibration device can use the standardized traffic participants of the scene elements as the benchmark instead of the calibration plate to perform internal parameter calibration and / or external parameter calibration on the vehicle-mounted sensor. The calibration method of the vehicle-mounted sensor can be decoupled from the media such as three-dimensional radar sensors, and there is no need to train a deep learning model for calibration with a large data set. The calibration time and labor cost can be greatly saved, making the calibration method universal. At the same time, the error of joint calibration of multiple sensors is reduced, and the fusion accuracy of sensor information is improved.

[0009] In combination with the first aspect, in a possible implementation, the sensor includes a camera, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: if the vehicle does not shake within a first time period, the first data set is used as a calibration board, and the sensor is calibrated with internal parameters according to multiple frames of data collected relative to the same traffic participant within the first time period.

[0010] In combination with the first aspect, in a possible implementation, the sensor includes an inertial measurement unit IMU, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: if the vehicle does not shake within a second time period, the first data set is used as a calibration board, and the sensor is calibrated with internal parameters according to the posture perception data collected within the second time period.

[0011] In combination with the first aspect, in a possible implementation, if the vehicle shakes, the external parameters of at least one sensor of the vehicle are jointly calibrated based on the first data set, including: during the shaking of the vehicle, using the first data set as a calibration board, and according to the perception data of traffic participants collected during the shaking process, the external parameters of at least one sensor of the vehicle are jointly calibrated.

[0012] In combination with the first aspect, in a possible implementation, determining the driving state of the vehicle includes: acquiring at least one driving parameter through the at least one sensor, wherein the at least one sensor includes at least one of the following types of sensors: a gyroscope, an accelerometer, or a magnetometer; and determining the driving state of the vehicle based on the at least one driving parameter.

[0013] In combination with the first aspect, in a possible implementation manner, obtaining the first data set associated with the at least one traffic participant includes: receiving the first data set from a cloud server.

[0014] In combination with the first aspect, in a possible implementation manner, the first data set is obtained according to a target tracking model, and the target tracking model includes a deep network model.

[0015] In combination with the first aspect, in a possible implementation, the method also includes: weighting the confidence of the at least one sensor based on prior information to obtain a weight factor associated with the at least one sensor; obtaining a posture estimation result of the vehicle based on the weight factor associated with the at least one sensor and the perception data of the at least one sensor; and performing error compensation on the on-board devices of the vehicle based on the posture estimation result and the three-dimensional rigid body model.

[0016] In combination with the first aspect, in a possible implementation, the vehicle-mounted device includes a light-sensing device of the vehicle, and the light-sensing device includes at least one of the following: an augmented reality head-up display AR-HUD device, an optical projection device, and an intelligent ranging device.

[0017] In a second aspect, an embodiment of the present application provides a sensor calibration device, wherein the sensor is one of at least one sensor arranged on a vehicle, and the device includes: an identification unit, used to identify at least one traffic participant in the environment where the vehicle is located; an acquisition unit, used to acquire a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; a determination unit, used to determine the driving state of the vehicle; an internal parameter calibration unit, used to perform internal parameter calibration of the sensor according to the first data set if the vehicle does not shake; and an external parameter calibration unit, used to perform joint calibration of external parameters of at least one sensor of the vehicle according to the first data set if the vehicle shakes.

[0018] In combination with the second aspect, in a possible implementation, the sensor includes a camera, and the internal parameter calibration unit is specifically used to: if the vehicle does not shake within a first time period, use the first data set as a calibration plate, and perform internal parameter calibration on the sensor based on multiple frames of data relative to the same traffic participant collected within the first time period.

[0019] In combination with the second aspect, in a possible implementation method, the sensor includes an inertial measurement unit IMU, and the intrinsic parameter calibration unit is specifically used to: if the vehicle does not shake within the second time length, use the first data set as a calibration board, and perform intrinsic parameter calibration on the sensor according to the posture perception data collected within the second time length.

[0020] In combination with the second aspect, in a possible implementation method, the external parameter calibration unit is specifically used to: during the shaking of the vehicle, use the first data set as a calibration board, and jointly calibrate the external parameters of at least one sensor of the vehicle according to the perception data of traffic participants collected during the shaking process.

[0021] In combination with the second aspect, in a possible implementation, the determination unit is specifically used to: obtain at least one driving parameter through the at least one sensor, wherein the at least one sensor includes at least one of the following types of sensors: a gyroscope, an accelerometer or a magnetometer; and determine the driving state of the vehicle based on the at least one driving parameter.

[0022] In combination with the second aspect, in a possible implementation manner, the acquisition unit is specifically used to: receive the first data set from a cloud server.

[0023] In combination with the second aspect, in a possible implementation manner, the first data set is obtained according to a target tracking model, and the target tracking model includes a deep network model.

[0024] In combination with the second aspect, in a possible implementation method, the device also includes an error compensation unit, which is used to weight the confidence of the at least one sensor according to prior information, and obtain a weight factor associated with the at least one sensor; obtain a posture estimation result of the vehicle according to the weight factor associated with the at least one sensor and the perception data of the at least one sensor; and perform error compensation on the on-board devices of the vehicle according to the posture estimation result and the three-dimensional rigid body model.

[0025] In combination with the second aspect, in a possible implementation, the vehicle-mounted device includes a light-sensing device of the vehicle, and the light-sensing device includes at least one of the following: an augmented reality head-up display AR-HUD device, an optical projection device, and an intelligent ranging device.

[0026] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, which is coupled to a memory: the processor is used to execute a computer program or instruction stored in the memory, so that the electronic device executes the method described in the first aspect and any possible implementation method of the first aspect.

[0027] In a fourth aspect, an embodiment of the present application provides a vehicle, comprising a unit for implementing the method described in the first aspect and any possible implementation of the first aspect.

[0028] In a fifth aspect, an embodiment of the present application provides a readable storage medium, including a program or an instruction. When the program or the instruction is executed, the method described in the first aspect and any possible implementation method of the first aspect is executed.

[0029] In a sixth aspect, an embodiment of the present application provides a computer program product. When the computer program product runs on a computer, the computer executes the method described in the first aspect and any possible implementation of the first aspect.

[0030] In a seventh aspect, an embodiment of the present application provides a terminal device, including a unit for implementing the method described in the first aspect and any possible design of the first aspect. By way of example, the terminal device includes, but is not limited to: intelligent transportation equipment (such as cars, ships, drones, trains, trucks, etc.), intelligent manufacturing equipment (such as robots, industrial equipment, intelligent logistics, intelligent factories, etc.), and intelligent terminals (mobile phones, computers, tablet computers, PDAs, desktops, headphones, speakers, wearable devices, vehicle-mounted devices, etc.).

[0031] Based on the implementations provided in the above aspects, the embodiments of the present application can be further combined to provide more implementations.

[0032] The technical effects that can be achieved by any possible implementation method in any of the second to seventh aspects mentioned above can be referred to the description of the technical effects that can be achieved by any possible implementation method in any of the first aspect mentioned above, and the repetitions will not be discussed here. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram showing an application scenario to which the embodiments of the present application are applicable;

[0034] Figure 2 A schematic diagram showing the principle of the calibration method of an embodiment of the present application is shown;

[0035] Figure 3 A schematic diagram showing a flow chart of a calibration method according to an embodiment of the present application is shown;

[0036] Figure 4 A schematic diagram showing a flow chart of a calibration method according to an embodiment of the present application is shown;

[0037] Figure 5 A schematic diagram showing the structure of a calibration device according to an embodiment of the present application is shown;

[0038] Figure 6 A schematic diagram of the structure of a communication device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0039] The embodiments of the present application provide a sensor calibration method and device for realizing adaptive calibration of a general multi-sensor to improve the fusion accuracy of the sensor's perception information. Among them, the method and the device are based on the same technical concept. Since the principles of solving problems by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated. In addition, in each embodiment of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment according to their inherent logical relationship.

[0040] It should be noted that in the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0041] Furthermore, unless otherwise specified, the ordinal numbers such as "first" and "second" mentioned in the embodiments of the present application are used to distinguish multiple objects, and are not used to limit the priority or importance of multiple objects. For example, the first sensor and the second sensor are only used to distinguish different sensors, rather than indicating the difference in priority or importance of the two sensors.

[0042] The sensor calibration scheme proposed in the embodiment of the present application can be applied to various types of sensors, illustratively, but not limited to gyroscopes, accelerometers, magnetometers, or vehicle-mounted optical sensing devices, such as augmented reality head-up display (AR-HUD) devices, optical projection devices (such as smart car lights), and intelligent distance measurement devices (such as adaptive distance measurement devices). More specifically, the sensors to which the calibration scheme is applicable can be applied in vehicle scenarios, as well as in drone scenarios, aircraft scenarios, and the like. In these scenarios, due to road conditions (such as road surface flatness) or environmental (such as weather effects such as wind and airflow), the vehicle body where the sensor is located may shake violently, thereby causing the problem of perception error. It is necessary to calibrate the sensors used in a timely manner or compensate for the perception information in order to achieve more accurate detection.

[0043] For ease of understanding, the following is an introduction to the specific details of the method by taking the calibration method of sensors applied to the vehicle field as an example. It should be understood that the vehicle here is an example of intelligent transportation equipment, and the vehicle can specifically be a car, a ship, a drone, a train, a truck, etc. In other optional embodiments, the vehicle can be replaced with intelligent manufacturing equipment, including but not limited to robots, industrial equipment, intelligent logistics, intelligent factory equipment, etc. In other optional embodiments, the vehicle can be replaced with an intelligent terminal, including but not limited to mobile phones, computers, tablets, PDAs, desktops, headphones, speakers, wearable devices, vehicle-mounted equipment, etc., which will not be repeated here.

[0044] Figure 1 A schematic diagram of an application scenario to which the embodiment of the present application is applicable is shown. In the application scenario, a vehicle 100 may be included. In a possible implementation, the application scenario may also include a cloud server 200, and the vehicle 100 and the cloud server 200 may communicate via a network. In one embodiment, the cloud server 200 may also be implemented by a virtual machine.

[0045] Some or all functions of the vehicle 100 are controlled by a computing platform 150 (or computer system). The computing platform 150 may include at least one processor 151, which may execute instructions 153 stored in a non-transitory computer-readable medium such as a memory 152. In some embodiments, the computing platform 150 may also be a plurality of computing devices that control individual components or subsystems of the vehicle 100 in a distributed manner. The processor 151 may be any conventional processor, such as a central processing unit (CPU). Alternatively, the processor 151 may also include a graphics processing unit (GPU), a field programmable gate array (FPGA), a system on chip (SoC), an application specific integrated circuit (ASIC), or a combination thereof.

[0046] Optionally, the vehicle 100 may be a car, a truck, a motorcycle, a bus, a ship, an airplane, a helicopter, a lawn mower, an amusement vehicle, an amusement park vehicle, construction equipment, a tram, a golf cart, a train, etc., which is not particularly limited in the embodiments of the present application. In a possible implementation, the vehicle 100 may be an electric vehicle (EV), such as a two-wheel drive electric vehicle or a four-wheel drive electric vehicle, which is not limited in the embodiments of the present application.

[0047] Figure 1 The structure of the vehicle should not be understood as a limitation to the embodiments of the present application.

[0048] The calibration method of the embodiment of the present application can be implemented by a calibration device, which can be an independent device or a Figure 1 The chip or component in the vehicle 100 shown may also be a software module, which may be deployed on the relevant on-board equipment of the vehicle 100. The embodiment of the present application does not limit the product form and deployment method of the calibration device. In specific implementation, the calibration of the internal parameters and / or external parameters of different sensors may be implemented by the same calibration device, or the calibration of the internal parameters and / or external parameters of associated sensors may be implemented by different calibration devices. The embodiment of the present application does not limit this. In the following, for the sake of ease of understanding and description, the calibration method of the embodiment of the present application is introduced by taking the calibration method of the internal parameters and / or external parameters of different sensors implemented by the same calibration device as an example.

[0049] See also Figure 2As shown, the calibration device can interact with other modules in the vehicle to implement the calibration scheme. For example, the vehicle can have a sensing system. On the one hand, the at least one sensor can include an inertial measurement unit (IMU), such as a gyroscope, an accelerometer or a magnetometer, which can be used to collect different driving parameters of the vehicle, such as the angular velocity, angular acceleration, and heading angle of the vehicle, so as to evaluate the driving state of the vehicle, for example, whether the vehicle is currently in a parked state, or in a stable running state, or in a shaking state (such as shaking caused by bumpy road conditions, shaking caused by speed bumps, etc., which is not limited in the embodiments of the present application).

[0050] On the other hand, the at least one sensor may include but is not limited to a camera, a laser radar (light detection and ranging, LIDAR), a millimeter-wave radar (millimeter-wave radar, RADAR) or other sensor devices, which can always monitor and collect data about the environment around the vehicle, and identify at least one traffic participant in the environment where the vehicle is currently located. The at least one traffic participant may be any traffic participant in the environment where the vehicle is located except the vehicle itself, including but not limited to the current lane of the vehicle, and the lane line of the current lane; the front and rear vehicles of the vehicle in the current lane; the adjacent lane of the current lane, and the lane lines of the adjacent lane; several other vehicles on the adjacent lane; pedestrians in the current lane, or adjacent lanes, or at the intersection of roads; various traffic facilities: such as crosswalks, road side units (Road Side Unit, RSU), traffic lights (such as motor vehicle lights, non-motor vehicle lights, crosswalk lights, direction indicators, lane lights, flashing lights, road and railway level crossing road lights, etc.), fences, lighting facilities, sight guidance signs, highway reflectors, highway information boards, etc. In an optional embodiment, the at least one road user may be a road user having a standardized size.

[0051] The calibration device may make a calibration decision according to the driving state of the vehicle and the first data set associated with the first traffic participant to determine how to perform an internal parameter calibration and / or an external parameter calibration on at least one sensor of the vehicle.

[0052] For example, the first data set may include standardized data of at least one traffic participant. If it is detected that the vehicle is not shaking, for example, the vehicle is in a stationary state or a stable running state, the calibration device may use the first data set as a calibration board, and perform internal parameter calibration on the vehicle's sensors according to the first data set, such as internal parameter calibration on the gyroscope, or internal parameter calibration on the accelerometer, or internal parameter calibration on the magnetometer, etc. If it is detected that the vehicle is shaking, the calibration device may use the first data set as a calibration board, and perform joint calibration on the external parameters of at least one sensor of the vehicle according to the first data set, such as joint calibration of the external parameters of the gyroscope, accelerometer, and magnetometer.

[0053] It should be understood that in specific implementation, the standardized data associated with traffic participants can be stored on a media medium accessible to the calibration device, such as a vehicle storage medium, a cloud server, or a third-party storage device, etc., and the embodiments of the present application do not limit this.

[0054] After completing the calibration, in an optional embodiment, the calibration device can also weight the confidence of different sensors according to the relative position of at least one identified traffic participant and the output value of the IMU, obtain the weight factor, solve the final pose estimation result by complementary filtering, and construct the transformation matrix of the IMU and other vehicle-mounted devices according to the external parameters, so as to transform the estimation result of the IMU to the vehicle-mounted device through the three-dimensional rigid body model and perform error compensation. Exemplarily, the vehicle-mounted device can be a photosensitive device, including but not limited to AR-HUD devices, optical projection devices (such as smart car lights), intelligent ranging devices (such as adaptive ranging devices), etc.

[0055] The above method uses the scene element standardized traffic participants as the benchmark instead of the calibration plate to perform internal and / or external parameter calibration on the vehicle-mounted sensor. This can decouple the calibration method of the vehicle-mounted sensor from the medium such as the three-dimensional radar sensor, and does not require a large amount of data sets to train the deep learning model used for calibration. This can greatly save calibration time and labor costs, making the calibration method universal, while reducing the error of joint calibration of multiple sensors and improving the fusion accuracy of sensor information. This method can reduce the estimation error of the sensor during vehicle shaking, and can also provide vehicle driving safety and provide auxiliary support for safe vehicle driving.

[0056] For ease of understanding, the calibration method of the embodiment of the present application is introduced below by taking the vehicle-mounted device as a vehicle-mounted light sensing device as an example.

[0057] like Figure 3 As shown, the calibration method may include the following steps:

[0058] S310: The calibration device identifies at least one traffic participant in the environment where the vehicle is located.

[0059] The at least one traffic participant may include any traffic participant in the vehicle's environment except the vehicle itself, including but not limited to the vehicle's current lane and the lane line of the current lane; the vehicle in front and behind the vehicle in the current lane; the adjacent lane of the current lane and the lane lines of the adjacent lane; several other vehicles on the adjacent lane; pedestrians in the current lane, adjacent lane, or road intersection; various traffic facilities: such as crosswalks, roadside units (RSU), traffic lights (such as motor vehicle lights, non-motor vehicle lights, crosswalk lights, direction indicators, lane lights, flashing lights, road and railway intersection lights, etc.), fences, lighting facilities, sight-guiding signs, highway reflectors, highway information boards, etc.

[0060] In an optional implementation, the at least one traffic participant may be a traffic participant with standardized size. The standardization may refer to an international standard, a national standard, a standard of a province, city, county, township, etc., or a standard of a designated area (such as a smart factory). The implementation method of this standardization is not limited in the embodiment of the present application.

[0061] When S310 is specifically implemented, the calibration device may identify at least one traffic participant in the environment where the vehicle is located through at least one sensor provided on the vehicle and a target detection algorithm.

[0062] S320: The calibration device obtains a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant.

[0063] In one example, when S320 is implemented, the calibration device may receive the first data set from the cloud server in real time. Alternatively, in another example, the local storage device of the vehicle may pre-store standardized data of various common traffic participants, and when S320 is implemented, the calibration device may directly obtain the first data set from the local storage device of the vehicle. The embodiment of the present application does not limit the method for obtaining the first data set.

[0064] It is worth noting that within a certain period of time, the standardized data of the same traffic participant is usually stable and unchanged. The calibration device only needs to execute S320 once within the period of time, and there is no need to repeat it, which can reduce energy consumption.

[0065] In an optional embodiment, the standardized data in the first data set can be obtained according to a target tracking model, which may include a deep network model, such as a DarkNet network. The target tracking model can construct a standard traffic sign data set (including pictures and annotation information) based on the obstacle information in the vehicle's current environment, and complete the training of the first data set based on transfer learning. In an optional embodiment, at least 50 pieces of each type of data can be obtained, and the training iteration is more than 5,000 times to ensure the accuracy of the standardized data obtained.

[0066] S330: The calibration device determines the driving state of the vehicle.

[0067] In a specific implementation, the calibration device obtains at least one driving parameter through at least one sensor of the vehicle, and determines the driving state of the vehicle according to the at least one driving parameter.

[0068] Exemplarily, the at least one sensor may include at least one of the following types of sensors: a gyroscope, an accelerometer, or a magnetometer. The gyroscope, accelerometer, or magnetometer may be disposed at the center of the roof of the vehicle body, or may be disposed on the same vertical line as the center of gravity of the vehicle body and closer to a horizontal setting. The gyroscope may be used to collect the angular velocity of three rotational degrees of freedom of the vehicle body (e.g., yaw angle, roll angle, or pitch angle). In an optional embodiment, the angular acceleration value may also be obtained by integrating the angular velocity value over time. The accelerometer may be used to collect the acceleration of the three rotational degrees of freedom of the vehicle body. The magnetometer may be used to measure the vehicle body in one rotational degree of freedom (e.g., yaw angle).

[0069] Take the evaluation of the driving state of a vehicle based on acceleration as an example. An acceleration threshold value can be preset, such as 1m / s^2. If the acceleration value obtained by the calibration device is lower than the acceleration threshold value, it can be considered that the vehicle is in a stable running state. If the acceleration value obtained by the calibration device is greater than or equal to the acceleration threshold value, it can be considered that the vehicle is shaking. If the vehicle linear velocity and acceleration values ​​obtained by the calibration device are both lower than the corresponding thresholds (for example, the linear velocity threshold is 1m / s, and the acceleration threshold is 1m / s^2), it can be considered that the vehicle is in a parked state.

[0070] It should be understood that this is merely an example of the method for determining the driving state of a vehicle in an embodiment of the present application, and not a limitation. In other embodiments, the calibration device may also determine the driving state of the vehicle by other means, which will not be repeated here.

[0071] S340: If the vehicle does not shake, the calibration device performs internal parameter calibration on the sensor according to the first data set.

[0072] For example, taking the sensor as a camera, when S340 is specifically implemented, if the vehicle does not shake within the first time period, for example, it is in a stable running state, the calibration device uses the first data set as a calibration plate, and performs internal parameter calibration on the sensor according to multiple frames of data collected within the first time period relative to the same traffic participant. In this process, the calibration device can obtain the internal parameter information of the camera, such as the affine transformation matrix from the image coordinate system to the left and right of the pixel, by using the Zhang Zhengyou internal parameter calibration method.

[0073] Exemplarily, taking the sensor as an IMU (e.g., including a gyroscope, an accelerometer, and a magnetometer), when S340 is specifically implemented, the vehicle does not shake within the second time period, for example, it is in a stable running state, and the calibration device uses the first data set as a calibration board, and performs internal parameter calibration on the sensor according to the posture perception data collected within the second time period. In this process, the calibration device can, for example, use the kalibr calibration tool to complete the internal parameter calibration.

[0074] S350: If the vehicle shakes, the calibration device jointly calibrates the external parameters of at least one sensor of the vehicle according to the first data set.

[0075] For example, the sensor is an IMU (for example, including a gyroscope, an accelerometer, and a magnetometer). When S350 is implemented, during the jittering of the vehicle, the calibration device can use the first data set as a calibration board, and perform joint calibration on the external parameters of at least one sensor of the vehicle according to the perception data of the traffic participants or the vehicle's own state information collected during the jittering process. In this process, the calibration device can, for example, use the kalibr calibration tool to complete the external parameter calibration.

[0076] Therefore, through the above method, the calibration device can use the standardized traffic participants of the scene elements as a benchmark instead of the calibration plate to perform internal parameter calibration and / or external parameter calibration on the vehicle-mounted sensors. The calibration method of the vehicle-mounted sensors can be decoupled from media such as three-dimensional radar sensors, and there is no need to train a deep learning model for calibration with a large data set. The calibration time and labor costs can be greatly saved, making the calibration method universal. At the same time, the error of joint calibration of multiple sensors is reduced, and the fusion accuracy of sensor information is improved.

[0077] After completing the calibration of the sensor, in an optional implementation, the calibration device can also perform error compensation on the on-board device that provides assisted driving function based on at least one sensor of the vehicle, so as to achieve anti-shake correction for the on-board device and ensure the accurate implementation of the assisted driving function.

[0078] For example, at least one sensor of a vehicle includes a gyroscope, an accelerometer and a magnetometer, and the vehicle-mounted device includes a vehicle-mounted optical sensing device. Figure 4 As shown, the method may also include the following steps:

[0079] S410: The calibration device weights the confidence of at least one sensor according to prior information to obtain a weight factor associated with the at least one sensor.

[0080] In specific implementation, the calibration device can capture the postures of different traffic participants relative to the vehicle body based on the camera, and the size of the position error estimated by the IMU, and obtain the corresponding weight factor by normalizing the errors of each posture.

[0081] Exemplarily, the weight factors corresponding to the gyroscope, accelerometer and magnetometer can be represented by parameters w1, w2 and w3 respectively, and these weight factors can satisfy the following expression (1):

[0082] w1+w2+w3=1 (1);

[0083] Among them, the magnetometer input accuracy is usually the highest, the gyroscope has a higher accuracy in a short time, and the accelerometer has a higher accuracy in a long time. The values ​​of w1 and w2 can change over time.

[0084] S420: The calibration device obtains a pose estimation result of the vehicle according to a weight factor associated with the at least one sensor and perception data of the at least one sensor.

[0085] For example, the calibration device may perform weighted summation based on a weight factor associated with at least one sensor and sensing data of at least one sensor to obtain a relatively accurate estimation value of each rotation angle.

[0086] S430: The calibration device performs error compensation on the on-board components of the vehicle according to the pose estimation result and the three-dimensional rigid body model.

[0087] For example, the calibration device can construct a transformation matrix from the IMU to the photosensitive device based on external parameters, and convert the estimation result of the IMU to the photosensitive device through a three-dimensional rigid body model to perform projection compensation.

[0088] Therefore, through the above method, after calibrating at least one sensor of the vehicle online, an error compensation method can be provided to the vehicle-mounted device that utilizes the perception data of the at least one sensor, so as to reduce the estimation error of the sensor during vehicle shaking, and also provide vehicle driving safety and provide auxiliary support for the safe driving of the vehicle.

[0089] The embodiment of the present application also provides a calibration device for executing the method performed by the calibration in the above method embodiment. The relevant features can be found in the above method embodiment and will not be repeated here.

[0090] like Figure 5 As shown, the device 500 may include: an identification unit 501, used to identify at least one traffic participant in the environment where the vehicle is located; an acquisition unit 502, used to acquire a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; a determination unit 503, used to determine the driving state of the vehicle; an internal parameter calibration unit 504, used to calibrate the internal parameters of the sensor according to the first data set if the vehicle does not shake; an external parameter calibration unit 505, used to jointly calibrate the external parameters of at least one sensor of the vehicle according to the first data set if the vehicle shakes. For specific implementation methods, please refer to the method steps implemented by the calibration device in the above method embodiment, which will not be repeated here.

[0091] It should be understood that the division of the units in the above device is only a division of logical functions, and in actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the units in the device can be implemented in the form of a processor calling software; for example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of the units of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all of the units can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by a configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be realized in the form of a processor calling software, or in the form of a hardware circuit, or in part by a processor calling software, and the rest by a hardware circuit.

[0092] In the embodiment of the present application, the processor is a circuit with signal processing capability. In one implementation, the processor can be a circuit with instruction reading and running capability, such as a CPU, a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP), etc.; in another implementation, the processor can realize certain functions through the logical relationship of the hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the processor loads a configuration document to implement the process of hardware circuit configuration, which can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (Neural Network Processing Unit, NPU) tensor processing unit (Tensor Processing Unit, TPU), deep learning processing unit (Deep learning Processing Unit, DPU), etc.

[0093] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0094] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0095] In a simple embodiment, those skilled in the art can imagine that the communication devices in the above embodiments can all be used Figure 6 The form shown.

[0096] like Figure 6 The device 600 shown includes at least one processor 610 and a communication interface 630. In an optional design, a memory 620 may also be included.

[0097] The specific connection medium between the processor 610 and the memory 620 is not limited in the embodiment of the present application.

[0098] In such Figure 6 In the device, the processor 610 can transmit data through the communication interface 630 when communicating with other devices.

[0099] When the communication device uses Figure 6 When the form shown is Figure 6 The processor 610 in the apparatus 600 can call the computer-executable instructions stored in the memory 620 so that the apparatus 600 can execute any of the above method embodiments.

[0100] An embodiment of the present application also relates to a chip system, which includes a processor for calling a computer program or computer instructions stored in a memory so that the processor executes the method of any of the above embodiments.

[0101] In a possible implementation manner, the processor may be coupled to the memory through an interface.

[0102] In a possible implementation, the chip system may also directly include a memory, in which a computer program or computer instructions are stored.

[0103] For example, the memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM).

[0104] An embodiment of the present application also relates to a processor, which is used to call a computer program or computer instruction stored in a memory so that the processor executes the method described in any of the above embodiments.

[0105] For example, in the embodiment of the present application, the processor is an integrated circuit chip with signal processing capabilities. For example, the processor can be an FPGA, a general-purpose processor, a DSP, an ASIC or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, a system chip (system on chip, SoC), a CPU, a network processor (network processor, NP), a microcontroller (microcontroller unit, MCU), a PLD or other integrated chip, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and performed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0106] It should be understood that the embodiments of the present application may be provided as methods, systems, or computer program products.

[0107] In a possible implementation, an embodiment of the present application provides a computer-readable storage medium, which stores a program code. When the program code runs on the computer, the computer executes the above method embodiment.

[0108] In a possible implementation, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the above method embodiment.

[0109] Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0110] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0111] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0112] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these changes and variations. In the various embodiments of the present application, if there is no special explanation and logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form new embodiments according to their inherent logical relationships.

Claims

1. A sensor calibration method, characterized in that: The sensor is one of at least one sensor disposed on a vehicle, and the method comprises: identifying at least one traffic participant in the environment of the vehicle; Acquire a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; determining a driving state of the vehicle; If the vehicle does not shake, calibrate the sensor internally according to the first data set; If the vehicle shakes, external parameters of at least one sensor of the vehicle are jointly calibrated according to the first data set.

2. The method according to claim 1, characterized in that The sensor includes a camera, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: If the vehicle does not shake within the first time period, the first data set is used as a calibration template, and the sensor is calibrated with internal parameters according to multiple data frames collected within the first time period relative to the same traffic participant.

3. The method according to claim 1 or 2, characterized in that: The sensor includes an inertial measurement unit IMU, and if the vehicle does not shake, the sensor is calibrated with internal parameters according to the first data set, including: If the vehicle does not shake within the second time period, the first data set is used as a calibration board, and the sensor is calibrated with internal parameters according to the posture perception data collected within the second time period.

4. The method according to any one of claims 1 to 3, characterized in that If the vehicle shakes, jointly calibrating external parameters of at least one sensor of the vehicle according to the first data set includes: During the jittering of the vehicle, the first data set is used as a calibration template, and the external parameters of at least one sensor of the vehicle are jointly calibrated according to the perception data of traffic participants collected during the jittering process.

5. The method according to any one of claims 1 to 4, characterized in that The determining the driving state of the vehicle includes: Acquiring at least one driving parameter through the at least one sensor, wherein the at least one sensor comprises at least one of the following types of sensors: a gyroscope, an accelerometer, or a magnetometer; A driving state of the vehicle is determined according to the at least one driving parameter.

6. The method according to any one of claims 1 to 5, characterized in that The obtaining of a first data set associated with the at least one traffic participant includes: The first data set is received from a cloud server.

7. The method according to claim 6, characterized in that The first data set is obtained according to a target tracking model, and the target tracking model includes a deep network model.

8. The method according to any one of claims 1 to 7, characterized in that The method further comprises: Weighting the confidence of the at least one sensor according to the prior information to obtain a weight factor associated with the at least one sensor; Obtaining a pose estimation result of the vehicle according to a weight factor associated with the at least one sensor and perception data of the at least one sensor; According to the pose estimation result and the three-dimensional rigid body model, error compensation is performed on the on-board components of the vehicle.

9. The method according to claim 8, characterized in that The vehicle-mounted device includes a light-sensing device of the vehicle, and the light-sensing device includes at least one of the following: an augmented reality head-up display AR-HUD device, an optical projection device, and an intelligent ranging device.

10. A sensor calibration device, characterized in that: The sensor is one of at least one sensor disposed on a vehicle, and the device comprises: an identification unit, configured to identify at least one traffic participant in the environment of the vehicle; An acquisition unit, configured to acquire a first data set associated with the at least one traffic participant, wherein the first data set includes standardized data of the at least one traffic participant; a determination unit, configured to determine a driving state of the vehicle; an internal parameter calibration unit, configured to perform internal parameter calibration on the sensor according to the first data set if the vehicle does not vibrate; An external parameter calibration unit is used to jointly calibrate the external parameters of at least one sensor of the vehicle according to the first data set if the vehicle shakes.

11. An electronic device, characterized in that: comprising a processor coupled to a memory: The processor is configured to execute a computer program or instruction stored in the memory, so that the electronic device executes the method as claimed in any one of claims 1 to 9.

12. A vehicle, characterized in that: The method comprises means for implementing the method as claimed in any one of claims 1 to 9.

13. A readable storage medium, characterized in that: The method comprises a program or an instruction, and when the program or the instruction is executed, the method according to any one of claims 1 to 9 is executed.

14. A computer program product, characterized in that When the computer program product is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 9.

Images