Multi-sensor joint calibration method, system, vehicle and equipment

Abstract

The invention discloses a multi-sensor joint calibration method and system, a vehicle and equipment. The multi-sensor joint calibration method comprises the steps that horizontal marks and masks of vertical objects in images collected by a plurality of sensors of a vehicle are extracted, and the vehicle is parked in a scene provided with ground mark lines and the vertical objects in advance; a matching relationship among the plurality of sensors is constructed, constraint conditions of the matching relationship among the plurality of sensors are determined, and the matching relationship comprises external parameters to be solved between any two sensors; and when the number of the targets in the scene is greater than a predetermined number, optimizing the external parameters among the plurality of sensors according to a constraint condition until an optimization exit condition is reached, and obtaining the external parameters among the plurality of optimized sensors. By adopting the method and the device, joint calibration of multiple sensors in a target-free natural scene is realized, dependence on a site and a target is effectively reduced, calibration and function integration is realized, and the method and the device have the advantages of simplicity and convenience in calibration.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a multi-sensor joint calibration method, system, vehicle and equipment. Background Art

[0002] In the intelligent driving of vehicles, multiple sensors such as multiple cameras need to be calibrated in advance. The existing multi-sensor joint calibration relies on specific targets, such as customized targets. The other method relies on a large amount of scene data to establish a matching relationship in three-dimensional space, and then calculate the external parameters of multiple sensors. There are the following technical problems:

[0003] Relying on the planar characteristics of lane lines, a large amount of data is required to improve calibration accuracy.

[0004] The vehicle needs to be manually driven to move along the center of the lane line, so the result is uncertain and time-consuming and laborious. Summary of the invention

[0005] Based on this, it is necessary to provide a multi-sensor joint calibration method, system, vehicle and equipment to address the above-mentioned technical problems, which realizes the joint calibration of multiple sensors in targetless natural scenes, effectively reduces the dependence on sites and targets, realizes the integration of calibration and function, and has the advantage of simple and convenient calibration.

[0006] In a first aspect, a multi-sensor joint calibration method is provided, comprising:

[0007] Extracting masks of horizontal markings and vertical objects in images acquired by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground markings and vertical objects;

[0008] Constructing a matching relationship between a plurality of sensors and determining constraint conditions of the matching relationship between the plurality of sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors;

[0009] When the number of targets in the scene is greater than a predetermined number, the external parameters between the multiple sensors are optimized according to the constraint conditions until an optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

[0010] In some examples, the horizontal identifier and the vertical object mask are represented as a horizontal object attribute H_obj and a vertical object attribute V_obj, respectively.

[0011] In some examples, the constraints include:

[0012] The matching relationship between the multiple sensors satisfies min(H_obj_m−f(H_obj_n)) and n_V_obj*H_obj=0, where f(H_obj_n)) represents the value of the mask area projected from sensor n to sensor m, and n_V_obj represents the normal vector of the plane perpendicular to the target attribute V_obj.

[0013] In some examples, when the number of targets in the scene is greater than a predetermined number, optimizing the external parameters between the multiple sensors according to the constraint condition until an optimization exit condition is reached, and obtaining the optimized external parameters between the multiple sensors includes:

[0014] Determining whether the number of targets in the scene is greater than a predetermined number, wherein the targets in the scene include horizontal markers and vertical objects;

[0015] If yes, optimizing the external parameters between the multiple sensors according to the constraint condition and the matching relationship between the multiple sensors;

[0016] Determine whether the number of optimizations reaches the predetermined number;

[0017] If yes, the optimized external parameters between the multiple sensors are obtained.

[0018] In some examples, the ground marking lines include lane markings, and the vertical objects include utility poles.

[0019] In a second aspect, a multi-sensor joint calibration system is provided, comprising:

[0020] An extraction module, for extracting masks of horizontal markings and vertical objects in images collected by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground marking lines and vertical objects;

[0021] A construction module, used to construct a matching relationship between multiple sensors and determine the constraint conditions of the matching relationship between the multiple sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors;

[0022] The calibration module is used to optimize the external parameters between the multiple sensors according to the constraint conditions when the number of targets in the scene is greater than a predetermined number, until the optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

[0023] In a third aspect, a vehicle is provided, comprising: a multi-sensor joint calibration system according to the second aspect.

[0024] In a fourth aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the multi-sensor joint calibration method of the first aspect and any possible implementation method of the first aspect are implemented.

[0025] In a fifth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the multi-sensor joint calibration method of the above-mentioned first aspect and any possible implementation method of the first aspect are implemented.

[0026] In a sixth aspect, a computer program product is provided, on which a computer program is stored. When the program is executed by a processor, the steps of the multi-sensor joint calibration method of the above-mentioned first aspect and any possible implementation method of the first aspect are implemented.

[0027] According to the embodiment of the present application, the masks of horizontal marks and vertical objects in the images collected by multiple sensors of a vehicle parked in a scene with ground marking lines and vertical objects are first extracted, and then the matching relationship between the multiple sensors is constructed, and the constraints of the matching relationship between the multiple sensors are determined. Finally, according to the constraints and the number of optimizations, the external parameters between the multiple sensors are optimized. Thus, the joint calibration of multiple sensors in targetless natural scenes is realized, which effectively reduces the dependence on the site and the target, realizes the integration of calibration and function, and has the advantages of simple and convenient calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0029] Figure 1 A flowchart of a multi-sensor joint calibration method provided in an embodiment of the present application;

[0030] Figure 2 A structural block diagram of a multi-sensor joint calibration system provided in an embodiment of the present application;

[0031] Figure 3 A structural block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The present application is further described in detail below in conjunction with the embodiments and drawings. It is to be understood that the specific embodiments described herein are only used to explain the relevant application, rather than to limit the application. It is also necessary to explain that, for ease of description, only the parts related to the application are shown in the drawings.

[0033] It should be noted that, in the absence of conflict, the embodiments of the present application, that is, the features of the embodiments, can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0034] The following describes in detail the multi-sensor joint calibration method, system, device and medium according to the embodiments of the present application in conjunction with the accompanying drawings.

[0035] Figure 1 FIG. 1 is a flow chart of a method for joint calibration of multiple sensors according to an embodiment of the present application. Figure 1 As shown, the joint calibration method of multiple sensors according to an embodiment of the present application includes the following steps:

[0036] S101: extracting masks of horizontal markings and vertical objects in images collected by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground marking lines and vertical objects.

[0037] Among them, the ground marking lines include lane lines, and the vertical objects include electric poles. For example, a vehicle is parked in a scene containing ground marking lines and vertical objects, such as lane lines and electric poles. Then, the masks of horizontal and vertical objects obtained by multiple sensors can be extracted through a pre-trained mask extraction model, which are the horizontal target attribute H_obj and the vertical target attribute V_obj. In other words, the masks of horizontal markings and vertical objects are represented as the horizontal target attribute H_obj and the vertical target attribute V_obj, respectively.

[0038] In one specific example, the sensor may be a camera.

[0039] S102: Constructing a matching relationship between a plurality of sensors, and determining constraint conditions of the matching relationship between the plurality of sensors, wherein the matching relationship includes an external parameter to be solved between any two sensors.

[0040] In one embodiment of the present application, the constraints include: the matching relationship between the multiple sensors satisfies min(H_obj_m–f(H_obj_n)) and n_V_obj*H_obj=0, where f(H_obj_n)) represents the value of the mask area projected from sensor n to sensor m, and n_V_obj represents the normal vector of the plane perpendicular to the target attribute V_obj.

[0041] For example, to build a matching relationship between multiple sensors Si:

[0042] f(x_mn,y_mn,z_mn,yaw_mn,pitch_mn,roll_mn)

[0043] Among them, x_mn, y_mn, z_mn, yaw_mn, pitch_mn, roll_mn are the external parameters to be solved between multiple sensors (m, n).

[0044] Set f(x_mn, y_mn, z_mn, yaw_mn, pitch_mn, roll_mn) to satisfy the constraints of min(H_obj_m–f(H_obj_n)) and n_V_obj*H_obj=0, where f(H_obj_n)) represents the value of the n sensor projected onto the m sensor mask area. n_V_obj represents the normal vector of the V_obj plane.

[0045] S103: When the number of targets in the scene is greater than a predetermined number, optimizing the external parameters between the multiple sensors according to the constraint conditions until an optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

[0046] In one embodiment of the present application, when the number of targets in the scene is greater than a predetermined number, the external parameters between the multiple sensors are optimized according to the constraint conditions until the optimization exit conditions are reached, thereby obtaining the optimized external parameters between the multiple sensors, including: determining whether the number of targets in the scene is greater than a predetermined number, wherein the targets in the scene include horizontal signs and vertical objects; if so, optimizing the external parameters between the multiple sensors according to the preset conditions and the matching relationship between the multiple sensors; determining whether the number of optimizations reaches a predetermined number; if so, obtaining the optimized external parameters between the multiple sensors.

[0047] For example, if the predetermined number is 8 and the predetermined times is 3, then when the number of targets in the scene meets the set conditions (the number of targets>8, and the target attributes include horizontal and vertical categories), the external parameters (R_i, t_i) between the multiple sensors are optimized according to the above matching relationship and constraints. The optimization cycle is performed, and when it is greater than 3 times, the optimized multi-sensor external parameters (R, t) are obtained.

[0048] According to the joint calibration method of multiple sensors in the embodiment of the present application, the horizontal markings and vertical object masks in the images collected by multiple sensors of a vehicle parked in a scene with ground marking lines and vertical objects are first extracted, and then the matching relationship between the multiple sensors is constructed, and the constraints of the matching relationship between the multiple sensors are determined. Finally, according to the constraints and the number of optimizations, the external parameters between the multiple sensors are optimized. In this way, the joint calibration of multiple sensors in targetless natural scenes is realized, which effectively reduces the dependence on the site and the target, realizes the integration of calibration and function, and has the advantage of simple and convenient calibration.

[0049] Figure 2FIG. 1 is a structural block diagram of a multi-sensor joint calibration system according to an embodiment of the present application. Figure 2 As shown, a multi-sensor joint calibration system according to an embodiment of the present application includes: an extraction module 210, a construction module 220 and a calibration module 230, wherein:

[0050] An extraction module 210, for extracting masks of horizontal markings and vertical objects in images collected by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground markings and vertical objects;

[0051] A construction module 220, configured to construct a matching relationship between a plurality of sensors and determine a constraint condition of the matching relationship between the plurality of sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors;

[0052] The calibration module 230 is used to optimize the external parameters between the multiple sensors according to the constraint conditions when the number of targets in the scene is greater than a predetermined number, until the optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

[0053] According to the multi-sensor joint calibration system of the embodiment of the present application, the horizontal markings and vertical object masks in the images collected by multiple sensors of the vehicle parked in the scene with ground marking lines and vertical objects are first extracted, and then the matching relationship between the multiple sensors is constructed, and the constraints of the matching relationship between the multiple sensors are determined. Finally, according to the constraints and the number of optimizations, the external parameters between the multiple sensors are optimized. In this way, the joint calibration of multiple sensors in targetless natural scenes is realized, which effectively reduces the dependence on the site and the target, realizes the integration of calibration and function, and has the advantages of simple and convenient calibration.

[0054] For the specific definition of the multi-sensor joint calibration system, please refer to the definition of the multi-sensor joint calibration method mentioned above, which will not be repeated here. The various modules of the above-mentioned multi-sensor joint calibration system can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0055] In one embodiment, a vehicle is provided, comprising: a joint calibration system for multiple sensors according to any one of the above embodiments. The vehicle first extracts the masks of horizontal marks and vertical objects in images collected by multiple sensors of a vehicle parked in a scene with ground marking lines and vertical objects, then constructs a matching relationship between multiple sensors, and determines the constraints of the matching relationship between multiple sensors. Finally, according to the constraints and the number of optimizations, the external parameters between the multiple sensors are optimized. Thus, the joint calibration of multiple sensors in targetless natural scenes is realized, which effectively reduces the dependence on sites and targets, realizes the integration of calibration and function, and has the advantage of simple and convenient calibration.

[0056] In addition, other structures and functions of the vehicle according to the embodiment of the present application are known to ordinary technicians in the field and will not be elaborated here.

[0057] In one embodiment, a computer device is provided. Figure 3 This is a block diagram of the computer device provided in the embodiment of the present application, refer to Figure 3 The computer device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the aforementioned multi-sensor joint calibration method embodiment when executing the computer program. For example, the following is performed: extracting masks of horizontal markings and vertical objects in images collected by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground marking lines and vertical objects;

[0058] Constructing a matching relationship between a plurality of sensors and determining constraint conditions of the matching relationship between the plurality of sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors;

[0059] When the number of targets in the scene is greater than a predetermined number, the external parameters between the multiple sensors are optimized according to the constraint conditions until an optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

[0060] The present application also provides a computer-readable storage medium, which stores a computer program. When the processor executes the computer program, the aforementioned multi-sensor joint calibration method embodiment is implemented. For example, the following steps are performed: extracting masks of horizontal markings and vertical objects in images collected by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground marking lines and vertical objects;

[0061] Constructing a matching relationship between a plurality of sensors and determining constraint conditions of the matching relationship between the plurality of sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors;

[0062] When the number of targets in the scene is greater than a predetermined number, the external parameters between the multiple sensors are optimized according to the constraint conditions until an optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

[0063] The present application embodiment provides a computer program product, which includes instructions. When the instructions are executed, the method described in the embodiment of the present application is executed. For example, it can be executed Figure 1 The steps of the multi-sensor joint calibration method shown, for example, are performed: extracting masks of horizontal markings and vertical objects in images captured by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground marking lines and vertical objects;

[0064] Constructing a matching relationship between a plurality of sensors and determining constraint conditions of the matching relationship between the plurality of sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors;

[0065] When the number of targets in the scene is greater than a predetermined number, the external parameters between the multiple sensors are optimized according to the constraint conditions until an optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

[0066] Those of ordinary skill in the art can understand that all or part of the processes in the methods for implementing the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0067] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0068] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A multi-sensor joint calibration method, characterized in that: include: Extracting masks of horizontal markings and vertical objects in images acquired by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground markings and vertical objects; Constructing a matching relationship between a plurality of sensors and determining constraint conditions of the matching relationship between the plurality of sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors; When the number of targets in the scene is greater than a predetermined number, the external parameters between the multiple sensors are optimized according to the constraint conditions until an optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

2. The multi-sensor joint calibration method according to claim 1, characterized in that: The horizontal mark and the vertical object mask are represented as a horizontal object attribute H_obj and a vertical object attribute V_obj, respectively.

3. The multi-sensor joint calibration method according to claim 2, characterized in that: The constraints include: The matching relationship between the multiple sensors satisfies min(H_obj_m-f(H_obj_n)) and n_V_obj*H_obj=0, where f(H_obj_n)) represents the value of the mask area projected from sensor n to sensor m, and n_V_obj represents the normal vector of the plane perpendicular to the target attribute V_obj.

4. The multi-sensor joint calibration method according to claim 3, characterized in that: When the number of targets in the scene is greater than a predetermined number, optimizing the external parameters between the multiple sensors according to the constraint conditions until the optimization exit condition is reached, and obtaining the optimized external parameters between the multiple sensors, including: Determining whether the number of targets in the scene is greater than a predetermined number, wherein the targets in the scene include horizontal markers and vertical objects; If yes, optimizing the external parameters between the multiple sensors according to the constraint condition and the matching relationship between the multiple sensors; Determine whether the number of optimizations reaches the predetermined number; If yes, the optimized external parameters between the multiple sensors are obtained.

5. The multi-sensor joint calibration method according to any one of claims 1 to 4, characterized in that: The ground marking lines include lane lines, and the vertical objects include electric poles.

6. A multi-sensor joint calibration system, characterized in that: include: An extraction module, for extracting masks of horizontal markings and vertical objects in images collected by multiple sensors of a vehicle, wherein the vehicle is pre-parked in a scene with ground marking lines and vertical objects; A construction module, used to construct a matching relationship between multiple sensors and determine the constraint conditions of the matching relationship between the multiple sensors, wherein the matching relationship includes an external parameter with a solution between any two sensors; The calibration module is used to optimize the external parameters between the multiple sensors according to the constraint conditions when the number of targets in the scene is greater than a predetermined number, until the optimization exit condition is reached, thereby obtaining the optimized external parameters between the multiple sensors.

7. A vehicle, characterized in that: include: The multi-sensor joint calibration system according to claim 6.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the multi-sensor joint calibration method according to any one of claims 1 to 5 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-sensor joint calibration method according to any one of claims 1 to 5 is implemented.

10. A computer program product having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-sensor joint calibration method according to any one of claims 1 to 5 is implemented.

Images