Sensor automatic calibration method, electronic device and storage medium

Abstract

An embodiment of the present invention provides a sensor automatic calibration method, an electronic device, and a storage medium. The method includes: acquiring laser point cloud data collected by a laser device and image data collected by a camera; identifying scene targets related to the driving state in the laser point cloud data according to the driving state of the mobile device, and obtaining a first coordinate of the scene target in the laser point cloud coordinate system; identifying scene targets related to the driving state in the image data according to the driving state of the mobile device, and obtaining a second coordinate of the scene target in the image coordinate system; reprojecting the first coordinate into a third coordinate in the image coordinate system; matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result; and calibrating the external parameters between the laser device and the camera based on the matching result. The embodiment of the present invention does not rely on traditional calibration scenarios and can achieve more accurate automatic calibration of external parameters between the laser device and the camera.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving, and in particular to a sensor automatic calibration method, an electronic device, and a storage medium. Background Art

[0002] Autonomous driving typically consists of a perception system, a decision-making system, an execution system, and a communication system. The vehicle collects data, processes and outputs the data, and ultimately makes decisions and controls. Due to the high safety requirements of autonomous driving, sensors must be able to more fully collect information about the surrounding environment to make reliable inferences.

[0003] However, no single sensor can achieve comprehensive perception, including distance and object information. Therefore, to obtain more comprehensive features, reliable sensor perception integration solutions include lidar and cameras. Lidar can better perceive the location information of surrounding objects with high accuracy, but lidar data is relatively sparse and cannot provide clear type characteristics for some obstacles encountered during autonomous driving. At the same time, it may fail in rain, snow, light dust, etc. Although such applications can be optimized through multiple echo solutions, the backend still needs to filter out abnormal targets. Cameras can better recognize semantic information in the scene and identify traffic obstacles, but segmentation is prone to failure in some scenarios. For example, objects with unclear boundaries may cause abnormalities in the overall detection.

[0004] Therefore, combining LiDAR and camera feature fusion information can better assist autonomous driving perception operations. Before fusion is used, parameters must be calibrated to ensure that there is no significant deviation between the corresponding target position in the laser and the camera. This allows for better fusion perception of the target.

[0005] In the process of implementing the present invention, the inventors discovered that there are at least the following problems in the related art:

[0006] The integration of cameras and lidar is fixed. During driving, bumps, sudden braking, and other factors can cause the relative positions of these fixed cameras and lidars to shift. At this point, the calibration parameters used during integration will no longer apply, necessitating recalibration of the extrinsic parameters between the camera and lidar. However, finding a suitable calibration environment for these fixed sensors is difficult, and specific scene parameters require specific target objects, making recalibration challenging. Therefore, achieving automatic calibration of the extrinsic parameters of cameras and lidars has become a pressing technical challenge. Summary of the Invention

[0007] The technical solution of the present invention solves the technical problem in the prior art that the external parameters of the camera and lidar cannot be automatically calibrated when the relative positions of the camera and lidar change during the driving of an autonomous vehicle. In the first aspect, an embodiment of the present invention provides a sensor automatic calibration method applied to a mobile device, comprising:

[0008] Obtain laser point cloud data collected by laser equipment;

[0009] Get the image data collected by the camera;

[0010] According to the driving state of the mobile device, a scene target related to the driving state in the laser point cloud data is identified to obtain a first coordinate of the scene target in a laser point cloud coordinate system;

[0011] identifying the scene object related to the driving state in the image data according to the driving state of the mobile device, and obtaining a second coordinate of the scene object in the image coordinate system;

[0012] Reprojecting the first coordinate into a third coordinate in the image coordinate system;

[0013] matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result;

[0014] Based on the matching result, extrinsic parameters between the laser device and the camera are calibrated.

[0015] In a second aspect, an embodiment of the present invention provides a sensor automatic calibration device, characterized by comprising:

[0016] Point cloud acquisition module, used to obtain laser point cloud data collected by laser equipment;

[0017] An image acquisition module is used to obtain image data collected by a camera;

[0018] a first coordinate determination module, configured to identify a scene target related to the driving state in the laser point cloud data according to the driving state of the mobile device, and obtain a first coordinate of the scene target in the laser point cloud coordinate system;

[0019] a second coordinate determination module, configured to identify the scene object related to the driving state in the image data according to the driving state of the mobile device, and obtain a second coordinate of the scene object in the image coordinate system;

[0020] A third coordinate determining module, configured to reproject the first coordinate into a third coordinate in an image coordinate system;

[0021] a matching module, configured to match the second coordinate with the third coordinate in the image coordinate system to obtain a matching result;

[0022] A calibration module is used to calibrate the external parameters between the laser device and the camera based on the matching result.

[0023] According to a third aspect, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the steps of the sensor automatic calibration method of any embodiment of the present invention.

[0024] In a fourth aspect, an embodiment of the present invention provides a mobile device, comprising a main body and the electronic device according to any embodiment of the present invention mounted on the main body.

[0025] In a fifth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, characterized in that when the program is executed by a processor, the steps of the sensor automatic calibration method of any embodiment of the present invention are implemented.

[0026] In a sixth aspect, an embodiment of the present invention further provides a computer program product, which, when executed on a computer, enables the computer to execute the sensor automatic calibration method described in any one of the embodiments of the present invention.

[0027] The beneficial effects of the embodiments of the present invention are: the technical solution of the present invention selects the corresponding scene target based on the driving state of the mobile device, matches the laser point cloud coordinates and image coordinates of the scene target, and realizes automatic calibration of the external parameters between the camera and the lidar according to the matching results. This solution does not rely on traditional calibration scenes and can realize more accurate automatic calibration between the laser equipment and the camera. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 This is a flow chart of a sensor automatic calibration method provided by one embodiment of the present invention;

[0030] Figure 2Schematic diagram of point cloud features of a signboard filtered by height and reflectivity in point cloud data of a sensor automatic calibration method provided by one embodiment of the present invention;

[0031] Figure 3 This is a schematic diagram of the effect of detecting road signs in image data of a sensor automatic calibration method provided by one embodiment of the present invention;

[0032] Figure 4 This is a schematic diagram of a traffic sign point cloud and image fusion display of a sensor automatic calibration method provided by an embodiment of the present invention;

[0033] Figure 5 This is a schematic diagram of pedestrian and vehicle projection matching between images and point clouds in a sensor automatic calibration method provided by one embodiment of the present invention;

[0034] Figure 6 This is a laser point cloud and image fusion calibration flow chart of a sensor automatic calibration method provided by one embodiment of the present invention;

[0035] Figure 7 This is a schematic structural diagram of a sensor automatic calibration device provided by one embodiment of the present invention;

[0036] Figure 8 A schematic structural diagram of an embodiment of an electronic device for automatic sensor calibration provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0038] Those skilled in the art will appreciate that the embodiments of the present application may be implemented as a system, apparatus, device, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software.

[0039] For ease of understanding, the technical terms involved in this application are explained below:

[0040] The "mobile device" referred to in this application can be any device with mobile capabilities, including but not limited to automobiles, ships, submarines, airplanes, aircraft and other equipment. Automobiles include vehicles with six autonomous driving technology levels L0-L5 as established by the Society of Automotive Engineers International (SAE International) or the Chinese national standard "Automotive Driving Automation Classification", hereinafter referred to as autonomous driving vehicles ADV (Auto-Driving Vehicle).

[0041] The "autonomous driving vehicle ADV" referred to in this application may be a vehicle device or a robotic device having the following functions:

[0042] (1) Passenger-carrying function, such as family cars and buses;

[0043] (2) Cargo carrying function, such as ordinary trucks, box trucks, trailer trucks, closed trucks, tank trucks, flatbed trucks, container trucks, dump trucks, trucks with special structures, etc.;

[0044] (3) Tool functions, such as logistics delivery vehicles, automated guided vehicles (AGVs), patrol cars, cranes, hoists, excavators, bulldozers, forklifts, rollers, loaders, off-road engineering vehicles, armored engineering vehicles, sewage treatment vehicles, sanitation vehicles, vacuum trucks, floor scrubbers, sprinkler trucks, sweeping robots, food delivery robots, shopping guide robots, lawn mowers, golf carts, etc.;

[0045] (4) Entertainment functions, such as entertainment vehicles, amusement park self-driving devices, balance vehicles, etc.;

[0046] (5) Special rescue functions, such as fire trucks, ambulances, power repair trucks, engineering rescue trucks, etc.

[0047] like Figure 1 FIG. 1 is a flow chart of a sensor automatic calibration method provided by an embodiment of the present invention, comprising the following steps:

[0048] S11: Acquire laser point cloud data collected by the laser device;

[0049] S12: Obtain image data collected by the camera;

[0050] S13: Identifying a scene target related to the driving state in the laser point cloud data according to the driving state of the mobile device, and obtaining a first coordinate of the scene target in the laser point cloud coordinate system;

[0051] S14: identifying the scene object related to the driving state in the image data according to the driving state of the mobile device, and obtaining a second coordinate of the scene object in the image coordinate system;

[0052] S15: Reprojecting the first coordinate into a third coordinate in the image coordinate system;

[0053] S16: Match the second coordinate and the third coordinate in the image coordinate system to obtain a matching result;

[0054] S17: Based on the matching result, calibrate the external parameters between the laser device and the camera.

[0055] The laser device of the present application may be, for example, a laser radar.

[0056] In order to improve the universality of the technical solution of this application, the scene targets in the embodiments of the present invention can be set to targets that are easy to collect around the vehicle. For example, dynamic targets include pedestrians, vehicles, etc., and static targets include lane lines, signs, etc.

[0057] For steps S11 and S12, laser point cloud data collected by the laser equipment carried by the vehicle and image data collected by the camera are used.

[0058] In an embodiment of the present invention, a correspondence between the driving state and the corresponding scene target is pre-set. For example, the driving state of the vehicle includes vehicle driving, vehicle starting, vehicle stopping, etc.; the scene targets corresponding to vehicle driving include dynamic targets such as pedestrians and / or vehicles, and the scene targets corresponding to vehicle starting and vehicle stopping include static targets such as lane lines and / or signboards.

[0059] In step S13, scene objects associated with the driving state are identified in the laser point cloud data. In one embodiment, a target detection model can be used to identify scene objects. For example, the target detection model can employ a CNN-segmentation network. The CNN-segmentation network uses semantic segmentation to detect objects, performs regression processing simultaneously with semantic segmentation, and then clusters objects based on the center offset and semantic segmentation results to obtain a single target detection result, thereby obtaining laser point cloud data corresponding to each scene object. The point cloud coordinates of the scene objects can be obtained from the laser point cloud data using a point cloud-based segmentation detection network. For example, the ground is removed from the laser point cloud data to extract foreground point clouds. The foreground point clouds are then clustered according to specific clustering features to obtain specific clustered target features. Target information corresponding to the scene objects is detected based on the clustered target features. The target information includes the scene object's bounding box, center point, and height information. Grid clustering information is obtained based on the clustered target features, and the target category is determined based on the grid features, thereby determining the target and obtaining several features such as the target's height, length, width, orientation, and center point. Map them into their respective coordinate systems to obtain the first coordinates of the scene target in the laser point cloud coordinate system.

[0060] For step S14, scene targets are identified from the image captured by the camera. The scene targets can be identified by a preset target detection model. Optionally, the target detection model can use the visual detection model of yolov5 to obtain the second coordinate of the scene target in the image coordinate system. The target image coordinates of the scene target can be obtained from the image by an image-based detection and segmentation model. The image-based detection and segmentation model is an end-to-end learning network that uses the first N layers to extract features and the last K layers to classify the mentioned features to obtain target information of the scene target. The target information includes a detection box, direction, center point, etc.

[0061] In step S15, since the point cloud coordinates are three-dimensional coordinates in the LiDAR coordinate system and the image coordinates are two-dimensional coordinates in the camera coordinate system, the two cannot be directly compared. The point cloud coordinates must first be projected into the camera coordinate system and converted into image coordinates. That is, the first coordinate in step S13 must be reprojected into the third coordinate in the image coordinate system. After this, the third coordinate converted to the image coordinate system and the second coordinate in the image coordinate system can be compared.

[0062] In step S16, the third coordinate obtained by conversion in the point cloud data is matched with the second coordinate of the image data to obtain a matching result. The matching method of the coordinates in the scene objects related to different driving states is different.

[0063] As an embodiment, when the driving state is a stopped state after starting, the scene object is a predetermined static object, which includes a lane line and / or a sign.

[0064] When the driving state is a stationary state after starting, matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result includes:

[0065] After reprojecting the first coordinates into third coordinates in an image coordinate system, determining a point cloud detection frame of the static object based on the third coordinates;

[0066] Determine a visual detection frame of the static target based on the second coordinates;

[0067] When the intersection-and-union ratio of the point cloud detection frame and the visual detection frame is greater than a set ratio, the relative posture between the laser device and the camera is determined as a matching result based on the reprojection relationship between the laser point cloud coordinate system and the image coordinate system.

[0068] In this embodiment, taking into account the wide range of applications and actual road scenarios, a preliminary parameter estimation is performed before the vehicle starts. Here, the calibration requires collecting the reflectivity of the LiDAR in the scene of the vehicle being stationary after starting.

[0069] In this case, the first choice of scene targets is to consider the ground area and the suspended area. For example, in a real environment, lane lines and signboards can be selected. Under normal circumstances, the reflectivity of road signboards under laser is close to the maximum reflectivity. Therefore, Figure 2 As shown in the figure, the lidar can use reflectivity features to filter out signage. Furthermore, the reflectivity of the lane markings on the ground is significantly different from that of other non-lane marking areas on the road. Therefore, using the lane marking and suspended sign features, the lane markings and suspended signage can be separated from the point cloud.

[0070] In visual images, using a trained sign detection model, such as Figure 3 As shown, the detection frame of the unobstructed sign can be easily obtained. Here, the yolov3 traffic sign detection network model can be used to detect the traffic signs in the upper half of the image for depth segmentation to obtain the sign position in the image. ( Figure 2 and Figure 3 Used as a symbol to indicate Figure 2 and Figure 3 It is obtained by the different reflectivity of the laser, and the real image is distinguished based on the difference in grayscale).

[0071] Using the reprojection relationship between the laser point cloud and the sign in the image, the point cloud is transformed into spatial coordinates to roughly obtain the rotation position relationship of the point cloud sign projected onto the image. The matching projection example is as follows: Figure 4 As shown in the figure, the signboards and lane lines in the image and point cloud are extracted to achieve preliminary matching and achieve the purpose of parameter initialization iteration.

[0072] Specifically, the projection of the sign segmented from the point cloud is transformed using a projection matrix to obtain the projection of the point cloud on the image. The point cloud detection frame and the visual detection frame calculate the IOU (Intersection over Union) to find the position with the largest overlap ratio. If the cumulative overlap ratio cannot reach the preset ratio threshold P0 (the ratio threshold is set to 75%, for example. Those skilled in the art can also flexibly set the ratio threshold according to actual needs, and this application does not make strict restrictions), it can be considered that the preliminary alignment effect cannot meet the requirements, and no parameter update or fine-tuning of the parameters will be performed.

[0073] Given the sparsity of point clouds, point cloud rotation features may exist during the detection frame matching process, so it is considered to add ground information for preliminary parameter optimization, such as lane lines on the ground. In the display of the point cloud, the lane lines on the ground extract background points, that is, ground information, and use the reflectivity characteristics in the ground information to extract the lane line part, complete the initialization matching of the laser device and vision, and achieve preliminary calibration of the calibration parameters of the laser device. It can be seen from this implementation that for the stationary state of the vehicle after startup, based on the initialization calibration parameters, a calibration method based on initialization of the laser point cloud reflectivity parameters is implemented. This method is applicable to many current multi-line autonomous driving lidars and has a wider applicability.

[0074] As another embodiment, when the driving state is motion, the scene target is a predetermined dynamic target, including pedestrians and / or vehicles.

[0075] When the driving state is moving, matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result includes:

[0076] After reprojecting the first coordinates into third coordinates in an image coordinate system, performing multi-object model matching on a first number of dynamic objects in the third coordinates and a second number of dynamic objects in the second coordinates;

[0077] Perform multi-point perspective imaging projection on the matching result to determine the relative position between the laser device and the camera as the matching result.

[0078] In this embodiment, as noted in the prior art, after driving for a period of time, the autonomous vehicle inevitably experiences shaking, which can cause slight shifts in the relative positions of the camera and laser. This shift in the relative positions of the camera and laser radar indicates motion, necessitating high-precision calibration of the laser device and camera.

[0079] In this case, the first choice of dynamic targets is more inclined to pedestrians and vehicles, because these are the easiest to obtain while driving. Similarly, the point cloud coordinates are three-dimensional coordinates in the lidar coordinate system, and the image coordinates are two-dimensional coordinates in the camera coordinate system. The two cannot be directly compared. It is still necessary to use reprojection to project the first coordinate in the point cloud coordinate system to the third coordinate in the image coordinate system. When the difference between the converted third coordinate (converted point cloud coordinate) and the second coordinate (image coordinate) is greater than or equal to the set threshold, it means that the relative position of the lidar and the camera has changed.

[0080] Matching is performed based on the targets obtained by image detection and point cloud segmentation detection. For example, the matching projection image can be obtained by gradient iteration, such as Figure 5 The optimization goal of the overall matching includes the second coordinate (x i ,y i ) and the third coordinate (m k , n k ). For the multi-objective model, according to the detected scene objectives, the following optimization objective function can be obtained:

[0081] min sum((x i -m k ) 2 +(y i -n k ) 2 ), i=0, 1, 2, 3,..., s; k=0, 1, 2, 3,..., s.

[0082] For the above objective function, we can use pnp (pespective-n-point, multi-point perspective imaging) to solve it, and we can get the external parameters between the camera and the lidar (that is, the rotation and translation matrix, the rotation matrix and the translation matrix).

[0083] In step S17, the matching result of the above steps is used to optimize the objective function to calibrate the external parameters between the laser device and the camera.

[0084] In some application scenarios, the vehicle is in a stationary state at the beginning, and then it is in a moving state after starting the vehicle. In order to further improve the accuracy of the external parameter calibration between the laser device and the camera, Figure 6 As shown, when the vehicle is started, the scene target (such as a static target, lane line and / or signboard, etc.) is determined, and the matching static target (such as lane line, signboard) is identified from the image captured by the camera and the laser point cloud data captured by the laser device. According to the matching result between the second coordinate and the third coordinate corresponding to the static target, it is determined whether the external parameter calibration between the camera and the laser device is needed to achieve preliminary parameter estimation (i.e., complete the first calibration); when the vehicle is moving, the scene target is determined to be a dynamic target (such as a pedestrian and / or vehicle), and the matching dynamic target is identified from the image captured by the camera and the laser point cloud data captured by the laser device. According to the matching result between the second coordinate and the third coordinate corresponding to the dynamic target, the external parameter between the camera and the laser device is calibrated to achieve the second calibration. In specific implementation, as Figure 6 As shown, a preliminary image-point cloud matching is performed using point clouds and images of scene targets (high-altitude signs, lane markings) in a stationary and moving state for Scene 1. By rotating the point cloud and performing a forward projection, parameter search and calculation are completed to determine the initial position parameters. Based on the initial parameters, data modeling is performed for Scene 2, which is in a moving state. Scene 2 relies on the ubiquitous presence of pedestrians and cars on the road. Pedestrians and vehicles are extracted through a deep network to obtain center point features. Multi-target point-to-point (PNP) matching is then performed, and the appropriate extrinsic parameters, namely the rotation matrix and offset matrix, are finally iterated to obtain the appropriate extrinsic parameters. The figure shows the Min F matching result obtained using the above objective function, comparing the pedestrians / vehicles detected based on the image with the vehicles and pedestrians obtained from the laser point cloud (after conversion to the image coordinate system). T0 is the preset threshold used for comparison, which is similar to P0 in the above example and will not be repeated here.

[0085] It can be seen from this embodiment that each time the vehicle is started, the calibration parameter test process is started. When the effect of the test parameter meets the expected threshold, that is, the camera target (x i ,y i ) and point cloud target (m i , n i ) is less than the set threshold, it can be considered that the laser equipment and the camera's external inclination angle have not shaken significantly. When the parameters are too large or the position does not meet the requirements, the parameter self-verification is achieved by judging the difference between the target point cloud coordinates and the target image coordinates. For the parameters that are not applicable due to shaking in autonomous driving, the system automatically collects data and updates the closed-loop process of parameters, which optimizes the algorithm calibration idea, broadens the usability, does not rely on traditional calibration scenarios, and can achieve more accurate automatic calibration of external parameters between laser equipment and cameras.

[0086] After laser point cloud calibration and optimization, this method also takes into account the wide range of applications and more special situations in actual roads, and can be adjusted in a targeted manner, for example:

[0087] Since the division of road targets is unclear during vision use, it is not possible to clearly determine whether some scene targets are foreground targets without obtaining corresponding features (such as ground patterns, etc.). Therefore, with the help of laser point cloud data, it is possible to clearly determine whether they are real targets.

[0088] Obstacles encountered during actual operation are difficult to clearly classify into specific, labeled categories based on object detection models. However, in autonomous driving detection, such objects must be protected against obstacles. Therefore, by extracting unspecified point clouds from the laser point cloud, we can confirm the actual presence of scene objects and ensure safety.

[0089] In scenes with weak vision, such as at night, laser point clouds are more reliable. Therefore, reducing the weight of visual targets through images can improve night driving performance to a certain extent.

[0090] Similarly, after image calibration optimization, the laser point cloud will have many abnormal points in abnormal weather conditions such as rainy days. With the help of visual image filtering, these abnormal points can be determined to be noise points in the lane, and filtering can be completed to ensure that abnormal obstacles are removed, avoid emergency braking and other misoperations, and improve the driving experience.

[0091] According to different situations, it can realize the screening of different abnormal targets in laser and vision, provide ideas for elimination and screening, facilitate the extraction of obstacles, and thus improve the optimization effect.

[0092] like Figure 7 FIG2 is a schematic structural diagram of a sensor automatic calibration device provided by an embodiment of the present invention. The system can execute the sensor automatic calibration method described in any of the above embodiments and be configured in a terminal.

[0093] A sensor automatic calibration device 10 provided in this embodiment includes: a point cloud acquisition module 11, an image acquisition module 12, a first coordinate determination module 13, a second coordinate determination module 14, a third coordinate determination module 15, a matching module 16 and a calibration module 17.

[0094] Among them, the point cloud acquisition module 11 is used to acquire laser point cloud data collected by the laser device; the image acquisition module 12 is used to acquire image data collected by the camera; the first coordinate determination module 13 is used to identify the scene target related to the driving state in the laser point cloud data according to the driving state of the mobile device, and obtain the first coordinate of the scene target in the laser point cloud coordinate system; the second coordinate determination module 14 is used to identify the scene target related to the driving state in the image data according to the driving state of the mobile device, and obtain the second coordinate of the scene target in the image coordinate system; the third coordinate determination module 15 is used to reproject the first coordinate into the third coordinate in the image coordinate system; the matching module 16 is used to match the second coordinate with the third coordinate in the image coordinate system to obtain a matching result; the calibration module 17 is used to calibrate the external parameters between the laser device and the camera based on the matching result.

[0095] Preferably, when the driving state is a stationary state after starting, the scene target is a predetermined static target.

[0096] Preferably, when the driving state is a stationary state after starting, the matching module 16 matches the second coordinate with the third coordinate in the image coordinate system, and obtains a matching result including: after reprojecting the first coordinate into the third coordinate in the image coordinate system, determining the point cloud detection frame of the static target based on the third coordinate; determining the visual detection frame of the static target based on the second coordinate; when the intersection and union ratio of the point cloud detection frame and the visual detection frame is greater than a set ratio, determining the relative posture between the laser device and the camera based on the reprojection relationship between the laser point cloud coordinate system and the image coordinate system as a matching result.

[0097] Preferably, the static target includes lane markings and / or signboards.

[0098] Preferably, when the driving state is motion, the scene target is a predetermined dynamic target.

[0099] Preferably, when the driving state is moving, the matching module 16 matches the second coordinate with the third coordinate in the image coordinate system, and obtains a matching result including: after reprojecting the first coordinate into the third coordinate in the image coordinate system, matching a first number of dynamic targets in the third coordinate with a second number of dynamic targets in the second coordinate in a multi-target model; performing multi-point perspective imaging projection on the matching result to determine the relative posture between the laser device and the camera as the matching result.

[0100] Preferably, the dynamic targets include pedestrians and / or vehicles.

[0101] Preferably, the laser device and the camera are triggered to collect laser point cloud data and image data in response to activation of the mobile device.

[0102] An embodiment of the present invention further provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions can execute the sensor automatic calibration method in any of the above method embodiments;

[0103] As an embodiment, the non-volatile computer storage medium of the present invention stores computer-executable instructions, and the computer-executable instructions are configured as follows:

[0104] Obtain laser point cloud data collected by laser equipment;

[0105] Get the image data collected by the camera;

[0106] According to the driving state of the mobile device, a scene target related to the driving state in the laser point cloud data is identified to obtain a first coordinate of the scene target in a laser point cloud coordinate system;

[0107] identifying the scene object related to the driving state in the image data according to the driving state of the mobile device, and obtaining a second coordinate of the scene object in the image coordinate system;

[0108] Reprojecting the first coordinate into a third coordinate in the image coordinate system;

[0109] matching the second coordinate and the third coordinate in the image coordinate system to obtain a matching result;

[0110] Based on the matching result, extrinsic parameters between the laser device and the camera are calibrated.

[0111] A non-volatile computer-readable storage medium can be used to store non-volatile software programs, non-volatile computer executable programs, and modules, such as the program instructions / modules corresponding to the methods described in the embodiments of the present invention. One or more program instructions stored in the non-volatile computer-readable storage medium, when executed by a processor, perform the automatic sensor calibration method described in any of the above method embodiments.

[0112] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform a sensor automatic calibration method.

[0113] In some embodiments, the present invention further provides a mobile device comprising a main body and an electronic device according to any of the preceding embodiments mounted on the main body. The mobile device may be an unmanned vehicle, such as an unmanned sweeper, unmanned floor scrubber, unmanned logistics vehicle, unmanned passenger vehicle, unmanned sanitation vehicle, unmanned minibus / bus, truck, mining vehicle, etc., or a robot.

[0114] In some embodiments, the present invention further provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the sensor automatic calibration method described in any one of the embodiments of the present invention.

[0115] Figure 8 FIG. 1 is a schematic diagram of the hardware structure of an electronic device for an automatic sensor calibration method according to another embodiment of the present invention. Figure 8 As shown, the device includes:

[0116] One or more processors 810 and memory 820, Figure 8 A processor 810 is used as an example. The device for the automatic sensor calibration method may further include: an input device 830 and an output device 840.

[0117] The processor 810, the memory 820, the input device 830 and the output device 840 may be connected via a bus or other means. Figure 8 The bus connection is taken as an example.

[0118] Memory 820, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the automatic sensor calibration method in the embodiments of the present application. Processor 810 executes the non-volatile software programs, instructions, and modules stored in memory 820 to execute various server functional applications and data processing, thereby implementing the automatic sensor calibration method in the aforementioned method embodiment.

[0119] The memory 820 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data, etc. In addition, the memory 820 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 820 may optionally include a memory remotely located relative to the processor 810, and these remote memories may be connected to the mobile device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The input device 830 can receive input digital or character information. The output device 840 can include a display device such as a display screen.

[0121] The one or more modules are stored in the memory 820 and, when executed by the one or more processors 810 , perform the sensor automatic calibration method in any of the above method embodiments.

[0122] The above-mentioned product can execute the method provided in the embodiment of this application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of this application.

[0123] The non-volatile computer-readable storage medium may include a program storage area and a data storage area, wherein the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created based on the use of the device, etc. In addition, the non-volatile computer-readable storage medium may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some embodiments, the non-volatile computer-readable storage medium may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] An embodiment of the present invention also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the steps of the sensor automatic calibration method of any embodiment of the present invention.

[0125] The electronic devices of the embodiments of the present application exist in various forms, including but not limited to:

[0126] (1) Mobile communication devices: These devices are characterized by their mobile communication capabilities and their primary purpose is to provide voice and data communications. These terminals include smartphones, multimedia phones, feature phones, and low-end phones.

[0127] (2) Ultra-mobile personal computer devices: These devices fall under the category of personal computers and have computing and processing capabilities, and generally also have mobile Internet access. These terminals include PDAs, MIDs, and UMPC devices, such as tablet computers.

[0128] (3) Portable entertainment devices: These devices can display and play multimedia content. They include audio and video players, handheld game consoles, e-books, smart toys, and portable car navigation devices.

[0129] (4) Other mobile devices with data processing capabilities.

[0130] In this document, the terms "include" and "comprising" include not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article or apparatus. In the absence of further limitations, the elements defined by the phrase "include..." do not exclude the presence of other identical elements in the process, method, article or apparatus that includes the elements.

[0131] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0132] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0133] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A sensor automatic calibration method, applied to a mobile device, wherein the sensor includes a laser device and a camera, the method comprising: First calibration with the vehicle at rest: Obtain laser point cloud data collected by laser equipment; Get the image data collected by the camera; Identifying lane lines and signboards in the laser point cloud data by using the reflectivity of the laser radar, and obtaining first coordinates of the lane lines and signboards in the laser point cloud coordinate system; Identify the lane line and sign in the image data to obtain second coordinates of the lane line and sign in an image coordinate system; Reprojecting the first coordinate into a third coordinate in the image coordinate system; Performing preliminary matching of lane lines and signboards on the second coordinate and the third coordinate in the image coordinate system to complete initialization parameter estimation for the first calibration; The second calibration of the vehicle in motion based on the initialization parameters: Obtain laser point cloud data collected by laser equipment; Get the image data collected by the camera; Detecting pedestrians and vehicles in the laser point cloud data based on point cloud segmentation to obtain second-calibrated first coordinates of the pedestrians and vehicles in the laser point cloud coordinate system; Obtaining a second calibration second coordinate of the pedestrian and vehicle in the image coordinate system based on the image detection; Reprojecting the second calibrated first coordinate into a third coordinate in the image coordinate system; The second calibration second coordinate and the third coordinate are subjected to multi-target matching of multi-point perspective imaging in the image coordinate system to obtain external parameters between the laser device and the camera, thereby achieving the second calibration.

2. The method according to claim 1, characterized in that Performing preliminary matching of lane lines and signboards on the second coordinate and the third coordinate in the image coordinate system includes: After reprojecting the first coordinate into a third coordinate in an image coordinate system, determining a point cloud detection frame of the lane line and the sign based on the third coordinate; Determine a visual detection frame of the lane line and sign based on the second coordinate; When the intersection-and-union ratio of the point cloud detection frame and the visual detection frame is greater than a set ratio, the relative posture between the laser device and the camera is determined as a matching result based on the reprojection relationship between the laser point cloud coordinate system and the image coordinate system.

3. The method according to claim 1, characterized in that Performing multi-object matching of multi-point perspective imaging on the second calibrated second coordinate and the third coordinate in the image coordinate system includes: After reprojecting the first coordinates into third coordinates in an image coordinate system, performing multi-object model matching on a first number of pedestrians and vehicles in the third coordinates and a second number of pedestrians and vehicles in the second coordinates; Perform multi-point perspective imaging projection on the matching result to determine the relative position between the laser device and the camera as the matching result.

4. The method according to claim 1, wherein The laser device and the camera are triggered to collect laser point cloud data and image data in response to the activation of the mobile device.

5. A sensor automatic calibration device, characterized in that: include: Point cloud acquisition module, used to obtain laser point cloud data collected by laser equipment; An image acquisition module is used to obtain image data collected by a camera; A first coordinate determination module is configured to, during a first calibration when the vehicle is stationary, identify lane markings and signboards in the laser point cloud data using the reflectivity of the laser radar to obtain first coordinates of the lane markings and signboards in the laser point cloud coordinate system; and, during a second calibration when the vehicle is in motion, detect pedestrians and vehicles in the laser point cloud data based on point cloud segmentation to obtain first coordinates of the pedestrians and vehicles in the laser point cloud coordinate system for the second calibration; a second coordinate determination module, configured to identify the lane lines and signboards in the image data, obtain second coordinates of the lane lines and signboards in the image coordinate system, and obtain second calibrated second coordinates of the pedestrians and vehicles in the image coordinate system based on image detection; A third coordinate determining module, configured to reproject the first coordinate into a third coordinate in an image coordinate system; The matching module is used to perform preliminary matching of the lane lines and signboards of the second coordinates and the third coordinates in the image coordinate system, complete the initialization parameter estimation of the first calibration, perform multi-target matching of the second coordinates and the third coordinates of the second calibration in the image coordinate system using multi-point perspective imaging, obtain the external parameters between the laser device and the camera, and realize the second calibration.

6. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the sensor automatic calibration method according to any one of claims 1 to 4.

7. A mobile device, characterized in that: The electronic device comprises a body and the electronic device according to claim 6 mounted on the body.

8. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the sensor automatic calibration method according to any one of claims 1 to 4 is implemented.

9. A computer program product, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the sensor automatic calibration method according to any one of claims 1 to 4.

Images