Camera and lidar joint calibration method, storage medium and electronic device

Abstract

The application discloses a camera and laser radar joint calibration method and device, medium and electronic equipment, and the method comprises the following steps: acquiring image data containing a calibration target object through a camera, and acquiring point cloud data containing the calibration target object through a laser radar; processing the image data by using a first neural network model to obtain 2D coordinate data true values of the calibration target object; mapping the point cloud data into a two-dimensional depth image based on spherical projection, and performing semantic segmentation on the two-dimensional depth image by using a second neural network model to obtain 3D coordinate data of the calibration target object; constructing a third neural network model, training the third neural network model by using the 3D coordinate data and corresponding 2D coordinate data true values, and obtaining the pose relationship between the camera and the laser radar according to the trained third neural network model. The method can avoid random errors caused by manual intervention, and improve the accuracy and efficiency of calibration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of calibration, in particular to a camera and laser radar joint calibration method, and a storage medium and an electronic device. BACKGROUND

[0002] In the related art, a camera and laser radar joint calibration method based on an L-shaped calibration board is disclosed. The method installs a camera and a laser radar on a device to be calibrated, places an L-shaped calibration board on the ground in the field of view of the camera and the laser radar to be calibrated, and respectively obtains image data and point cloud data containing the L-shaped calibration board. Corner point detection is performed on the obtained image data to obtain the coordinates of the checkerboard corner points on the two planes of the L-shaped calibration board in the pixel coordinate system. Plane segmentation and fitting are performed on the obtained point cloud data to obtain two plane equations of the L-shaped calibration board. Through geometric information, the coordinates of the checkerboard corner points on the two planes of the L-shaped calibration board in the laser radar coordinate system are obtained. Based on the coordinates of the corner points of the L-shaped calibration board in the pixel coordinate system and the coordinates of the corner points of the L-shaped calibration board in the laser radar coordinate system, the pose change of the camera and the laser radar is calculated.

[0003] However, the above method needs to construct a complex spatial mapping relationship and a large number of parameter constraint equations, and has high computational complexity, and there are uncertain and no solution of parameter equation solutions. At the same time, the point cloud target is not easy to extract directly, and there is manual intervention and manual calibration, which is low in efficiency and difficult in repeatability. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, one object of the present application is to propose a camera and laser radar joint calibration method. This method can avoid random errors introduced by manual intervention, and can effectively suppress the disturbance caused by random noise, and improve the calibration accuracy and convergence efficiency.

[0005] A second object of the present application is to propose a computer readable storage medium.

[0006] A third object of the present application is to propose an electronic device.

[0007] To achieve the above object, the first aspect of the present application provides a camera and lidar joint calibration method, comprising the following steps: acquiring image data containing a calibration target object by a camera, and acquiring point cloud data containing the calibration target object by a lidar; processing the image data by using a first neural network model to obtain 2D coordinate data of the calibration target object; mapping the point cloud data into a two-dimensional depth image based on spherical projection, and performing semantic segmentation of the calibration target object on the two-dimensional depth image by using a second neural network model to obtain 3D coordinate data of the calibration target object; constructing a third neural network model, and inputting the 3D coordinate data into the third neural network model to output 2D coordinate data prediction value; training the third neural network model by using the 2D coordinate data and the 2D coordinate data prediction value, and obtaining the pose relationship between the camera and the lidar according to the trained third neural network model.

[0008] In addition, the camera and lidar joint calibration method of the present application can have the following additional technical features:

[0009] According to an embodiment of the present application, the first neural network model and the second neural network model both adopt a convolutional neural network model.

[0010] According to one embodiment of the present application, the processing of the image data by the first neural network model to obtain the 2D coordinate data true value of the calibration target object comprises: A1, downsampling the image data, inputting the sampling result into a network structure composed of a Focus module, a CBL module and a CSP module connected in sequence for feature learning to obtain a corresponding output image; B1, downsampling the output image of step A1, inputting the sampling result into a network structure composed of a CBL module and a CSP module connected in sequence for feature learning to obtain a corresponding output image; C1, downsampling the output image of step B1, inputting the sampling result into a network structure composed of a CBL module, an SPP module, a CSP module and a CBL module connected in sequence for feature learning to obtain a corresponding output image; D1, upsampling the output image of step C1, combining the sampling result with the input image of step C1 to realize image feature channel splicing of the same dimension through a feature fusion Concat module to obtain a corresponding output image; E1, inputting the output image of step D1 into a network structure composed of a CSP module and a CBL module connected in sequence for feature learning to obtain a corresponding output image; F1, upsampling the output image of step E1, combining the sampling result with the input image of step B1 to realize image feature channel splicing of the same dimension through a feature fusion Concat module, and fusing features through a CSP module to obtain a corresponding output image; G1, performing convolution operation on the output image of step F1 to obtain a first predicted prior box; H1, inputting the output image of step F1 into a CBL module for feature learning, combining the input image of step F1 through a feature fusion Concat module to realize image feature channel splicing of the same dimension, and fusing features through a CSP module to obtain a corresponding output image; I1, performing convolution operation on the output image of step H1 to obtain a second predicted prior box; J1, inputting the output image of step H1 into a CBL module for feature learning, combining the input image of step E1 through a feature fusion Concat module to realize image feature channel splicing of the same dimension, and fusing features through a CSP module, and then performing convolution operation to obtain a third predicted prior box; K1, based on the first predicted prior box, the second predicted prior box and the third predicted prior box, calculating to obtain the probability of each category label of the calibration target object, taking the category with the maximum probability as the final output category, and obtaining the 2D coordinate data true value according to the final output category.

[0011] According to one embodiment of the application, the second neural network model is used to calibrate the two-dimensional depth image to obtain the 3D coordinate data of the calibrated target object, comprising: A2, downsampling the two-dimensional depth image, and inputting the sampling result into a network structure composed of a Focus module and a CBL*1 module connected in sequence for feature learning to obtain a corresponding output image; B2, downsampling the output image of step A2, and inputting the sampling result into a network structure composed of a ResUnit*1, a ResUnit*2, and a ResUnit*8 connected in sequence for feature learning to obtain a corresponding output image; C2, downsampling the output image of step B2, and inputting the sampling result into a network structure composed of a ResUnit*8 and a ResUnit*4 connected in sequence for feature learning to obtain a corresponding output image; D2, inputting the output image of step C2 into a CBL*5 module for feature learning to obtain a corresponding output image; E2, performing convolution operation on the output image of step D2 to obtain a fourth predicted prior box; F2, performing convolution operation on the output image of step D2, and upsampling the image after convolution operation, combining the sampling result with the input image of step C2, realizing image feature channel splicing of the same dimension through a feature fusion Concat module, and performing feature learning through a CBL*5 module to obtain a corresponding output image; G2, performing convolution operation on the output image of step F2 to obtain a fifth predicted prior box; H2, performing convolution operation on the output image of step E2, and upsampling the image after convolution operation, combining the sampling result with the input image of step C2, realizing image feature channel splicing of the same dimension through a feature fusion Concat module, and performing feature learning through a CBL*5 module, and then performing convolution operation to obtain a sixth predicted prior box; I2, based on the fourth predicted prior box, the fifth predicted prior box, and the sixth predicted prior box, calculating the probability of each class label on the calibrated target object, and taking the class with the largest probability as the final output class; J2, according to the final output class, mapping the two-dimensional depth image back to the point cloud space through K nearest neighbor search matching to obtain the 3D coordinate data.

[0012] According to one embodiment of the present application, the Focus module comprises a plurality of slice units connected in parallel, and a Concat module and a CBL module connected in sequence after the slice units; the CSP module comprises a convolution block, two CBL modules, and a Concat module, the convolution block is connected in parallel with one CBL module, and is connected in sequence with the Concat module and another CBL module after being connected in parallel, the convolution block comprises one CBL module and X ResUnits connected in sequence, the X ResUnits are connected in series, each ResUnit comprises two CBL modules and one ADD module connected in sequence, the CBL modules in the convolution block are connected with the first CBL module and the ADD module in each ResUnit, respectively, X = 1, 2, 4 or 8; the SPP module comprises two CBL modules and a plurality of maximum pooling layers connected in parallel between the two CBL modules; the CBL module comprises a Conv convolution layer, a BN batch normalization layer and a Leaky Relu activation function connected in sequence.

[0013] According to one embodiment of the present application, the third neural network model is a feedforward neural network model using a back propagation algorithm, comprising an input layer, a hidden layer and an output layer, and each layer is connected through network weights and biases.

[0014] According to one embodiment of the present application, the training of the third neural network model by using the 2D coordinate data true value and the 2D coordinate data predicted value comprises: constructing an error function according to the 2D coordinate data predicted value, the corresponding 2D coordinate data true value and the current network parameters of the third neural network model; and updating the network parameters of the third neural network model according to the error function.

[0015] According to one embodiment of the present application, the error function is represented by the following formula:

[0016]

[0017] wherein E is the error function, j = 1, 2, …, N, N is the number of 3D coordinates input into the third neural network model, x j is the jth 3D coordinate data, 0 < λ < 1, λ is a preset parameter, w j is the connection weight of the jth neuron, b j is the bias value of the jth neuron, is the x j neighborhood data fitting 3D space center coordinate data, y j ’ is the x j corresponding 2D coordinate data true value.

[0018] To achieve the above object, the second aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the camera and laser radar joint calibration method of the present application.

[0019] To achieve the above object, the third aspect of the present application provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory, and the computer program is executed by the processor to realize the camera and laser radar joint calibration method of the present application.

[0020] According to the camera and laser radar joint calibration method, the storage medium and the electronic device of the present application, the point cloud target and the image target are automatically extracted and screened, the target object is stably extracted, random errors caused by manual intervention are avoided, the distance feature is introduced into the error function, the weight update step is adaptively optimized, the network convergence is avoided to be too slow, the distance feature can effectively suppress the disturbance of random noise on the network weight update, and the calibration precision and the convergence efficiency are improved.

[0021] Additional aspects and advantages of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flow chart of the camera and laser radar joint calibration method of the present application;

[0023] Figure 2 is a flow chart of step S102 of one embodiment of the present application;

[0024] Figure 3 is a structure schematic diagram of the first neural network of one embodiment of the present application;

[0025] Figure 4 is a structure schematic diagram of the Focus module, the CSP module, the SPP module and the CBL module in the first neural network of one embodiment of the present application;

[0026] Figure 5 is a flow chart of obtaining 3D coordinate data in step S103 of one embodiment of the present application;

[0027] Figure 6 is a structure schematic diagram of the second neural network of one embodiment of the present application;

[0028] Figure 7 is a structure schematic diagram of the third neural network of one embodiment of the present application. DETAILED DESCRIPTION

[0029] Embodiments of the present application are described below in detail, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0030] A camera and lidar joint calibration method of embodiments of the present application, and a storage medium and an electronic device are described below with reference to the drawings.

[0031] Figure 1 is a flowchart of a camera and lidar joint calibration method of embodiments of the present application.

[0032] As shown in Figure 1 , the camera and lidar joint calibration method comprises the following steps:

[0033] S101, acquiring image data containing a calibration target object by a camera, and acquiring point cloud data containing the calibration target object by a lidar.

[0034] Wherein, the calibration target object can be any ready-made object, such as traffic signs, trees, animals, sculptures, etc., without human setting.

[0035] Specifically, each point in the point cloud data acquired by the lidar contains three-dimensional coordinate information, which can include the commonly said X, Y, Z three directions coordinates. Wherein, the point cloud data of the calibration target object can be obtained by emitting laser signals by the lidar installed on the vehicle, and then collecting the reflected laser signals; the image data of the target object can be directly acquired by the camera installed on the vehicle.

[0036] S102, processing the image data by using a first neural network model to obtain the 2D coordinate data real value of the calibration target object.

[0037] Specifically, generally a pixel can be represented by its horizontal and vertical coordinates, and by using the first neural network model to process the image data containing the calibration target object acquired in S101, the position information of the calibration target object in pixel coordinates can be obtained, i.e. the data real value of the calibration target object in 2D pixel coordinates is obtained.

[0038] Preferably, the first neural network model can adopt a convolutional neural network model.

[0039] S103, mapping the point cloud data into a two-dimensional depth image based on spherical projection, and using a second neural network model to perform semantic segmentation of the calibration target object on the two-dimensional depth image to obtain the 3D coordinate data of the calibration target object.

[0040] Specifically, point cloud data labeling is more difficult than two-dimensional image labeling. Therefore, for target object detection and identification of point cloud labeling in a laser radar, the point cloud data can be projected into a two-dimensional depth image first, such as projecting the point cloud data into a two-dimensional depth image by using a RangeNet++ model, and then a second neural network model is used to perform semantic segmentation of the labeled target object according to the two-dimensional depth image. The depth image is also referred to as a distance image, which refers to an image in which the distance from an image collector to each point in a scene is taken as a pixel value. The depth image can be used to conveniently solve the description problem in a 3D target.

[0041] Preferably, the second neural network model can adopt a convolutional neural network model.

[0042] S104, a third neural network model is constructed, and 3D coordinate data is input into the third neural network model to output a 2D coordinate data prediction value.

[0043] S105, the third neural network model is trained by using the 2D coordinate data true value and the 2D coordinate data prediction value, and the pose relationship between the camera and the laser radar is obtained according to the trained third neural network model.

[0044] Specifically, the third neural network model is constructed, and the 3D coordinate data of the labeled target object obtained in S103 is input, and the 2D coordinate data true value of the labeled target object obtained in S102 is output, and the third neural network model is trained. The trained third neural network model can obtain a 2D coordinate data prediction value through 3D coordinate data. That is, the trained third neural network model can obtain the position relationship and the attitude relationship between the camera and the laser radar in a specified coordinate system.

[0045] The training of the third neural network model by using the 2D coordinate data true value and the 2D coordinate data prediction value includes: constructing an error function according to the 2D coordinate data prediction value, the corresponding 2D coordinate data true value, and the current network parameters of the third neural network model; and updating the network parameters of the third neural network model according to the error function, so as to achieve the purpose of training the third neural network model.

[0046] Figure 2 is a flowchart of step S102 of one embodiment of the present application.

[0047] As a feasible implementation manner, the first neural network model is used to process the image data to obtain a 2D coordinate data true value of the labeled target object, as shown in Figure 2The main steps include: firstly, data labeling is performed on the acquired image data, then a corresponding data set is trained, the trained data set is subjected to convolutional neural network to detect and locate the calibration target, and the extracted calibration target is subjected to position sorting to obtain 2D coordinate data real values.

[0048] Specifically, the structure of the first neural network model is as shown in Figure 3 Figure 3 The detailed steps of acquiring the 2D coordinate data real values of the calibration target from the image data by using the first neural network model include:

[0049] A1, the image data acquired in step S101 is down-sampled, and the sampling result is input into a network structure composed of a Focus module, a CBL module and a CSP (Cross-Stage-Partial) module connected in sequence for feature learning to obtain a corresponding output image.

[0050] As shown in Figure 4 The Focus module can include a plurality of slide units connected in parallel and a Concat module, a CBL module connected in sequence after the slide units. The CSP module can include a convolution block, two CBL modules and a Concat module, the convolution block and one CBL module are connected in parallel, and the parallel connection is connected in sequence with the Concat module and the other CBL module, wherein the convolution block can further include one CBL module and X ResUnits connected in sequence, the X ResUnits are connected in series, each ResUnit can include two CBL modules and one ADD module connected in sequence, the CBL modules in the convolution block are connected with the first CBL module and the ADD module in each ResUnit, respectively, and X can be 1, 2, 4 or 8. The CBL module can include a Conv convolution layer, a BN batch normalization layer and a Leaky Relu activation function connected in sequence.

[0051] Specifically, down-sampling can be performed from both the width and height directions, the sampling step length of the width and height can be 2, and the down-sampled image is input into a network structure composed of a Focus module, a CBL module and a CSP (Cross-Stage-Partial) module connected in sequence, which can eliminate the memory pressure caused by repeated calculation, and the hierarchical design of feature fusion can balance the interlayer calculation.

[0052] B1, the output image of step A1 is down-sampled, and the sampling result is input into a network structure composed of a CBL module and a CSP module connected in sequence for feature learning to obtain a corresponding output image.

[0053] ​C1, down-sampling the output image of step B1, inputting the sampling result into a network structure composed of sequentially connected CBL modules, SPP (Spatial Pyramid Pooling) modules, CSP modules, and CBL modules for feature learning, and obtaining a corresponding output image.

[0054] The SPP module can include two CBL modules and a plurality of max-pooling layers connected in parallel between the two CBL modules. By performing spatial pyramid pooling (SPP), the network can avoid repeated feature extraction and improve the target expression ability after fusion of different scale feature maps.

[0055] D1, up-sampling the output image of step C1, combining the sampling result with the input image of step C1, and realizing image feature channel splicing of the same dimension through a feature fusion Concat module to obtain a corresponding output image.

[0056] The feature fusion Concat module realizes image feature channel splicing of the same dimension, which can restore the details lost by down-sampling.

[0057] E1, inputting the output image of step D1 into a network structure composed of sequentially connected CSP modules and CBL modules for feature learning, and obtaining a corresponding output image.

[0058] F1, up-sampling the output image of step E1, combining the sampling result with the input image of step B1, realizing image feature channel splicing of the same dimension through a feature fusion Concat module, and fusing features through a CSP module to obtain a corresponding output image.

[0059] G1, performing convolution operation on the output image of step F1 to obtain a first predicted prior box.

[0060] H1, inputting the output image of step F1 into a CBL module for feature learning, combining the input image of step F1, realizing image feature channel splicing of the same dimension through a feature fusion Concat module, and fusing features through a CSP module to obtain a corresponding output image.

[0061] I1, performing convolution operation on the output image of step H1 to obtain a second predicted prior box.

[0062] J1, inputting the output image of step H1 into a CBL module for feature learning, combining the input image of step E1, realizing image feature channel splicing of the same dimension through a feature fusion Concat module, fusing features through a CSP module, and then performing convolution operation to obtain a third predicted prior box.

[0063] K1, based on the first predicted prior box, the second predicted prior box and the third predicted prior box, the probability of each category label of the calibration target is calculated, the category with the maximum probability is taken as the final output category, and the 2D coordinate data true value is obtained according to the final output category.

[0064] Figure 5 is a flow chart for acquiring 3D coordinate data in step S103 of an embodiment of the application.

[0065] As a feasible implementation, the second neural network model is used for semantic segmentation of the two-dimensional depth image to obtain the 3D coordinate data of the calibration target, see Figure 5 , the main steps include: the laser radar maps the acquired 3D point cloud data to a depth image through spherical projection, the depth image is subjected to semantic segmentation by the second neural network, class statistics are performed, the calibration target is extracted, the extracted calibration target is subjected to depth query and distance sorting to obtain the 3D point cloud information of the calibration target.

[0066] Specifically, the structure of the second neural network is as shown in Figure 6 , see Figure 6 , the detailed steps of acquiring the 3D coordinate data of the calibration target from the two-dimensional depth image by using the second neural network model include:

[0067] A2, the two-dimensional depth image is down-sampled, and the sampling result is input into a network structure composed of a Focus module, a CBL*1 module connected in sequence for feature learning to obtain a corresponding output image.

[0068] Among them, the two-dimensional depth image can be down-sampled from the width and height directions.

[0069] B2, the output image of step A2 is down-sampled, and the sampling result is input into a network structure composed of a ResUnit (Rectified Linear Unit, activation function) *1, a ResUnit*2 and a ResUnit*8 connected in sequence for feature learning to obtain a corresponding output image.

[0070] C2, the output image of step B2 is down-sampled, and the sampling result is input into a network structure composed of a ResUnit*8 and a ResUnit*4 connected in sequence for feature learning to obtain a corresponding output image.

[0071] D2, the output image of step C2 is input into a CBL*5 module for feature learning to obtain a corresponding output image.

[0072] E2, a convolution operation is performed on the output image of step D2 to obtain a fourth predicted prior box.

[0073] F2, performing convolution operation on the output image of step D2, and performing up-sampling on the image after the convolution operation, combining the sampling result with the input image of step S304, realizing image feature channel splicing of the same dimension through a feature fusion Concat module, and performing feature learning through a CBL*5 module to obtain a corresponding output image.

[0074] G2, performing convolution operation on the output image of step F2 to obtain a fifth predicted prior box.

[0075] H2, performing convolution operation on the output image of step E2, and performing up-sampling on the image after the convolution operation, combining the sampling result with the input image of step C2, realizing image feature channel splicing of the same dimension through a feature fusion Concat module, and performing feature learning through a CBL*5 module, and then performing convolution operation to obtain a sixth predicted prior box.

[0076] I2, based on the fourth predicted prior box, the fifth predicted prior box and the sixth predicted prior box, calculating the probability of each type of label on the calibration target object to obtain the final output class.

[0077] J2, according to the final output class, mapping the two-dimensional depth image back to the point cloud space through K-nearest neighbor search matching to obtain 3D coordinate data.

[0078] Figure 7 is a structural schematic diagram of a third neural network of an embodiment of the present application.

[0079] As shown in Figure 7 , the third neural network model can be a shallow neural network model, such as a feedforward neural network model using a backpropagation algorithm, including an input layer, a hidden layer and an output layer, connected through network weights and biases, and a non-linear continuous function is simulated by setting different (commonly three layers) network layers to establish the mapping relationship between the point cloud coordinates and the pixel coordinates.

[0080] In this example, the depth distance feature can be introduced into the error function to adaptively optimize the weight update step size and improve the network convergence efficiency, and the error function can be represented by the following formula:

[0081]

[0082] wherein E is an error function, j is a positive integer, j = 1, 2, …, N, N is the number of 3D coordinates input into the third neural network model, x j is the jth 3D coordinate data, 0 < λ < 1, λ is a preset parameter, w j is the connection weight of the jth neuron, and bj bias value of the jth neuron, x j 3D space center coordinate data of neighborhood data fitting, y j x j corresponding 2D coordinate data true value.

[0083] To sum up, the camera and laser radar joint calibration method of the embodiment of the application does not need to construct complex spatial constraint relationship and large parameter constraint equation, the execution process is more simple, and compared with the uncertainty and no solution of parameter equation solution, the scheme based on network optimization can obtain stable equation solution; through automatic extraction and screening of point cloud targets and image targets, random errors introduced by manual intervention are avoided, distance features are introduced into the error function, convergence of the network is avoided to be slow, and disturbance caused by random noise is effectively inhibited, so that the calibration precision and convergence efficiency are improved.

[0084] Further, the application provides a computer readable storage medium.

[0085] In the embodiment of the application, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the camera and laser radar joint calibration method.

[0086] Further, the application provides an electronic device.

[0087] In the embodiment of the application, the electronic device includes a memory, a processor and a computer program stored in the memory, and the computer program is executed by the processor to realize the camera and laser radar joint calibration method.

[0088] It is to be appreciated that the above description and the examples that follow are intended to be illustrative only and that changes can be made to the description and examples without departing from the scope of the application. Note that in the description and drawings of the embodiments, changes in the order of

[0089] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the described embodiments various steps or methods can be implemented in software or firmware which are stored in memory and executed by a suitable instruction execution system using any suitable computer readable medium. For example, if implemented in hardware, and in another embodiment, any of the following technology, which are known in the art, can be used to implement: a hybrid of the above technologies, discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), or the like.

[0090] In the description of the present application, the terms "one embodiment", "some embodiments", "an example", "specific examples", or "some examples" means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. The illustrative appearances of the above terms in various places in the specification are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0091] In the description of the application, it is to be understood that the orientations or positional relationships indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like are based on the orientations or positional relationships shown in the drawings, and are only for the purpose of facilitating the description of the application and simplifying the description, and are not intended to indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application.

[0092] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can be explicitly or implicitly included at least one of the features. In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified and limited.

[0093] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0094] In the present application, unless otherwise explicitly specified and limited, the first feature "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature "on", "above" and "on" the second feature can be that the first feature is directly above or obliquely above the second feature, or it can only mean that the horizontal height of the first feature is higher than that of the second feature. The first feature "under", "below" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or it can only mean that the horizontal height of the first feature is less than that of the second feature.

[0095] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for camera and lidar joint calibration, characterized in that, The method comprises: acquiring image data containing a calibration target object through a camera and point cloud data containing the calibration target object through a laser radar; processing the image data using a first neural network model to obtain 2D coordinate data true values of the calibration target object; mapping the point cloud data into a two-dimensional depth image based on spherical projection and performing calibration target object semantic segmentation on the two-dimensional depth image using a second neural network model to obtain 3D coordinate data of the calibration target object; constructing a third neural network model and inputting the 3D coordinate data into the third neural network model to output 2D coordinate data predicted values; training the third neural network model using the 2D coordinate data true values and the 2D coordinate data predicted values and obtaining a pose relationship of the camera and the laser radar according to the trained third neural network model; the first neural network model and the second neural network model both adopt a convolutional neural network model; the first neural network model comprises a Focus module, a CBL module, a CSP module and an SPP module; the second neural network model comprises a network structure composed of a Focus module and a CBL*1 module; the third neural network model is a feedforward neural network model adopting a back propagation algorithm and comprises an input layer, a hidden layer and an output layer, and the layers are connected through network weights and biases.

2. The camera and lidar co-calibration method of claim 1, wherein, the processing of the image data using the first neural network model to obtain the 2D coordinate data true values of the calibration target object comprises: A1, performing down-sampling on the image data, inputting the sampling result into a network structure composed of a Focus module, a CBL module and a CSP module connected in sequence for feature learning to obtain a corresponding output image; B1, performing down-sampling on the output image of step A1, inputting the sampling result into a network structure composed of a CBL module and a CSP module connected in sequence for feature learning to obtain a corresponding output image; C1, performing down-sampling on the output image of step B1, inputting the sampling result into a network structure composed of a CBL module, an SPP module, a CSP module and a CBL module connected in sequence for feature learning to obtain a corresponding output image; D1, performing up-sampling on the output image of step C1, combining the sampling result with the input image of step C1, realizing image feature channel splicing of the same dimension through a feature fusion Concat module to obtain a corresponding output image; E1, inputting the output image of step D1 into a network structure composed of a CSP module and a CBL module connected in sequence for feature learning to obtain a corresponding output image; F1, performing up-sampling on the output image of step E1, combining the sampling result with the input image of step B1, realizing image feature channel splicing of the same dimension through a feature fusion Concat module and fusing features through a CSP module to obtain a corresponding output image; G1, performing convolution operation on the output image of step F1 to obtain first predicted prior boxes; H1, input the output image of step F1 into the CBL module for feature learning, combine the input image of step F1, realize the same dimension image feature channel splicing through the feature fusion Concat module, and fuse the features through the CSP module to obtain the corresponding output image; I1, perform convolution operation on the output image of step H1 to obtain a second predicted prior box; J1, input the output image of step H1 into the CBL module for feature learning, combine the input image of step E1, realize the same dimension image feature channel splicing through the feature fusion Concat module, and fuse the features through the CSP module, and then perform convolution operation to obtain a third predicted prior box; K1, based on the first predicted prior box, the second predicted prior box and the third predicted prior box, calculate the probability of each category label of the calibration target object, take the category with the maximum probability as the final output category, and obtain the 2D coordinate data true value according to the final output category.

3. The camera and lidar co-calibration method of claim 2, wherein, The second neural network model is used for semantic segmentation of the two-dimensional depth image to obtain the 3D coordinate data of the calibration target object, comprising: A2, downsample the two-dimensional depth image, and input the sampling result into the network structure composed of Focus module, CBL*1 module connected in sequence for feature learning to obtain the corresponding output image; B2, downsample the output image of step A2, and input the sampling result into the network structure composed of ResUnit*1, ResUnit*2 and ResUnit*8 connected in sequence for feature learning to obtain the corresponding output image; C2, downsample the output image of step B2, and input the sampling result into the network structure composed of ResUnit*8 and ResUnit*4 connected in sequence for feature learning to obtain the corresponding output image; D2, input the output image of step C2 into the CBL*5 module for feature learning to obtain the corresponding output image; E2, perform convolution operation on the output image of step D2 to obtain a fourth predicted prior box; F2, perform convolution operation on the output image of step D2, and upsample the image after convolution operation, combine the sampling result with the input image of step D2, realize the same dimension image feature channel splicing through the feature fusion Concat module, and perform feature learning through the CBL*5 module to obtain the corresponding output image; G2, perform convolution operation on the output image of step F2 to obtain a fifth predicted prior box; H2, perform convolution operation on the output image of step E2, and upsample the image after convolution operation, combine the sampling result with the input image of step C2, realize the same dimension image feature channel splicing through the feature fusion Concat module, and perform feature learning through the CBL*5 module, and then perform convolution operation to obtain a sixth predicted prior box; I2, based on the fourth predicted bounding box, the fifth predicted bounding box and the sixth predicted bounding box, calculate the probability of each category label on the calibration target, and take the category with the maximum probability as the final output category; J2, according to the final output category, map the two-dimensional depth image back to the point cloud space through K nearest neighbor search matching to obtain the 3D coordinate data.

4. The camera and laser radar joint calibration method of claim 3, wherein The Focus module includes a plurality of slice units connected in parallel, and a Concat module, a CBL module connected in sequence after the slice units; The CSP module includes a convolution block, two CBL modules, and a Concat module, the convolution block is connected in parallel with one CBL module, and is connected in sequence with the Concat module and another CBL module after being connected in parallel, the convolution block includes one CBL module and X ResUnits connected in sequence, X ResUnits are connected in series, each ResUnit includes two CBL modules and one ADD module connected in sequence, the CBL modules in the convolution block are connected with the first CBL module and the ADD module in each ResUnit, respectively, X = 1, 2, 4 or 8; The SPP module includes two CBL modules and a plurality of max pooling layers connected in parallel between the two CBL modules; The CBL module includes a Conv convolution layer, a BN batch normalization layer and a Leaky Relu activation function connected in sequence.

5. The camera and lidar co-calibration method of claim 4, wherein, The training of the third neural network model using the 2D coordinate data true value and the 2D coordinate data predicted value includes: According to the 2D coordinate data predicted value and its corresponding 2D coordinate data true value, and the current network parameters of the third neural network model, an error function is constructed; According to the error function, the network parameters of the third neural network model are updated.

6. The camera and lidar co-calibration method of claim 5, wherein, The error function is represented by the following formula: wherein E is the error function, j = 1, 2, …, N, N is the number of 3D coordinates input into the third neural network model, x j is the jth 3D coordinate data, 0 < λ < 1, λ is a preset parameter, w j is the connection weight of the jth neuron, b j is the bias value of the jth neuron, is the x j neighborhood data fitting 3D space center coordinate data, y j ’ is the x j corresponding 2D coordinate data true value.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-6.

Images