Visual sensor calibration method, controller, server and storage medium

Abstract

The embodiment of the present invention discloses a visual sensor calibration method, a controller, a server and a storage medium. The method includes: obtaining a target image collected by the visual sensor to be calibrated and the environmental parameters of the visual sensor to be calibrated; uploading the target image and the environmental parameters to a cloud server to obtain calibration parameters; and calibrating the visual sensor to be calibrated according to the calibration parameters. The embodiment of the present invention uploads the target image collected by the visual sensor to be calibrated and its environmental parameters to a cloud server to obtain calibration parameters, and calibrates the visual sensor to be calibrated according to the calibration parameters, so as to realize the calibration of different cameras of different models through a universal platform, thereby saving calibration costs.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of automatic control technology, and in particular to a visual sensor calibration method, a controller, a server and a storage medium. Background Art

[0002] Internet technology is developing rapidly, and vehicle intelligence has become an important development direction in the automotive field. For intelligent vehicles, sensors provide a channel for vehicles to perceive external information, and play a vital role in the degree of vehicle intelligence. During the production and manufacturing process of vehicles, the correctness of the calibration of visual sensors directly affects the performance of the vehicle.

[0003] The current common visual sensor calibration method mainly uses the visual sensor to perform preliminary calibration on a chessboard grid black and white pattern in different fields of view, and uses internal parameters and distortion factors to correct the preliminary calibration to achieve the calibration of the visual sensor. However, this method needs to be executed by each car. In a production environment, the calibration process takes a lot of time, which seriously affects the production efficiency of the vehicle. Summary of the invention

[0004] The present invention provides a visual sensor calibration method, a controller, a server and a storage medium to reduce the calibration time of the visual sensor and improve the production efficiency of the vehicle.

[0005] In a first aspect, an embodiment of the present invention provides a visual sensor calibration method, which is applied to a controller, and the method includes:

[0006] Acquire a target image collected by the visual sensor to be calibrated and environmental parameters of the visual sensor to be calibrated;

[0007] Uploading the target image and the environmental parameters to a cloud server to obtain calibration parameters;

[0008] The visual sensor to be calibrated is calibrated according to the calibration parameters.

[0009] In a second aspect, an embodiment of the present invention further provides a visual sensor calibration method, which is applied to a cloud server, and the method includes:

[0010] Get the target image and environmental parameters uploaded by the controller;

[0011] Generating calibration parameters corresponding to the target image and the environmental parameters according to a calibration neural network model, wherein the calibration neural network model is generated by training using a threshold number of image sets;

[0012] The calibration parameters are transmitted to the controller so that the controller calibrates the visual sensor to be calibrated according to the calibration parameters.

[0013] In a third aspect, an embodiment of the present invention further provides a controller, the controller comprising:

[0014] one or more processors;

[0015] a memory for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the visual sensor calibration method as described in the embodiment of the present invention.

[0017] In a fourth aspect, an embodiment of the present invention further provides a server, the server comprising:

[0018] one or more processors;

[0019] a memory for storing one or more programs,

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the visual sensor calibration method as described in the embodiment of the present invention.

[0021] In a fifth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the visual sensor calibration method as described in the embodiment of the present invention is implemented.

[0022] The technical solution of the embodiment of the present invention is to obtain the target image collected by the visual sensor to be calibrated and the environmental parameters of the visual sensor to be calibrated, upload the target image and the environmental parameters to the cloud server to obtain the calibration parameters, and calibrate the visual sensor to be calibrated according to the calibration parameters. The above technical solution adopts a method of obtaining the calibration parameters by uploading the target image collected by the visual sensor to be calibrated and its environmental parameters to the cloud server, and calibrating the visual sensor to be calibrated according to the calibration parameters, so as to realize the calibration of different cameras of different models through a universal platform, save the calibration cost, reduce the calibration time of the visual sensor to be calibrated, and improve the production efficiency of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A flowchart of a visual sensor calibration method provided in Embodiment 1 of the present invention;

[0024] Figure 2 A flowchart of a visual sensor calibration method provided in Embodiment 2 of the present invention;

[0025] Figure 3 A schematic diagram of a vehicle front-view camera calibration implementation provided by an embodiment of the present invention;

[0026] Figure 4 A schematic diagram of a process of mapping world coordinates to pixel coordinates provided by an embodiment of the present invention;

[0027] Figure 5 A schematic diagram of coordinate system conversion provided by an embodiment of the present invention;

[0028] Figure 6 A schematic diagram of a calibration implementation of the visual sensor calibration method provided in an embodiment of the present invention;

[0029] Figure 7 A flowchart of a visual sensor calibration method provided in Embodiment 3 of the present invention;

[0030] Figure 8 A flowchart of a visual sensor calibration method provided in Embodiment 4 of the present invention;

[0031] Fig. 9 A schematic diagram of a back propagation neural network training process provided by an embodiment of the present invention;

[0032] Fig.10 A schematic diagram of an image corrected by a back propagation neural network provided in an embodiment of the present invention;

[0033] Fig.11 Another schematic diagram of implementing the calibration method of the visual sensor provided in the embodiment of the present invention;

[0034] Fig.12 is a schematic diagram of the structure of a controller provided by Embodiment 5 of the present invention;

[0035] Fig.13 It is a structural diagram of a server provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention are shown in the accompanying drawings, rather than all structures. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0037] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0038] Embodiment 1

[0039] Figure 1 A flowchart of a visual sensor calibration method provided in Embodiment 1 of the present invention is provided. This embodiment is applicable to the case of calibrating a visual sensor to be calibrated, for example, the case of calibrating a vehicle camera. The method can be executed by a vehicle controller.

[0040] like Figure 1 As shown, a visual sensor calibration method provided by an embodiment of the present invention specifically includes the following steps:

[0041] S110, acquiring a target image collected by the visual sensor to be calibrated and environmental parameters of the visual sensor to be calibrated.

[0042] Among them, the visual sensor can be an instrument that uses optical elements and imaging devices to obtain image information of the external environment. The image resolution is usually used to describe the performance of the visual sensor. The accuracy of the visual sensor is not only related to the resolution, but also to the detection distance of the object being measured. The farther the object is, the worse its absolute position accuracy is. The visual sensor can be a charge coupled device (CCD) camera and a digital camera, etc. The embodiment of the present invention does not specifically limit the selection of the visual sensor.

[0043] It should be noted that the visual sensor mentioned in the embodiment of the present invention is arranged on the vehicle body, and the installation position is not limited. In addition, the embodiment of the present invention does not limit the number of visual sensors. Those skilled in the art can select the number and installation position of the visual sensors according to actual application needs. For example, the visual sensor can be installed on a suitable grid or logo in front of the vehicle body to collect image data in front of the vehicle.

[0044] The target image may be the image information of the calibration target collected by the visual sensor to be calibrated on the vehicle. The calibration target may be a reference target whose spatial position of each feature point is measured in advance, for example, the distance between two points is accurate to 1.0000 cm (accurate to several decimal places). The image information of the calibration target is collected by the visual sensor to be calibrated, and the internal and external parameters of the visual sensor to be calibrated can be calculated based on the relationship between the image coordinates of the feature points in the image information and the real three-dimensional space coordinates. The calibration target may include a single plane checkerboard calibration target, a circular array calibration target, a plane parallel line calibration target, etc.

[0045] The environmental parameters may be vehicle environmental information of a vehicle equipped with the visual sensor to be calibrated, and may include information such as the vehicle position, vehicle angle, vehicle model, and controller model.

[0046] In an embodiment of the present invention, first, the controller may issue a calibration instruction to the visual sensor to be calibrated, and then, the visual sensor to be calibrated may transmit the captured target image and the environmental parameters of the visual sensor to be calibrated to the controller, so that the controller can obtain the target image captured by the visual sensor to be calibrated and the environmental parameters of the visual sensor to be calibrated.

[0047] S120, uploading the target image and environmental parameters to the cloud server to obtain calibration parameters.

[0048] Among them, the cloud server can be a simple, efficient, safe and reliable computing service with elastically scalable processing power. Its management method is simpler and more efficient than that of a physical server. Users can quickly create or release any number of cloud servers without purchasing hardware in advance. It has the characteristics of being highly distributed and highly virtualized, so that network resources can be fully utilized. For example, it can be a cloud server such as Alibaba Cloud, Tencent Cloud, and Huawei Cloud. The embodiment of the present invention does not limit the cloud platform used by the cloud server. The cloud server can provide abundant computing resources to facilitate improving the accuracy of visual sensor calibration.

[0049] The calibration parameters may be parameters related to the visual sensor to be calibrated and position parameters of the visual sensor to be calibrated in the world coordinate system. The visual sensor to be calibrated may correct the image captured by the visual sensor to be calibrated through the calibration parameters.

[0050] In an embodiment of the present invention, the target image acquired by the visual sensor to be calibrated and the environmental parameters of the visual sensor to be calibrated can be imported into the vehicle controller, and the controller can upload the target image and the environmental parameters to the cloud server, and the cloud server executes a calculation process to obtain calibration parameters corresponding to the target image and the environmental parameters, and the cloud server sends the calibration parameters to the controller, so that the controller can obtain the calibration parameters.

[0051] S130: Calibrate the visual sensor to be calibrated according to the calibration parameters.

[0052] In an embodiment of the present invention, after the calibration process is completed through the cloud server, the calibration parameters can be sent to the local vehicle terminal, so that the calibration parameters can be obtained through the cloud server and the visual sensor to be calibrated can be calibrated.

[0053] According to the technical solution of the embodiment of the present invention, the controller obtains the target image collected by the visual sensor to be calibrated and the environmental parameters of the visual sensor to be calibrated, uploads the target image and the environmental parameters to the cloud server to obtain the calibration parameters, and calibrates the visual sensor to be calibrated according to the calibration parameters. The above technical solution, through the controller uploading the target image collected by the visual sensor to be calibrated and the environmental parameters of the visual sensor to be calibrated to the cloud server, and calibrating the visual sensor to be calibrated according to the calibration parameters obtained from the cloud server, reduces the calculation overhead of the controller, improves the accuracy of the calibration of the visual sensor to be calibrated, and uses the calibration parameters of the cloud server to calibrate different visual sensors to be calibrated, which can improve the calibration efficiency of the visual sensor, thereby improving the production efficiency of the vehicle.

[0054] Embodiment 2

[0055] Figure 2 This is a flow chart of a visual sensor calibration method provided in the second embodiment of the present invention. The embodiment of the present invention is optimized based on the above embodiment. Figure 2 As shown, a visual sensor calibration method provided by Embodiment 2 of the present invention specifically includes the following steps:

[0056] S210, controlling the visual sensor to be calibrated on the vehicle to collect image information of the calibration target as a target image, wherein the calibration target includes a black and white checkerboard calibration target, and the vehicle is in a calibration preparation state.

[0057] The black and white chessboard calibration target may be a calibration target made using black and white chessboards, or may be a black and white chessboard calibration target of one or more two-dimensional images formed by shooting the black and white chessboard from multiple perspectives.

[0058] The vehicle being in a calibration preparation state may refer to the preparations that the vehicle needs to make before calibration, which may include the position of the vehicle in front of a fixed calibration target device and the vehicle being at a normal driving speed.

[0059] In an embodiment of the present invention, the visual sensor to be calibrated on the control vehicle can use the collected image information of the black and white chessboard calibration target as the target image, which can reduce the calculation amount of calibration. In addition, the chessboard corner points have obvious characteristics, which can also enhance the accuracy of the calibration result. In addition, when the visual sensor to be calibrated is collecting the target image, the vehicle is in a calibration preparation state, which ensures the accuracy and stability of the target image collected by the visual sensor to be calibrated.

[0060] For example, Figure 3 FIG. 1 is a schematic diagram of a vehicle front view camera calibration implementation provided by an embodiment of the present invention, such as Figure 3 As shown, the front of the vehicle 260 has a front-view camera 270, and the front-view camera 270 faces the black and white checkerboard calibration target 290 of the standard plate 280 of the calibration target in front, so that the front-view camera can capture the calibration target image.

[0061] S220, collecting vehicle body posture information, vehicle model and controller model as environmental parameters.

[0062] Among them, the body posture information can be collected by the body height sensor or the angle of the wheel to collect the body posture, for example, it can collect information on whether the vehicle is on a plane. Among them, the vehicle model can be a number composed of pinyin letters and Arabic numerals assigned to a type of vehicle for vehicle identification. The number composition structure can be divided into three parts: the first section represents the manufacturer, the middle section represents the vehicle type code, main parameters and product serial number, and the last section represents the company's custom code. It is different from the ID card that is accurate to each person. It can have the characteristics of a unified definition of the same type of vehicles, for example, it can be CA7189ATE5T. Among them, the controller model can be a number set for identifying the controller. For example, it can be MAM-100.

[0063] In an embodiment of the present invention, the vehicle's body posture information can be collected through a body height sensor or a wheel angle, and the vehicle model and the controller model can be obtained by accessing a memory storing the vehicle model and the controller model. Thus, the sensor to be calibrated can be calibrated according to the characteristics of the environment in which the sensor to be calibrated is located, thereby ensuring the accuracy of the calibration of each sensor to be calibrated.

[0064] S230, uploading the target image and environmental parameters to the cloud server, so that the cloud server uses the calibration neural network model to generate calibration parameters corresponding to the target image and environmental parameters.

[0065] Among them, the calibration neural network model can be a calibration neural network model trained by a neural network model through a pre-set training set including target images and environmental parameter information. The embodiment of the present invention does not limit the specific neural network model used, for example, it can be a convolutional neural network, an attention mechanism, etc.

[0066] In an embodiment of the present invention, the target image acquired by the visual sensor to be calibrated and the environmental parameters of the visual sensor to be calibrated can be imported into a controller, and the controller can package the target image and environmental parameters and upload them to a cloud server. The cloud server inputs the target image and environmental parameters into a pre-trained calibration neural network model, and after processing, the model outputs calibration parameters corresponding to the target image and environmental parameters.

[0067] S240. Obtain calibration parameters sent by the cloud server, where the calibration parameters include: camera intrinsic parameters, camera extrinsic parameters, and distortion parameters.

[0068] The camera internal parameters may be parameters related to the camera itself, for example, the focal length, pixel size, etc. of the camera.

[0069] The camera extrinsic parameters can be parameters in the world coordinate system, for example, the position and rotation direction of the camera. The camera extrinsic parameters can be divided into rotation matrix and translation torque. The rotation matrix and translation matrix together describe how to transform a point from the world coordinate system to the camera coordinate system. The rotation matrix describes the direction of the coordinate axis of the world coordinate system relative to the camera coordinate axis, and the translation matrix describes the position of the origin of the space in the camera coordinate system.

[0070] Among them, the distortion parameter may come from the distortion of the camera itself, which can be divided into radial distortion and tangential distortion. The radial distortion may be caused by the shape of the lens, and the tangential distortion may be caused by the installation process of the camera itself.

[0071] In an embodiment of the present invention, after the cloud server can generate calibration parameters corresponding to the target image and environmental parameters through a calibration neural network model, the cloud server can send the calibration parameters corresponding to the target image and environmental parameters to the controller via wired or wireless means, and the local controller can receive them so that the local controller can obtain the calibration parameters sent by the cloud server.

[0072] S250: Calibrate the visual sensor to be calibrated according to the calibration parameters.

[0073] In the embodiment of the present invention, the controller may receive and store the calibration parameters sent by the cloud server to implement the calibration of the visual sensor to be calibrated.

[0074] Exemplarily, in an exemplary embodiment, taking the conversion of an image using calibration parameters as an example, Figure 4 A schematic diagram of a process of mapping world coordinates to pixel coordinates provided by an embodiment of the present invention is shown in FIG. Figure 4 As shown, the process of mapping world coordinates to pixel coordinates can be expressed as follows: when the intrinsic and extrinsic parameters and distortion parameters of the camera are determined, the world coordinate system can be converted into the camera coordinate system through the parameters, the camera coordinate system can be converted into the image coordinate system, and finally the image coordinate system can be converted into the pixel coordinate system. Figure 5 The coordinate system conversion diagram provided by the embodiment of the present invention, Ow-YwZwXw is the world coordinate system, describing the camera position; O-YcZcXc is the camera coordinate system, with the optical center as the origin; O1-xy is the image coordinate system, with the optical center as the center point of the image, uv is the pixel coordinate system, with the origin as the upper left corner of the image. The process of mapping world coordinates to pixel coordinates, the conversion matrix of the mapping process can be expressed as follows:

[0075]

[0076] Where f is the focal length of the camera, which is equal to the distance between O1 and O, f = ||O1-O||, and the actual physical size of each pixel along the x-axis is S x , the actual physical size along the y-axis is S y , K 3×3 is the camera internal parameter matrix, R 3×3 and T 3×1 represents external parameter, P 3×4 is the perspective photography matrix, is the point in the world coordinate system; s = Z c is the scale factor.

[0077] The technical solution of the embodiment of the present invention is to upload the target image collected by the visual sensor to be calibrated on the vehicle in the calibration preparation state and the environmental parameters of the vehicle to the cloud server, and then the cloud server generates calibration parameters corresponding to the target image and environmental parameters through the calibration neural network model, and can calibrate the visual sensor to be calibrated through the calibration parameters sent by the cloud server. In this way, the cloud server can generate effective calibration parameters based on accurate target images and environmental parameters, thereby ensuring the accuracy and stability of the calibration of the sensor to be calibrated.

[0078] Further, based on the above embodiment, the vehicle is in a calibration preparation state, including:

[0079] The vehicle has completed four-wheel alignment;

[0080] The vehicle's tire pressure is at standard pressure;

[0081] The vehicle's air suspension is at standard suspension height;

[0082] The vehicle's body running axis is horizontal to the normal line of the standard plate of the calibration target;

[0083] The vehicle's speed reaches the standard speed.

[0084] Among them, the installation of the steering wheel, steering knuckle and front axle of the car has a certain relative position. This installation with a certain relative position can be called steering wheel alignment, or front wheel alignment. Front wheel alignment can include four contents: kingpin castor (angle), kingpin inclination (angle), front wheel camber (angle) and front wheel toe. This is for the two steering front wheels. For the two rear wheels, there can also be a relative position between the installation and the rear axle, which can be called rear wheel alignment. Rear wheel alignment includes wheel camber (angle) and rear wheel toe-in one by one. In this way, front wheel alignment and rear wheel alignment can be collectively called four-wheel alignment.

[0085] Among them, tire pressure can refer to the pressure of the air inside the tire, and the level of tire pressure plays a vital role in the performance and power of the car. The standard tire pressure can be based on the regulations and requirements of international standards, for example, it can be 2.4-2.5bar.

[0086] The air suspension can be mainly composed of a control computer, air intake holes, air exhaust holes, pneumatic front and rear shock absorbers, and an air distributor. It can control the horizontal posture of the vehicle body and adjust the stability system of the vehicle body, so that the sport utility vehicle (SUV) can have both the comfort of a sedan and the passing performance of an off-road vehicle. The standard suspension height can be 2.5 cm.

[0087] The vehicle body driving axis may be the center of the vehicle. The standard vehicle speed may be 0.

[0088] In an embodiment of the present invention, before the visual sensor is calibrated, the vehicle may have completed four-wheel alignment and the tire pressures of all the vehicle's tires may be at standard pressures. If the vehicle is an air suspension model, the air suspension of the vehicle may be at a standard suspension height, the center line of the vehicle's body and the standard version of the calibration target may be at a normal level, and the vehicle reaches a standard speed, so that the vehicle is ready for calibration.

[0089] Figure 6 A schematic diagram of a calibration implementation of the visual sensor calibration method provided by an embodiment of the present invention is shown in FIG. Figure 6As shown, the calibration process begins, the diagnostic instrument reads the vehicle VIN code and ECU serial number, enters the extended mode, unlocks the security access, and sends a calibration request. The diagnostic instrument can send a calibration request to the advanced autonomous driving controller through the central gateway; the advanced autonomous driving controller uploads the calibration request and the target information and vehicle information collected by the camera to the cloud server, the cloud server receives the calibration request and stores the data, and the cloud server sends the processed calibration parameters downlink information so that the controller of the local terminal can write the parameters into the controller, and returns a calibration success instruction, and the calibration process ends. If any link fails, it needs to be repeated three times. If it still fails three times, the calibration process will be exited and the reason for the failure will be returned, such as data upload failure.

[0090] Embodiment 3

[0091] Figure 7 This is a flow chart of a visual sensor calibration method provided in Embodiment 3 of the present invention, which can be executed by a cloud server.

[0092] like Figure 7 As shown, a visual sensor calibration method provided by an embodiment of the present invention specifically includes the following steps:

[0093] S310, obtaining the target image and environmental parameters uploaded by the controller.

[0094] In the embodiment of the present invention, the cloud server can obtain the target image and environmental parameters uploaded by the local controller in a wireless or wired manner, so that the cloud server can generate calibration parameters.

[0095] S320. Generate calibration parameters corresponding to the target image and environmental parameters according to the calibration neural network model, wherein the calibration neural network model is generated by training using a threshold number of image sets.

[0096] The threshold quantity may be a quantity of 2 Gigabyte (GB), 2.5 GB, 3 GB, etc. The embodiment of the present invention does not limit the threshold quantity of the image set.

[0097] It should be noted that the calibration neural network model has been trained before obtaining the target image and environmental parameters uploaded by the controller.

[0098] In an embodiment of the present invention, a calibration neural network model can be generated by training a threshold number of image sets, so that the calibration parameters corresponding to the target image and environmental parameters generated by the calibration neural network model are more accurate, thereby improving the calibration accuracy.

[0099] S330 , transmitting calibration parameters to the controller, so that the controller calibrates the visual sensor to be calibrated according to the calibration parameters.

[0100] In an embodiment of the present invention, the cloud server can transmit the calibration parameters corresponding to the target image and environmental parameters generated by the calibration neural network model to the controller via wired or wireless means, and the controller can store the calibration parameters so that the controller can calibrate the visual sensor to be calibrated according to the calibration parameters.

[0101] In an embodiment of the present invention, the cloud server can obtain the target image and environmental parameters uploaded by the controller, and can generate calibration parameters corresponding to the target image and environmental parameters by using a calibration neural network model generated in advance using a threshold number of image sets, and transmit the calibration parameters to the local controller, and the local controller stores the calibration parameters so that the local controller can effectively calibrate the visual sensor to be calibrated according to the calibration parameters.

[0102] Embodiment 4

[0103] Figure 8 This is a flow chart of a visual sensor calibration method provided by Embodiment 4 of the present invention. The embodiment of the present invention is optimized based on the above embodiment. Figure 8 As shown, a visual sensor calibration method provided by Embodiment 4 of the present invention specifically includes the following steps:

[0104] S410, obtaining the target image and environmental parameters uploaded by the controller.

[0105] S420, inputting the target image and environmental parameters into the calibration neural network model.

[0106] In the embodiment of the present invention, the target image and environmental parameters uploaded by the controller can be first converted into the input vector required by the calibration neural network model, and then the input vector is input into the pre-trained calibration neural network model to output the calibration parameters corresponding to the target image and environmental parameters, so that the calibration neural network model can recognize the target image and environmental parameters and perform corresponding processing.

[0107] S430. Collect the output vector of the calibration neural network model as calibration parameters, wherein the calibration parameters include camera intrinsic parameters, camera extrinsic parameters and distortion parameters, wherein the calibration neural network model is generated by training using a threshold number of image sets.

[0108] The output vector may be a vector corresponding to the input vector, and may be 1000-dimensional. The embodiment of the present invention does not limit the dimension of the output vector.

[0109] In an embodiment of the present invention, the calibration neural network model can be generated in advance by training with a threshold number of image sets, and the output vector of the calibration neural network model is used as a calibration parameter, and the calibration parameter includes camera intrinsic parameters, camera extrinsic parameters and distortion parameters, so that the calibration neural network model can output the calibration parameters required for the calibration of the sensor to be calibrated.

[0110] S440: Transmit the calibration parameters to the controller, so that the controller calibrates the visual sensor to be calibrated according to the calibration parameters.

[0111] In an embodiment of the present invention, the cloud server can obtain the target image and environmental parameters uploaded by the controller, input the target image and environmental parameters into a calibration neural network model, the model outputs an output vector corresponding to the input vector, and uses the vector output by the model as the calibration parameters corresponding to the target image and environmental parameters, and transmits the calibration parameters to the local controller, which stores the calibration parameters so that the local controller can effectively calibrate the visual sensor to be calibrated according to the calibration parameters.

[0112] Further, based on the above embodiment, the training process of calibrating the neural network model includes:

[0113] a1. Obtain a threshold number of image sets, wherein the image sets at least include historical target images, target standard images, and historical environmental parameters.

[0114] Among them, the image set can be divided into an image training set and an image verification set. The model can be trained using the image training set, and the effect of model learning can be verified based on the image verification set. The model can also be adjusted based on the effect of the verification set.

[0115] Among them, the historical target image can be a public target image downloaded from the Internet in advance, can be a target image collected in advance using a black and white checkerboard calibration method, can be a target image taken by different cameras at different angles, positions and under different lighting conditions. The target image can be high-definition or blurred. The embodiment of the present invention does not limit the size and resolution of the target image.

[0116] The historical environment parameters may be historical environment parameters corresponding to a public target image pre-downloaded from the Internet, or may be historical environment parameters corresponding to a target image pre-collected using a black and white chessboard calibration method.

[0117] The target standard image may be a correct target image corresponding to a historical target image and historical environmental parameters.

[0118] An embodiment of the present invention can obtain a threshold number of image sets that include at least historical target images, target standard images, and historical environmental parameters, so that the model can effectively learn the relationship between historical target images and historical environmental parameters and target standard images.

[0119] b1. Inputting historical target images and historical environmental parameters into the network model to be trained to obtain the weights and biases of the network model to be trained, wherein the network model to be trained is a back propagation neural network model.

[0120] The connection strength between neurons can be represented by weights. Bias can be an important parameter in the model, which is equivalent to adding a constant to correctly classify samples.

[0121] Among them, the Back Propagation Neural Network (BPNN) model can be a multi-layer feedforward neural network trained according to the error back propagation algorithm, which can be composed of three layers: input layer, hidden layer and output layer, wherein the hidden layer transmits important information between the input layer and the output layer. It can also be composed of one or more hidden layers, thereby allowing the network to model complex functions. The calculation process of the back propagation neural network can be composed of forward calculation and reverse calculation processes. In the forward propagation process, the input pattern is processed layer by layer from the input layer through the hidden layer and turned to the output layer. The state of each layer of neurons only affects the state of the next layer of neurons. If the expected output cannot be obtained in the output layer, it will turn to back propagation, return the error signal along the original connection path, and minimize the error signal by modifying the weights of each neuron.

[0122] In the embodiment of the present invention, the historical target images and historical environment parameters can be input into the back-propagation neural network model to be trained to obtain the weights and biases in the forward propagation of the model.

[0123] c1. Obtain the output vector of the network model to be trained, and use the output vector to generate a corrected image of the historical target image.

[0124] In the embodiment of the present invention, the output vector of the network model to be trained is obtained, and the input historical target image is corrected according to the output vector to obtain a corrected image of the historical target image.

[0125] d1. If the error between the correction image and the target standard image is less than the preset error, the network model to be trained is determined as the calibration neural network model to complete the training;

[0126] The preset error may be 0.1 or 0.01, and the embodiment of the present invention does not limit the specific value of the preset error.

[0127] In an embodiment of the present invention, after obtaining a corrected image of a historical target image, if the error between the corrected image and the corresponding target standard image is less than a preset error, the model can stop training, and it can be considered that the network model to be trained at this time has completed training, and the trained model is used as a calibration neural network model.

[0128] e1. If the error between the corrected image and the target standard image is greater than or equal to the preset error, the weights and biases of the network model to be trained are adjusted and the historical target image and historical environment parameters are re-input for training.

[0129] In an embodiment of the present invention, after obtaining a corrected image of a historical target image, if the error between the corrected image and the corresponding target standard image is greater than or equal to a preset error, the weights and biases obtained in the forward propagation are corrected and adjusted in the back propagation, and then the historical target image and historical environment parameters are re-input for model training.

[0130] In an embodiment of the present invention, a threshold number of image sets including at least historical target images, target standard images, and historical environmental parameters are obtained, and the historical target images and historical environmental parameters are input into a back propagation neural network model to be trained to obtain weights and biases in the forward propagation of the model to be trained, and a correction image corresponding to the historical target image is generated according to the output vector of the model. If the error between the correction image and the target standard image is less than a preset error, the model training is stopped. If the error between the correction image and the target standard image is greater than or equal to the preset error, the weights and biases obtained in the forward propagation are corrected and adjusted in the back propagation of the model, and the historical target image and historical environmental parameters are re-input for model training. The model training is stopped until the error between the correction image and the target standard image is less than the preset error or the preset number of training times is reached. A model with higher calibration accuracy and more stable effect can be trained, so that in the subsequent calibration process, more effective calibration parameters can be generated according to the locally uploaded target images and environmental parameters, thereby obtaining an image with minimal distortion.

[0131] Exemplarily, in an exemplary embodiment, taking the process of using back propagation neural network training as an example, Fig. 9 A schematic diagram of the back propagation neural network training process provided by an embodiment of the present invention is shown in FIG. Fig. 9As shown in the figure, first, the back propagation neural network includes an input layer, a hidden layer and an output layer. The input layer includes 2 neurons, the hidden layer includes 3 neurons, and the output layer includes 2 neurons. In the forward propagation process, the input layer is processed layer by layer through the hidden layer and then transferred to the output layer. The state of each layer of neurons only affects the state of the next layer of neurons. If the expected output cannot be obtained in the output layer, it will turn to back propagation, return the error signal along the original connection path, and minimize the error signal by modifying the weights of each neuron.

[0132] For example, Fig.10 A schematic diagram of a back propagation neural network for image correction provided by an embodiment of the present invention is shown in FIG. Fig.10 As shown, u and v are pixel coordinates. After the back-propagation neural network, the input is weighted and added with the corresponding bias to obtain the corrected pixel coordinates u,, v,, so that the spatial point coordinates can be reconstructed.

[0133] Exemplarily, in an exemplary embodiment, a calibration implementation method of a visual sensor calibration method is provided. Fig.11 Another calibration implementation diagram of the visual sensor calibration method provided by the embodiment of the present invention is shown in FIG. Fig.11 As shown, the diagnostic device sends a calibration request to the advanced automatic driving controller through the central gateway, and the advanced automatic driving controller sends a calibration instruction to the camera; the camera, as a module to be calibrated, imports the calibration target image information into the advanced automatic driving controller; after receiving the target image information and the vehicle information provided by the whole vehicle, the advanced automatic driving controller packages the data and sends it to the central gateway, and passes it to the smart antenna through the central gateway. The smart antenna uploads the data to the cloud server through the data upload instruction, and the calibration process of the cloud server begins to execute; wherein, the controller can upload the image information, body posture information, vehicle platform, model, controller code and other information collected by the local camera to the cloud server through the data upload link, and the cloud server stores the camera code, calibration parameters, vehicle parameters and other information in the database, and the cloud server performs calibration calculation, and sends the calibration parameters to the advanced automatic driving controller of the local vehicle terminal at the same time, wherein the calibration parameters include camera internal and external parameters and distortion parameters. The advanced automatic driving controller writes the camera internal and external parameters (internal parameter matrix, focal length; rotation matrix, translation matrix) and distortion parameters, and feeds back the successful write instruction to the cloud server to complete the camera calibration.

[0134] Embodiment 5

[0135] Embodiment 5 of the present invention provides a controller. Fig.12 Schematic diagram of the structure of a controller provided by Embodiment 5 of the present invention. Fig.12As shown, this embodiment provides a controller, which may include: a processor and a memory. The memory stores at least one instruction, and the instruction is executed by the processor, so that the controller executes the visual sensor calibration method as described in the above method embodiments 1 and 2.

[0136] Reference Fig.12 The controller may specifically include: a processor 1220, a memory 1210, an input device 1230, and an output device 1240; the number of the processor 1220 in the controller may be one or more. Fig.12 In the example, a processor 1220 is used; the number of memories 1210 in the controller may be one or more. Fig.12 A memory 1210 is taken as an example. The processor 1220, the memory 1210, the input device 1230 and the output device 1240 in the controller can be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0137] Specifically, in the embodiment, when the processor 1220 executes one or more programs stored in the memory 1210, the following operations may be implemented: obtaining a target image collected by the visual sensor to be calibrated and environmental parameters of the visual sensor to be calibrated;

[0138] Uploading the target image and the environmental parameters to a cloud server to obtain calibration parameters;

[0139] The visual sensor to be calibrated is calibrated according to the calibration parameters.

[0140] The visual sensor calibration method, controller, server, and storage medium provided in the above embodiments can execute the visual sensor calibration method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the method. For technical details not described in detail in the above embodiments, please refer to the visual sensor calibration method provided in any embodiment of the present invention.

[0141] Embodiment 6

[0142] Embodiment 6 of the present invention provides a server. Fig.13 Schematic diagram of the structure of a server provided by Embodiment 6 of the present invention. Fig.13 As shown, this embodiment provides a server, which may include: a processor and a memory. The memory stores at least one instruction, and the instruction is executed by the processor, so that the server executes the visual sensor calibration method as described in the above method embodiment 3 and embodiment 4.

[0143] Reference Fig.13The server may specifically include: a processor 1320, a memory 1310, an input device 1330, and an output device 1340; the number of the processor 1320 in the server may be one or more. Fig.13 In the example, a processor 1320 is used; the number of memories 1310 in the server may be one or more. Fig.13 A memory 1310 is taken as an example. The processor 1320, the memory 1310, the input device 1330 and the output device 1340 in the server can be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0144] Specifically, in the embodiment, when the processor 1320 executes one or more programs stored in the memory 1310, the following operations may be implemented: obtaining the target image and environmental parameters uploaded by the controller;

[0145] Generating calibration parameters corresponding to the target image and the environmental parameters according to a calibration neural network model, wherein the calibration neural network model is generated by training using a threshold number of image sets;

[0146] The calibration parameters are transmitted to the controller so that the controller calibrates the visual sensor to be calibrated according to the calibration parameters.

[0147] The visual sensor calibration method, controller, server, and storage medium provided in the above embodiments can execute the visual sensor calibration method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the method. For technical details not described in detail in the above embodiments, please refer to the visual sensor calibration method provided in any embodiment of the present invention.

[0148] Embodiment 7

[0149] Embodiment 7 of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the visual sensor calibration method provided by all the embodiments of the present invention is implemented. The method includes: acquiring a target image collected by the visual sensor to be calibrated and environmental parameters of the visual sensor to be calibrated;

[0150] Uploading the target image and the environmental parameters to a cloud server to obtain calibration parameters;

[0151] The visual sensor to be calibrated is calibrated according to the calibration parameters.

[0152] or

[0153] Get the target image and environmental parameters uploaded by the controller;

[0154] Generating calibration parameters corresponding to the target image and the environmental parameters according to a calibration neural network model, wherein the calibration neural network model is generated by training using a threshold number of image sets;

[0155] The calibration parameters are transmitted to the controller so that the controller calibrates the visual sensor to be calibrated according to the calibration parameters.

[0156] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0157] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0158] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0159] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0160] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A visual sensor calibration method, characterized in that: Applied to a controller, the method comprises: Acquire a target image collected by the visual sensor to be calibrated and environmental parameters of the visual sensor to be calibrated; Uploading the target image and the environmental parameters to a cloud server to obtain calibration parameters; Calibrate the visual sensor to be calibrated according to the calibration parameters; The uploading of the target image and the environmental parameters to the cloud server to obtain calibration parameters includes: Uploading the target image and the environmental parameters to the cloud server, so that the cloud server generates calibration parameters corresponding to the target image and the environmental parameters using a calibration neural network model; Obtaining the calibration parameters sent by the cloud server; The training process of the calibration neural network model includes: Acquire a threshold number of image sets, wherein the image sets at least include historical target images, target standard images, and historical environmental parameters; Inputting the historical target image and the historical environment parameters into a network model to be trained to obtain weights and biases of the network model to be trained, wherein the network model to be trained is a back propagation neural network model; Obtaining an output vector of the network model to be trained, and using the output vector to generate a corrected image of the historical target image; If the error between the corrected image and the target standard image is less than a preset error, determining that the network model to be trained is used as a calibration neural network model to complete training; If the error between the corrected image and the target standard image is greater than or equal to the preset error, the weight and the bias of the network model to be trained are adjusted and the historical target image and the historical environment parameters are re-input for training.

2. The method according to claim 1, characterized in that The step of acquiring a target image collected by the visual sensor to be calibrated and environmental parameters of the visual sensor to be calibrated includes: Controlling the visual sensor to be calibrated on the vehicle to collect image information of a calibration target as a target image, wherein the calibration target includes a black and white checkerboard calibration target, and the vehicle is in a calibration preparation state; The vehicle body posture information, vehicle model and controller model of the vehicle are collected as environmental parameters.

3. The method according to claim 2, characterized in that The vehicle is in a calibration preparation state, including: The vehicle completes four-wheel alignment; The tire pressure of the vehicle tire is at the standard tire pressure; The air suspension of the vehicle is at a standard suspension height; The vehicle body running axis is horizontal to the normal line of the standard plate of the calibration target; The vehicle speed reaches the standard vehicle speed.

4. The method according to claim 1, characterized in that: The calibration parameters include: camera intrinsic parameters, camera extrinsic parameters and distortion parameters.

5. A visual sensor calibration method, characterized in that: Applied to a cloud server, the method includes: Get the target image and environmental parameters uploaded by the controller; Generating calibration parameters corresponding to the target image and the environmental parameters according to a calibration neural network model, wherein the calibration neural network model is generated by training using a threshold number of image sets; Transmitting the calibration parameters to the controller so that the controller calibrates the visual sensor to be calibrated according to the calibration parameters; The training process of the calibration neural network model includes: Acquire a threshold number of image sets, wherein the image sets at least include historical target images, target standard images, and historical environmental parameters; Inputting the historical target image and the historical environment parameters into a network model to be trained to obtain weights and biases of the network model to be trained, wherein the network model to be trained is a back propagation neural network model; Obtaining an output vector of the network model to be trained, and using the output vector to generate a corrected image of the historical target image; If the error between the corrected image and the target standard image is less than a preset error, determining that the network model to be trained is used as a calibration neural network model to complete training; If the error between the corrected image and the target standard image is greater than or equal to the preset error, the weight and the bias of the network model to be trained are adjusted and the historical target image and the historical environment parameters are re-input for training.

6. The method according to claim 5, characterized in that The step of generating calibration parameters corresponding to the target image and the environmental parameters according to the calibration neural network model includes: Inputting the target image and the environmental parameters into the calibration neural network model; The output vector of the calibration neural network model is collected as a calibration parameter, wherein the calibration parameter includes a camera intrinsic parameter, a camera extrinsic parameter and a distortion parameter.

7. A controller, characterized in that: The controller comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the visual sensor calibration method as described in any one of claims 1 to 4.

8. A server, characterized in that: The server comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the visual sensor calibration method as described in any one of claims 5-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the visual sensor calibration method as described in any one of claims 1-4 or 5-6 is implemented.

Images