Camera parameter calibration method, electronic device, computer program product and vehicle

Abstract

The embodiment of the invention is suitable for the technical field of computers, and provides a camera parameter calibration method, electronic equipment, a computer program product and a vehicle, and the method comprises the steps: inputting a calibration image corresponding to any vehicle-mounted camera into an overall feature extraction network, and determining an overall feature corresponding to the calibration image; the overall features comprise features of a to-be-calibrated area where a calibration object is located and features of a non-to-be-calibrated area in the calibration image; inputting a to-be-calibrated area where the calibration object is located in the calibration image and the overall features into the coordinate detection network, and generating image coordinates of the calibration object in an image coordinate system; and determining external parameters corresponding to the any vehicle-mounted camera based on the image coordinates and the calibration information. According to the method, the accuracy of the image coordinates can be improved, so that the accuracy of the external parameters determined by the electronic equipment according to the image coordinates is improved.

Description

Technical Field

[0001] The embodiments of the present application belong to the field of computer technology, and in particular, relate to a camera parameter calibration method, electronic equipment, computer program product, and vehicle. Background Art

[0002] With the increasing demand of consumers for driving safety and the development of automobile intelligence, more and more cars are equipped with on-board panoramic view systems (Around View Monitor, AVM). Among them, the on-board panoramic view system is mainly composed of an image processing unit, a display screen, and multiple on-board cameras installed at different positions of the vehicle. The on-board panoramic view system can obtain environmental images through on-board cameras at different positions, and then the image processing unit needs to combine the external parameters of the on-board cameras at different positions to perform operations such as correction and splicing on the environmental images at different positions, thereby generating a panoramic image around the vehicle body. Finally, the on-board panoramic view system can display the panoramic image through the display screen to provide the driver with a panoramic image around the vehicle body, reduce the driver's visual blind spots, and thus improve driving safety.

[0003] Since the accuracy of the panoramic image generated by the on-board panoramic surround view system is closely related to the external parameters of each on-board camera, technicians are required to calibrate the external parameters of the on-board camera before the vehicle leaves the factory. In the prior art, when calibrating the external parameters of the on-board camera, the technicians usually drive the vehicle into a calibration scene where multiple calibration plates are installed, and use the on-board camera to shoot the calibration scene to obtain a calibration image. Then, the technicians can obtain the image coordinates of the calibration plate in the image coordinate system through the corner detection algorithm, and determine the external parameters of the on-board camera based on the image coordinates and the world coordinates of the calibration plate in the vehicle coordinate system. However, the existing corner detection algorithm has the problem of being unable to fully extract all the features in the image, thereby reducing the accuracy of the image coordinates, and then reducing the accuracy of the calibrated external parameters. Summary of the invention

[0004] In view of this, embodiments of the present application provide a camera parameter calibration method, electronic device, computer program product and vehicle to improve the accuracy of camera external parameters.

[0005] A first aspect of an embodiment of the present application provides a camera parameter calibration method, comprising:

[0006] Inputting the corresponding input of any vehicle-mounted camera into the overall feature extraction network to determine the overall features corresponding to the calibration image; the overall features include the features of the area to be calibrated where the calibration object is located and the features of the area not to be calibrated in the calibration image;

[0007] Inputting the area to be calibrated where the calibration object is located and the overall features into a coordinate detection network to generate image coordinates of the calibration object in an image coordinate system;

[0008] Based on the image coordinates and the calibration information, the external parameters corresponding to any of the vehicle-mounted cameras are determined.

[0009] In a possible implementation manner of the first aspect, inputting the calibration image into an overall feature extraction network to determine the overall features corresponding to the calibration image includes:

[0010] Extracting features from the calibration image through a plurality of first convolutional layers in the overall feature extraction network to generate a plurality of feature images related to the calibration image;

[0011] According to the network layer order of the first convolutional layer corresponding to each of the feature images, downsampling operations are performed on each of the feature images in sequence to generate feature information corresponding to each of the feature images;

[0012] The overall features corresponding to the calibration image are generated based on the feature information corresponding to each network level.

[0013] In a possible implementation manner of the first aspect, the method further includes:

[0014] The calibration image is input into a polygon fitting algorithm and a region detection algorithm respectively; the polygon fitting algorithm is an algorithm for fitting polygons in the calibration image according to a preset calibration object shape to determine the region where the calibration object is located from the calibration image; the region detection algorithm is an algorithm obtained by training a deep learning algorithm based on a first training sample generated by the calibration object;

[0015] If the polygon fitting algorithm detects a first candidate region based on the shape of the calibration object, the first candidate region is used as the region to be calibrated.

[0016] In a possible implementation manner of the first aspect, the method further includes:

[0017] If the polygon fitting algorithm fails to detect the first candidate region based on the shape of the calibration object, the second candidate region determined by the region detection algorithm is used as the region to be calibrated.

[0018] In a possible implementation manner of the first aspect, the coordinate detection network includes a regional feature extraction network and a second convolutional layer;

[0019] The step of inputting the to-be-calibrated area where the calibration object in the calibrated image is located and the overall features into the coordinate detection network to generate the image coordinates of the calibration object in the image coordinate system comprises:

[0020] Inputting the area to be calibrated into a regional feature extraction network to obtain regional features corresponding to the area to be calibrated;

[0021] Performing feature fusion on the regional features and the overall features to determine fusion features;

[0022] The fused features are input into the second convolutional layer to generate the image coordinates of the calibration object in the image coordinate system.

[0023] In a possible implementation manner of the first aspect, the method further includes:

[0024] Acquire a second training sample and a model to be trained corresponding to the vehicle-mounted panoramic view system, and process the training image in the training sample by the model to be trained to generate initial coordinates corresponding to the calibration object in the training image; the vehicle-mounted panoramic view system includes at least one vehicle-mounted camera; the second training sample includes training images corresponding to each vehicle-mounted camera and expected coordinates corresponding to the calibration object in the training image;

[0025] Determining a loss value between the initial coordinates and the expected coordinates based on a preset loss function;

[0026] The model to be trained is updated based on the loss value until the loss value satisfies a preset training stop condition, and the model to be trained corresponding to when the loss value satisfies the training stop condition is used as a coordinate extraction model; the coordinate extraction model includes the overall feature extraction network and the coordinate detection network.

[0027] In a possible implementation manner of the first aspect, the loss value includes a coordinate loss value and a pixel loss value; the training sample also includes a desired panoramic image spliced ​​from all training images in the training sample;

[0028] The determining, based on a preset loss function, a loss value between the initial coordinates and the expected coordinates corresponding to the calibration object includes:

[0029] For any on-board camera in the on-board panoramic surround view system, inputting the initial coordinates of the training image corresponding to any on-board camera and the expected coordinates into a first loss function to generate a coordinate loss value corresponding to any on-board camera;

[0030] Determine the initial parameters corresponding to any of the on-board cameras based on the initial coordinates, the world coordinates of the calibration object in the vehicle coordinate system, and the internal parameters corresponding to any of the on-board cameras;

[0031] Based on the initial parameters corresponding to each of the vehicle-mounted cameras, all the training images in the training sample are stitched together to determine an initial panoramic image;

[0032] The initial panoramic image and the desired panoramic image are input into a second loss function to determine the pixel loss value.

[0033] A second aspect of an embodiment of the present application provides a camera parameter calibration device, including:

[0034] A feature acquisition module, used to input the corresponding input of any vehicle-mounted camera into the overall feature extraction network to determine the overall features corresponding to the calibration image; the overall features include the features of the area to be calibrated where the calibration object is located and the features of the area not to be calibrated in the calibration image;

[0035] A coordinate generation module, used for inputting the area to be calibrated where the calibration object is located and the overall features into the coordinate detection network to generate image coordinates of the calibration object in an image coordinate system;

[0036] A parameter determination module is used to determine the external parameters corresponding to any vehicle-mounted camera based on the image coordinates and calibration information.

[0037] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the camera parameter calibration method as described in the first aspect above is implemented.

[0038] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the camera parameter calibration method as described in the first aspect above is implemented.

[0039] A fifth aspect of the embodiments of the present application provides a computer program product. When the computer program product is run on a computer, the computer executes the camera parameter calibration method described in the first aspect.

[0040] A sixth aspect of the embodiment of the present application provides a vehicle, including an electronic device, wherein the electronic device is used to execute the camera parameter calibration method as described in the first aspect.

[0041] Compared with the prior art, the embodiments of the present application have the following advantages:

[0042] In an embodiment of the present application, the electronic device can determine the image coordinates of the calibration object in the image coordinate system according to the overall features of the calibration image and the area to be calibrated where the calibration object is located, wherein the overall features may include the features of the area to be calibrated where the calibration object is located in the calibration image and the features of the area not to be calibrated. When the prior art only relies on the area to be calibrated where the calibration object is located for coordinate calibration, it is easy to introduce errors caused by local features in the process of coordinate calibration. However, this solution combines the overall features of the image for coordinate detection on the basis of the area to be calibrated. Furthermore, since the overall features include not only the features of the area to be calibrated, but also the features of the area not to be calibrated, therefore, combining the overall features for coordinate calibration enables the electronic device to constrain and adjust the errors caused by the local area to be calibrated according to the global overall features during coordinate calibration, thereby improving the accuracy of the image coordinates, and further improving the accuracy of the external parameters calibrated by the vehicle-mounted equipment based on the image coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or prior art descriptions. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a schematic diagram of a camera parameter calibration method provided in an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of a calibration site provided in an embodiment of the present application;

[0046] Figure 3 is a schematic diagram of an overall feature extraction network provided in an embodiment of the present application;

[0047] Figure 4 It is a specific flow diagram of a coordinate detection network provided in an embodiment of the present application;

[0048] Figure 5 is a schematic diagram of another camera parameter calibration method provided in an embodiment of the present application;

[0049] Figure 6 It is a specific flow chart of a coordinate extraction model provided in an embodiment of the present application;

[0050] Figure 7 is a schematic diagram of another camera parameter calibration method provided in an embodiment of the present application;

[0051] Figure 8This is a schematic diagram of a loss value calculation process provided in an embodiment of the present application;

[0052] Fig. 9 is a schematic diagram of a camera parameter calibration device provided in an embodiment of the present application;

[0053] Fig.10 It is a schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In the following description, specific details such as specific system structures, technologies, etc. are proposed for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from hindering the description of the present application.

[0055] With the development of automobile intelligence, consumers' requirements for driving safety and convenience are also increasing. Therefore, more and more consumers will pay attention to the display effect of the on-board panoramic view system when buying cars. Among them, the on-board panoramic view system is mainly composed of an image processing unit, a display screen, and multiple on-board cameras installed in different positions of the vehicle.

[0056] On-board cameras: The on-board panoramic view system includes at least 4 on-board cameras, which are installed in the front, rear, left and right of the vehicle. Some high-end models may also add cameras on the side of the vehicle to provide the driver with a more comprehensive perspective. These on-board cameras can be used to collect environmental images of the vehicle's environment.

[0057] Image processing unit: The image processing unit can be a unit inside the display screen, which is used to process, stitch and correct the environmental images collected by each vehicle camera to generate a coherent panoramic image. The image processing unit usually has powerful computing power, can process image data in real time, and generate high-definition and high-accuracy panoramic images.

[0058] Display screen: The display screen can be installed on the center console or dashboard of the vehicle to display the panoramic image generated by the image processing unit. The driver can clearly see the environment around the vehicle through the panoramic image displayed on the display screen. Among them, the size and resolution of the display screen will affect the display effect of the panoramic image. A high-resolution display screen can display a clearer panoramic image.

[0059] Since the panoramic image of the on-board panoramic surround view system needs to reflect the position and distance of the vehicle in the real world as much as possible, the image processing unit needs to correct and splice the environmental images collected by each on-board camera according to the external parameters of each on-board camera when generating the panoramic image. Therefore, the accuracy of the external parameters of the on-board camera will greatly affect the imaging effect of the on-board panoramic surround view system. Before the vehicle leaves the factory or after the installation position / attitude of the on-board camera changes, technicians are often required to calibrate the external parameters of each on-board camera in the on-board panoramic surround view system. In general, the external parameter calibration can be roughly divided into the following steps:

[0060] Step 1: Image preprocessing: Improve the contrast between the calibration object and the background in the calibration image by using a preset image enhancement method;

[0061] Step 2: Calibration object detection: Detect the calibration object in the predetermined calibration image to determine the image coordinates of the calibration object in the calibration image. When the on-board camera in the on-board panoramic view system is a fisheye camera, the technician can directly perform calibration object detection on the calibration image to obtain the distorted coordinates of the calibration object in the distorted calibration image, and then use the distortion coefficient corresponding to the on-board camera to dedistort the distorted coordinates to obtain the image coordinates corresponding to the calibration object; the technician can also first dedistort the calibration image, and then perform calibration object detection on the dedistorted image calibration image to obtain the image coordinates corresponding to the calibration object. This step has the greatest impact on the accuracy of the calibrated external parameters;

[0062] Step 3: Calculate external parameters: Calculate the image coordinates of the calibration object in the calibration image and the world coordinates of the calibration object in the vehicle coordinate system to obtain the external parameters of the vehicle camera, such as rotation matrix, translation vector, position information of the vehicle camera, and attitude information of the vehicle camera.

[0063] Step 4: Effect verification: Use the calculated external parameters and the internal parameters of the vehicle-mounted camera to project the environmental image captured by the vehicle-mounted camera and verify the effect after projection.

[0064] In the prior art, when detecting the calibration object in step 2, technicians usually obtain the image coordinates of the calibration plate in the image coordinate system through a corner detection algorithm. However, the corner detection algorithm has the problem of being unable to fully extract all features in the image. In view of this, the embodiment of the present application provides a camera parameter calibration method for detecting the calibration object in combination with the overall features of the calibration image, so as to improve the accuracy of the image coordinates detected by the calibration object.

[0065] The technical solution of the present application is described below through specific embodiments.

[0066] Reference Figure 1, shows a schematic diagram of a camera parameter calibration method provided in an embodiment of the present application. The method can be applied to electronic devices such as computers, tablet computers, mobile terminals, and vehicle-mounted terminals. The camera parameter calibration method may specifically include the following steps:

[0067] S101, inputting the corresponding input of any vehicle-mounted camera into the overall feature extraction network to determine the overall features corresponding to the calibration image.

[0068] In this embodiment, when a technician needs to calibrate the external parameters of the on-board panoramic view system on a vehicle, the technician can drive the vehicle into a calibration scene with multiple calibration plates installed, and use each on-board camera in the on-board panoramic view system to shoot the calibration scene to obtain a calibration image. After the vehicle enters the calibration scene, the technician can control each on-board camera in the on-board panoramic view system to shoot a calibration image of the scene. Among them, the calibration image captured by each on-board camera can include at least one calibration object.

[0069] See also Figure 2 , shows a schematic diagram of a calibration site provided in an embodiment of the present application. Figure 2 As shown, the calibration object in the calibration scene can be a black square calibration plate. Among them, a black square calibration plate can be arranged at the upper left corner, lower left corner, lower right corner and upper right corner of the vehicle as a calibration object. It should be noted that the calibration object can be an object of any shape, such as a circular calibration plate, and the embodiment of the present application is not used to specifically limit the calibration object. Figure 2 In the calibration scene shown, the calibration image captured by the vehicle camera may include two calibration objects. For example, the calibration image captured by the vehicle camera installed in front of the vehicle may include a calibration object in the upper left corner of the vehicle and a calibration object in the upper right corner of the vehicle.

[0070] After any on-board camera in the on-board panoramic view system captures the calibration image, the on-board panoramic view system can transmit the calibration image to an electronic device equipped with a coordinate extraction model. Among them, the electronic device can be an on-board terminal in the display screen, or it can be an external device such as a computer or a tablet computer connected to the display screen. Specifically, the electronic device can directly determine the external parameters of the on-board camera through the on-board terminal, and store the determined external parameters. The electronic device can also be an external device. For example, when the technician needs to obtain the external parameters of the on-board camera, the technician can establish a data transmission channel between the external device equipped with the coordinate extraction model and the on-board terminal in the display screen. After the on-board terminal obtains the calibration image, it can transmit the calibration image to the external device through the data transmission channel to determine the external parameters of the on-board camera through the external device. The external device can transmit the determined external parameters to the on-board terminal of the display screen so that the on-board terminal can store the external parameters.

[0071] After receiving the calibration image corresponding to any vehicle-mounted camera, the electronic device can input the calibration image of the vehicle-mounted camera into the coordinate extraction model. The coordinate extraction model may include an overall feature extraction network. The electronic device can obtain the overall features corresponding to the calibration image through the overall feature extraction network in the coordinate extraction model. The overall features may include the features of the to-be-calibrated area where the calibration object is located in the calibration image and the features of the non-to-be-calibrated area.

[0072] Specifically, the overall feature extraction network is a network model that extracts the global information of an image from a calibration image. After obtaining a calibration image corresponding to a vehicle-mounted camera, the electronic device can input the calibration image into the overall feature extraction network to determine the overall features corresponding to the calibration image through the overall feature extraction network.

[0073] In one possible implementation, see Figure 3 , which shows a schematic diagram of an overall feature extraction network provided by an embodiment of the present application. Figure 3 As shown, the overall feature extraction network may include multiple first convolution layers, such as first convolution layer A to first convolution layer C. The electronic device may sequentially perform convolution processing on the image to be detected through multiple first convolution layers. After each first convolution layer performs convolution processing on the calibration image, it may generate a feature image corresponding to the current convolution layer, and input the current feature image into the next first convolution layer for secondary convolution processing. The electronic device may generate multiple feature images with different degrees of convolution through multiple first convolution layers in the overall feature extraction network. Figure 3As shown, the electronic device can generate feature image A through the first convolution layer A, generate feature image B through the first convolution layer B, and generate feature image C through the first convolution layer C. Exemplarily, the convolution kernel size of the first convolution layer can be 3*3, and the step size can be 2.

[0074] After the electronic device generates multiple feature images with different convolutional layers through multiple convolutional layers, it can perform upsampling operations on each feature image in the feature extraction network according to the network level order of the convolutional layer corresponding to each feature image, so as to generate feature information corresponding to each feature image. Specifically, for a certain network level order, the electronic device can first perform a downsampling operation on the feature image of the current network level order, so that the pixel value of the feature image after the current network level order is downsampled is the same as the pixel value of the feature image of the next network level order. Then, the electronic device can fuse the feature image after the current network level order is downsampled with the feature image of the next network level order. In the fusion process, in order to ensure that the number of channels between feature images of different network level orders is the same, the electronic device can copy the feature image with a lower network level order through a broadcast mechanism. Exemplarily, the electronic device can add the feature image after the current network level order is downsampled with the feature image of the next network level order pixel by pixel to generate feature information corresponding to the current network level order. Then, the electronic device can generate the overall features corresponding to the calibration image according to the feature information corresponding to each network level.

[0075] S102: Input the area to be calibrated and the overall features of the calibration object into a coordinate detection network to generate image coordinates of the calibration object in the image coordinate system.

[0076] In this embodiment, the coordinate extraction model may also include a coordinate detection network. The coordinate detection network is a network model that generates image coordinates corresponding to the calibration object based on the overall features of the calibration image and the area to be calibrated where the calibration object is located. After the electronic device determines the overall features corresponding to the calibration image through the overall feature extraction network in the coordinate extraction model, the overall features and the area to be calibrated can be input into the coordinate detection network to generate the image coordinates of the calibration object in the image coordinate system. The image coordinate system can be a coordinate axis with the intersection of the optical axis of the vehicle-mounted camera and the projection surface as the origin, and the horizontal and vertical axes are parallel to the projection surface of the vehicle-mounted camera. The image coordinate system can be used to represent the position of an object on the camera imaging plane.

[0077] In one possible implementation, to further illustrate the specific process of the coordinate detection network generating image coordinates, see Figure 4 , shows a specific flow chart of a coordinate detection network provided by an embodiment of the present application. Figure 4As shown, the coordinate detection network may include a regional feature extraction network, a second convolutional layer, and a pooling layer. After determining the area to be calibrated of the calibration object in the calibration image, the electronic device may input the area to be calibrated into the regional feature extraction network to obtain the regional features corresponding to the area to be calibrated through the regional feature extraction network. Then, the electronic device may perform a feature connection operation on the regional features and the overall features to generate features to be fused. The electronic device may input the features to be fused into the feature fusion layer to perform feature fusion on the features to be fused through the feature fusion layer to determine the fusion features corresponding to the calibration image. After determining the fusion features, the electronic device may input the fusion features into the second convolutional layer to perform a dimensionality reduction operation on the fusion features. Then, the electronic device may input the fusion features after dimensionality reduction into the pooling layer to generate the image coordinates of the calibration object in the image coordinate system.

[0078] In the embodiment of the present application, since the electronic device can perform coordinate calibration based on the fusion features formed by splicing the regional features corresponding to the area to be calibrated and the overall features, that is, in the process of coordinate calibration, the overall features can provide a macroscopic reference framework for the entire calibration process of the image coordinates, such as the relative position relationship between the calibration object and the surrounding environment, and the regional features corresponding to the area to be calibrated can provide more accurate local information, such as the edge shape of the calibration object, therefore, the method provided in this embodiment can improve the accuracy of the image coordinates calibrated by the electronic device.

[0079] In addition, since the overall feature extraction network and the coordinate detection network in the coordinate extraction model are two independent neural network structures, there is no structural connectivity between the two. Therefore, technicians can separately optimize the overall feature extraction network and the regional coordinate detection network during model training without the need to adjust parameters together. Therefore, the method provided in this embodiment can improve the training efficiency of the coordinate extraction model.

[0080] S103: Determine the external parameters corresponding to any vehicle-mounted camera based on the image coordinates and the calibration information.

[0081] In this embodiment, for any vehicle-mounted camera in the vehicle-mounted panoramic view system, after the electronic device obtains the image coordinates corresponding to the calibration object in the calibration image of the vehicle-mounted camera, it can determine the external parameters corresponding to the vehicle-mounted camera based on the image coordinates of the calibration object, the world coordinates of the calibration object in the vehicle coordinate system and the internal parameters corresponding to any vehicle-mounted camera.

[0082] Among them, the internal parameters of the vehicle camera are parameters representing the imaging model of the vehicle camera, which describe the conversion relationship between the pixel coordinate system and the image coordinate system, and can be used to realize the conversion between the pixel coordinate system and the image coordinate system. Specifically, the internal parameters of the vehicle camera may include parameters such as the center of the optical axis of the vehicle camera, focal length, and distortion coefficient. The external parameters of the vehicle camera are parameters describing the position and posture of the camera in the vehicle coordinate system, which can be used to realize the conversion between the image coordinate system and the vehicle coordinate system. The external parameters of the vehicle camera may include parameters such as the conversion matrix, the position information of the vehicle camera, and the posture information of the vehicle camera. Among them, the vehicle coordinate system may be a coordinate axis with the center of mass of the vehicle as the origin, the horizontal axis aligned with the driving direction of the vehicle, and the vertical axis perpendicular to the driving direction of the vehicle. The vehicle coordinate system can be used to describe the movement of the vehicle.

[0083] Exemplarily, the electronic device can calculate the transformation matrix of the vehicle camera according to the image coordinates and world coordinates of the calibration object. The electronic device can also input the image coordinates, world coordinates and internal parameters into the perspective n-point (PNP) function to determine the external parameters corresponding to the vehicle camera. The electronic device can calculate the external parameters by any method known to those skilled in the art, and the embodiments of the present application are not used to specifically limit the calculation method of the external parameters.

[0084] Through the method provided in this embodiment, since the overall features of the image include the features of the area to be calibrated where the calibration object is located in the calibration image and the features of the non-to-be-calibrated area, macro information about the entire scene, such as the overall structure of the image, edge distribution, etc., can be provided. Therefore, during the coordinate calibration process, the electronic device can determine the macro information of the entire scene, such as the overall structure, object distribution, etc., through the overall features to preliminarily locate the calibration object. Further, the area to be calibrated provides more detailed local information, so the electronic device can focus on the calibration object itself according to the area to be calibrated. Combining the overall features and the area to be calibrated, the electronic device can more accurately determine the position of the calibration object. In summary, since the overall features can help eliminate some false matches caused by local interference or noise, and the detailed information of the area to be calibrated can accurately determine the specific coordinates of the calibration object, the method provided in this embodiment improves the calibration accuracy of the image coordinates. Further, since the external parameters of the vehicle-mounted camera are calibrated on the basis of the image coordinates, the method provided in this embodiment can improve the calibration accuracy of the external parameters.

[0085] On the basis of the first embodiment of the present application, in order to further illustrate a specific method for an electronic device to determine a to-be-calibrated area where a calibration object is located, Figure 5FIG. 1 shows a specific implementation flow chart of a camera parameter calibration method S102 provided in the second embodiment of the present application. Figure 5 , compared to Figure 1 In the embodiment, a camera parameter calibration method provided in the embodiment includes before S102: S501 to S503, which are described in detail as follows:

[0086] S501 , inputting the calibration image into the polygon fitting algorithm and the region detection algorithm respectively.

[0087] The coordinate extraction model may include a polygon fitting algorithm and a region detection algorithm. After the electronic device obtains the calibration image corresponding to a certain vehicle-mounted camera, it can input the calibration image into the polygon fitting algorithm and the region detection algorithm respectively. Among them, the polygon fitting algorithm can be an algorithm for fitting the polygons in the calibration image according to the preset calibration object shape to determine the area where the calibration object is located from the calibration image. It should be noted that the polygon fitting algorithm can be any polygon fitting algorithm known to those skilled in the art, such as the least squares polygon fitting algorithm and the Douglas-Peucker polygon fitting algorithm, and the embodiment of the present application is not used to specifically limit the polygon fitting algorithm. The region detection algorithm can be an algorithm obtained by training the deep learning algorithm with the first training sample corresponding to the calibration object. Exemplarily, the region detection algorithm can be a detection network with a corner detection head such as YOLO-Pose. It should be noted that the embodiment of the present application is not used to specifically limit the region detection algorithm.

[0088] S502: If the polygon fitting algorithm detects a first candidate region based on the shape of the calibration object, the first candidate region is used as the region to be calibrated.

[0089] In this embodiment, after the electronic device inputs the calibration image into the polygon fitting algorithm and the region detection algorithm respectively, it can determine whether the polygon fitting algorithm detects a first candidate region that conforms to the shape of the calibration object in the calibration image based on the preset calibration object shape. If the electronic device determines that the polygon fitting algorithm detects a first candidate region that conforms to the shape of the calibration object in the calibration image, the electronic device can use the first candidate region determined by the polygon fitting algorithm as the region to be calibrated corresponding to the calibration object.

[0090] S503: If the polygon fitting algorithm fails to detect the first candidate region based on the shape of the calibration object, the second candidate region determined by the region detection algorithm is used as the region to be calibrated.

[0091] In this embodiment, if the electronic device determines that the polygon fitting algorithm fails to detect the first candidate area that matches the shape of the calibration object in the calibration image, the electronic device may use the second candidate area determined by the area detection algorithm as the area to be calibrated corresponding to the calibration object.

[0092] Since the polygon fitting algorithm determines the first candidate area where the calibration object is located from the calibration image according to the preset calibration object shape, the first candidate area determined by the polygon fitting algorithm has a high accuracy. However, since the calibration object shape used by the polygon fitting algorithm is fixed, and the shape of the calibration object in the calibration image taken by the vehicle camera may be severely distorted due to the shooting angle problem, in this case, the polygon fitting algorithm may not be able to determine the first candidate area from the calibration image according to the calibration object shape. In contrast, the region detection algorithm is an algorithm obtained by training the deep learning algorithm with the first training sample corresponding to the calibration object. Therefore, the accuracy of the second candidate area detected by the region detection algorithm may be low, but the region detection algorithm can still detect the second candidate area from the calibration image when the shape of the calibration object in the calibration image is severely distorted.

[0093] In this embodiment, when the polygon fitting algorithm can determine the first candidate area with higher accuracy, since the electronic device can use the first candidate area with higher accuracy as the area to be calibrated, compared with the method of determining the area where the calibration object is located only by the area detection algorithm, the method provided by this embodiment can improve the accuracy of the area to be calibrated determined by the coordinate extraction model. In addition, since this embodiment also combines the area detection algorithm to determine the area where the calibration object is located on the basis of the polygon fitting algorithm, when the polygon fitting algorithm cannot determine the first candidate area with higher accuracy, the electronic device can use the second candidate area calibrated by the area detection algorithm as the area to be calibrated, therefore, the method provided by this embodiment can determine the area where the calibration object is located even when the shape of the calibration object is severely distorted. It can be seen that compared with the method of determining the area where the calibration object is located only by the polygon fitting algorithm, the method provided by this embodiment can improve the availability of the coordinate extraction model.

[0094] In order to further explain the connection relationship between the various algorithm networks in the coordinate extraction model. Figure 6 , shows a specific flow chart of a coordinate extraction model provided by an embodiment of the present application. Figure 6As shown, the coordinate extraction model may include a polygon fitting algorithm, a region detection algorithm, an overall feature extraction network, and a coordinate detection network. After acquiring the calibration image, the electronic device may input the calibration image into the overall feature extraction network to obtain the overall features corresponding to the calibration image. The electronic device may also input the calibration image into the polygon fitting algorithm and the region detection algorithm respectively to determine the area to be calibrated of the calibration object in the calibration image. The electronic device may input the overall features and the area to be calibrated into the coordinate detection network to determine the image coordinates corresponding to the calibration object through the coordinate detection network.

[0095] On the basis of the first embodiment of the present application, in order to further improve the accuracy of the coordinate extraction model, the inventor proposes a coordinate extraction model training method to further improve the accuracy of the coordinate extraction model through the training model training method. Figure 7 FIG. 4 shows a specific implementation flow chart of a camera parameter calibration method before S101 provided in the third embodiment of the present application. Figure 7 , compared to Figure 1 In the embodiment, a camera parameter calibration method provided in the embodiment includes before S101: S701 to S703, which are described in detail as follows:

[0096] S701: Obtain a second training sample and a model to be trained corresponding to the vehicle-mounted panoramic view system, process a training image in the training sample by using the model to be trained, and generate initial coordinates corresponding to the calibration object in the training image.

[0097] In this embodiment, before the electronic device obtains the image coordinates of the calibration object in the calibration image through the coordinate extraction model, it can first obtain the second training sample and the model to be trained corresponding to the vehicle-mounted panoramic surround view system. The second training sample may include the training images corresponding to each vehicle-mounted camera in the vehicle-mounted panoramic surround view system and the expected coordinates corresponding to the calibration object in each training image. The electronic device can process the training images in the training sample through the model to be trained to generate the initial coordinates corresponding to each calibration object in the training image.

[0098] S702: Determine the loss value between the initial coordinates and the expected coordinates based on a preset loss function.

[0099] In this embodiment, after determining the initial coordinates corresponding to each calibration object in the training image, the electronic device can determine the loss value between the initial coordinates corresponding to the calibration object and the expected coordinates according to the loss function preset by the R&D personnel.

[0100] In a possible implementation, the training sample may also include a desired panoramic image formed by stitching all the training images in the training sample. Figure 8, shows a schematic diagram of a loss value calculation process provided by an embodiment of the present application. Figure 8 As shown, the loss function in the electronic device includes a first loss function and a second loss function. The loss value calculated by the electronic device through the loss function includes a coordinate loss value and a pixel loss value. After determining the initial coordinates corresponding to each calibration object in the training image, the electronic device can input the initial coordinates and expected coordinates corresponding to each calibration object into the first loss function to generate a coordinate loss value corresponding to any vehicle-mounted camera. Exemplarily, the first loss function can be any loss function known to those skilled in the art, such as a mean square error loss function, and the embodiments of the present application are not used to specifically limit the first loss function.

[0101] After obtaining the initial coordinates of the calibration object in the training image corresponding to a vehicle-mounted camera, the electronic device can determine the initial parameters corresponding to the vehicle-mounted camera according to the internal parameters corresponding to the vehicle-mounted camera, the initial coordinates of the calibration object, and the world coordinates of the calibration object in the vehicle coordinate system. Exemplarily, the electronic device can input the internal parameters, the initial coordinates, and the world coordinates into the perspective n-point function to determine the initial parameters corresponding to the vehicle-mounted camera.

[0102] Then, the electronic device can splice all the training images in the training sample according to the initial parameters corresponding to each on-board camera to determine the initial panoramic image corresponding to the initial coordinates. Specifically, the electronic device can project the training images corresponding to each on-board camera to the bird's-eye view (Bird Eye View, BEV) according to the initial parameters corresponding to each on-board camera in the on-board panoramic surround view system to generate an initial panoramic image. After determining the initial panoramic image, the electronic device can input the initial panoramic image and the expected panoramic image into the second loss function to determine the pixel loss value. Among them, the pixel loss value can be the loss value of each pixel in the overlapping area when each training image in the initial panoramic image is spliced, that is, the loss value of each pixel in the overlapping area that can be captured in the field of view of adjacent on-board cameras. Among them, the overlapping area can be the calibration area where the calibration object is located.

[0103] Specifically, when calculating the pixel loss value, the electronic device may input the initial panoramic image and the expected panoramic image into the mean absolute error function to calculate the initial loss value corresponding to each pixel in the overlapping area. Then, the electronic device may input the loss value corresponding to each pixel into the adjustment function to adjust the initial loss value of each pixel and determine the pixel loss value. The adjustment function may be as follows.

[0104]

[0105] Among them, L pixelIt can be used to represent the initial loss value corresponding to a pixel in the overlapping area. pixel It can be used to represent the pixel loss value corresponding to a certain pixel in the overlapping area. t1 can be used to represent the first coefficient. Exemplarily, the value range of the first coefficient t1 can be [0.005, 0.25]. t2 can be used to represent the second coefficient. Exemplarily, the value range of the second coefficient t2 can be [0.01, 0.1].

[0106] In one possible implementation, the electronic device may also preprocess the stitched initial panoramic image before calculating the pixel loss value to reduce the image difference in the overlapping area caused by the difference in the shooting angle and internal parameters of the vehicle-mounted camera in the initial panoramic image. Specifically, the electronic device may smooth the initial panoramic image through a low-pass filter to reduce the impact of noise on the overlapping area. Then, the electronic device may adjust the smoothed initial panoramic image according to preset brightness parameters and hue parameters to reduce the impact of the difference in light input and response rate between different vehicle-mounted cameras on the overlapping area. Then, the electronic device may input the preprocessed initial panoramic image and the expected panoramic image into the second loss function to determine the pixel loss value.

[0107] In this embodiment, the electronic device can not only calculate the coordinate loss value corresponding to the initial coordinates, but also calculate the pixel loss value after the image is processed based on the external parameters determined by the initial coordinates. In terms of improving the accuracy of loss value calculation, the coordinate loss value can intuitively present the deviation between the model prediction and the real coordinates from the coordinate dimension, while the pixel loss value can focus on the difference between the pixel points after image processing. Therefore, the coordinate loss value is combined with the pixel loss value, which can integrate the coordinate information with the pixel detail information, and comprehensively consider the gap between the model prediction result and the actual situation. In addition, since the spliced ​​image requires geometric transformation of the image, the calculation of the pixel loss value can take into account the error caused by the geometric transformation, so that the loss value can accurately reflect the pixel changes caused by the coordinate deviation and image transformation, and the accuracy is greatly improved compared to the coordinate loss value alone. In summary, the method provided in this embodiment can improve the calculation accuracy of the loss value, thereby improving the accuracy of the trained model.

[0108] In addition, since the overlapping area of ​​a panoramic image formed by stitching together multiple images is difficult to be completely consistent with the area in the desired panoramic image, the pixel loss value is calculated so that the electronic device can weaken the impact of pixels with lower loss values ​​on the overall loss value by adjusting the function, which helps to speed up the training efficiency of the model to be trained and improve the training efficiency of the electronic device for the model to be trained.

[0109] S703, updating the model to be trained based on the loss value until the loss value meets the preset training stop condition, and using the model to be trained corresponding to when the loss value meets the training stop condition as the coordinate extraction model.

[0110] In this embodiment, after determining the coordinate loss value and the pixel loss value, the electronic device can update the model to be trained according to the coordinate loss value and the pixel loss value. Then, the electronic device can process the training sample through the updated model to be trained, generate the initial coordinates, and calculate the loss value corresponding to the new initial coordinates. The electronic device can train the model to be trained multiple times through the training samples until the loss value meets the preset training stop condition. At this time, the electronic device can use the model to be trained corresponding to the loss value meeting the training stop condition as the coordinate extraction model. Among them, the coordinate extraction model can include an overall feature extraction network and a coordinate detection network.

[0111] In one possible implementation, the electronic device may determine whether the loss value meets the training stop condition through a loss threshold value pre-set by a developer. Specifically, the loss threshold value pre-set by the developer may include a first loss threshold value and a second loss threshold value. When the electronic device determines that the coordinate loss value is less than or equal to the first loss threshold value, and the pixel loss value is less than or equal to the second loss threshold value, the electronic device may determine that the loss value meets the training stop condition, and the electronic device may use the model to be trained at this time as a coordinate extraction model. When the electronic device determines that the coordinate loss value is greater than the first loss threshold value, and / or the pixel loss value is greater than the second loss threshold value, the electronic device may determine that the loss value does not meet the training stop condition, and the electronic device updates the model to be trained according to the coordinate loss value and the pixel loss value, and returns to execute steps S701 to S703.

[0112] In this embodiment, the coordinate loss value mainly measures the difference between the predicted coordinates and the actual coordinates. When training the model, relying solely on the coordinate loss value may cause the electronic device to over-focus on the accuracy of the coordinates and ignore the information at the pixel level. The pixel loss value measures the difference between the predicted pixel-level content and the actual pixel content. When training the model, if only the pixel loss value is considered, the model may focus too much on the restoration of pixel details and ignore the accuracy of the coordinates. In this embodiment, since the electronic device can consider the loss value of the model to be trained in combination with the coordinate loss value and the pixel loss value when training the model, the training method can improve the accuracy of the overall features extracted by the model while improving the accuracy of the coordinate extraction with the help of more accurate overall features.

[0113] It should be noted that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0114] Reference Fig. 9 , shows a schematic diagram of a camera parameter calibration device provided in an embodiment of the present application, which may specifically include a feature acquisition module 901, a coordinate generation module 902 and a parameter determination module 903, wherein:

[0115] The feature acquisition module 901 is used to input the corresponding input of any vehicle-mounted camera into the overall feature extraction network to determine the overall features corresponding to the calibration image; the overall features include the features of the area to be calibrated where the calibration object is located and the features of the area not to be calibrated in the calibration image;

[0116] A coordinate generation module 902 is used to input the area to be calibrated where the calibration object is located and the overall features into a coordinate detection network to generate image coordinates of the calibration object in an image coordinate system;

[0117] The parameter determination module 903 is used to determine the external parameters corresponding to any vehicle-mounted camera based on the image coordinates and calibration information.

[0118] The coordinate generation module can also be used to extract features from the calibration image through multiple first convolutional layers in the overall feature extraction network to generate multiple feature images about the calibration image; downsample each feature image in turn according to the network level order of the first convolutional layer corresponding to each feature image to generate feature information corresponding to each feature image; and generate the overall features corresponding to the calibration image based on the feature information corresponding to each network level.

[0119] The coordinate generation module can also be used to input the calibration image into a polygon fitting algorithm and a region detection algorithm respectively; the polygon fitting algorithm is an algorithm for fitting polygons in the calibration image according to a preset calibration object shape to determine the region where the calibration object is located from the calibration image; the region detection algorithm is an algorithm obtained by training a deep learning algorithm based on a first training sample generated based on the calibration object; if the polygon fitting algorithm detects a first candidate region based on the calibration object shape, the first candidate region is used as the region to be calibrated.

[0120] The coordinate generation module may also be configured to use a second candidate area determined by the area detection algorithm as the area to be calibrated if the polygon fitting algorithm fails to detect the first candidate area based on the shape of the calibration object.

[0121] The coordinate generation module can also be used to input the area to be calibrated into the regional feature extraction network to obtain the regional features corresponding to the area to be calibrated; perform feature fusion on the regional features and the overall features to determine the fusion features; input the fusion features into the second convolution layer to generate the image coordinates of the calibration object in the image coordinate system.

[0122] The coordinate acquisition module can also be used to obtain a second training sample and a model to be trained corresponding to the vehicle-mounted panoramic view system, and process the training image in the training sample through the model to be trained to generate the initial coordinates corresponding to the calibration object in the training image; the vehicle-mounted panoramic view system includes at least one vehicle-mounted camera; the second training sample includes training images corresponding to each vehicle-mounted camera and expected coordinates corresponding to the calibration object in the training image; the loss value between the initial coordinates and the expected coordinates is determined based on a preset loss function; the model to be trained is updated based on the loss value until the loss value meets the preset training stop condition, and the model to be trained corresponding to the loss value meeting the training stop condition is used as the coordinate extraction model; the coordinate extraction model includes the overall feature extraction network and the coordinate detection network.

[0123] The coordinate acquisition module can also be used to input the initial coordinates of the training image corresponding to any vehicle-mounted camera in the vehicle-mounted panoramic view system and the expected coordinates into the first loss function to generate a coordinate loss value corresponding to any vehicle-mounted camera; determine the initial parameters corresponding to any vehicle-mounted camera based on the initial coordinates, the world coordinates of the calibration object in the vehicle coordinate system and the internal parameters corresponding to any vehicle-mounted camera; splice all the training images in the training sample based on the initial parameters corresponding to each of the vehicle-mounted cameras to determine the initial panoramic image; input the initial panoramic image and the expected panoramic image into the second loss function to determine the pixel loss value.

[0124] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiment part.

[0125] Reference Fig.10 , shows a schematic diagram of an electronic device provided by an embodiment of the present application. Fig.10 As shown, the electronic device 1000 in the embodiment of the present application includes: a processor 1010, a memory 1020, and a computer program 1021 stored in the memory 1020 and executable on the processor 1010. When the processor 1010 executes the computer program 1021, the steps of each embodiment of the parameter calibration method of the above-mentioned camera are implemented, for example Figure 1Alternatively, when the processor 1010 executes the computer program 1021, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Fig. 9 Functions of modules 901 to 902 are shown.

[0126] Exemplarily, the computer program 1021 may be divided into one or more modules / units, which are stored in the memory 1020 and executed by the processor 1010 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which may be used to describe the execution process of the computer program 1021 in the electronic device 1000. For example, the computer program 1021 may be divided into a coordinate acquisition module and a parameter determination module, and the specific functions of each module are as follows:

[0127] A coordinate acquisition module, used to input a calibration image corresponding to any vehicle-mounted camera into a coordinate extraction model, and obtain the image coordinates of the calibration object in the calibration image in the image coordinate system through the overall features determined by the coordinate extraction model; the coordinate extraction model includes an overall feature extraction network; the overall feature extraction network is used to obtain the overall features corresponding to the calibration image;

[0128] A parameter determination module is used to determine the external parameters corresponding to any vehicle-mounted camera based on the image coordinates, the world coordinates of the calibration object in the vehicle coordinate system and the internal parameters corresponding to any vehicle-mounted camera.

[0129] The electronic device 1000 may be a computing device such as a desktop computer, a cloud server, or a vehicle-mounted terminal. The electronic device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art will appreciate that Fig.10 It is only an example of the electronic device 1000 and does not constitute a limitation of the electronic device 1000. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 1000 may also include input and output devices, network access devices, buses, etc.

[0130] The processor 1010 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0131] The memory 1020 may be an internal storage unit of the electronic device 1000, such as a hard disk or memory of the electronic device 1000. The memory 1020 may also be an external storage device of the electronic device 1000, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1000. Further, the memory 1020 may also include both an internal storage unit of the electronic device 1000 and an external storage device. The memory 1020 is used to store the computer program 1021 and other programs and data required by the electronic device 1000. The memory 1020 may also be used to temporarily store data that has been output or is to be output.

[0132] An embodiment of the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the camera parameter calibration method described in the aforementioned embodiments is implemented.

[0133] The embodiments of the present application further disclose a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the camera parameter calibration method described in the aforementioned embodiments is implemented.

[0134] The embodiment of the present application further discloses a computer program product. When the computer program product is run on a computer, the computer is enabled to execute the camera parameter calibration method described in the aforementioned embodiments.

[0135] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above-mentioned embodiments, a person skilled in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A camera calibration method, characterized in that: include: Inputting the corresponding input of any vehicle-mounted camera into the overall feature extraction network to determine the overall features corresponding to the calibration image; the overall features include the features of the area to be calibrated where the calibration object is located and the features of the area not to be calibrated in the calibration image; Inputting the area to be calibrated where the calibration object is located and the overall features into a coordinate detection network to generate image coordinates of the calibration object in an image coordinate system; Based on the image coordinates and the calibration information, the external parameters corresponding to any of the vehicle-mounted cameras are determined.

2. The method according to claim 1, characterized in that The step of inputting the calibration image into an overall feature extraction network to determine the overall features corresponding to the calibration image includes: Extracting features from the calibration image through a plurality of first convolutional layers in the overall feature extraction network to generate a plurality of feature images related to the calibration image; According to the network layer order of the first convolutional layer corresponding to each of the feature images, downsampling operations are performed on each of the feature images in sequence to generate feature information corresponding to each of the feature images; The overall features corresponding to the calibration image are generated based on the feature information corresponding to each network level.

3. The method according to claim 1, characterized in that Also includes: Inputting the calibration image into a polygon fitting algorithm and a region detection algorithm respectively; The polygon fitting algorithm is an algorithm for fitting polygons in the calibration image according to a preset calibration object shape to determine the area where the calibration object is located from the calibration image; the area detection algorithm is an algorithm obtained by training a deep learning algorithm based on a first training sample generated based on the calibration object; If the polygon fitting algorithm detects a first candidate region based on the shape of the calibration object, the first candidate region is used as the region to be calibrated.

4. The method according to claim 3, characterized in that Also includes: If the polygon fitting algorithm fails to detect the first candidate region based on the shape of the calibration object, the second candidate region determined by the region detection algorithm is used as the region to be calibrated.

5. The method according to claim 1, characterized in that The coordinate detection network includes a regional feature extraction network and a second convolutional layer; The step of inputting the to-be-calibrated area where the calibration object in the calibrated image is located and the overall features into the coordinate detection network to generate the image coordinates of the calibration object in the image coordinate system comprises: Inputting the area to be calibrated into a regional feature extraction network to obtain regional features corresponding to the area to be calibrated; Performing feature fusion on the regional features and the overall features to determine fusion features; The fused features are input into the second convolutional layer to generate the image coordinates of the calibration object in the image coordinate system.

6. The method according to any one of claims 1 to 5, characterized in that: Also includes: Acquire a second training sample and a model to be trained corresponding to the vehicle-mounted panoramic view system, and process the training image in the training sample by the model to be trained to generate initial coordinates corresponding to the calibration object in the training image; the vehicle-mounted panoramic view system includes at least one vehicle-mounted camera; The second training samples include training images corresponding to the respective vehicle-mounted cameras and expected coordinates corresponding to the calibration objects in the training images; Determining a loss value between the initial coordinates and the expected coordinates based on a preset loss function; The model to be trained is updated based on the loss value until the loss value satisfies a preset training stop condition, and the model to be trained corresponding to when the loss value satisfies the training stop condition is used as a coordinate extraction model; the coordinate extraction model includes the overall feature extraction network and the coordinate detection network.

7. The method according to claim 6, characterized in that The loss value includes a coordinate loss value and a pixel loss value; the training sample also includes a desired panoramic image stitched together from all training images in the training sample; The determining, based on a preset loss function, a loss value between the initial coordinates and the expected coordinates corresponding to the calibration object includes: For any on-board camera in the on-board panoramic surround view system, inputting the initial coordinates of the training image corresponding to any on-board camera and the expected coordinates into a first loss function to generate a coordinate loss value corresponding to any on-board camera; Determine the initial parameters corresponding to any of the on-board cameras based on the initial coordinates, the world coordinates of the calibration object in the vehicle coordinate system, and the internal parameters corresponding to any of the on-board cameras; Based on the initial parameters corresponding to each of the vehicle-mounted cameras, all the training images in the training sample are stitched together to determine an initial panoramic image; The initial panoramic image and the desired panoramic image are input into a second loss function to determine the pixel loss value.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the camera parameter calibration method as described in any one of claims 1 to 7.

9. A computer program product, when the computer program product is run on a computer, enables the computer to execute the camera parameter calibration method according to any one of claims 1 to 7.

10. A vehicle, characterized in that: It comprises an electronic device, and the electronic device is used to execute the parameter calibration method of the camera as described in any one of claims 1-7.

Images